How to increase the upload/download speed?

[ironbelly]

Situation:

• Our linux server is in Phoenix AZ, around 4100KM away from us.  I have no intention of using a windows server to host Plastic(or anything else for that matter)
• This distance means that our TCP saturation point is reached at around 15-30Mbps per thread
• Our office has 250Mbps fiber 

I am evaluating Plastic to see if we want to switch over from Perforce and the first thing that troubles me is that it appears like Plastic transfers over a single TCP thread.  Even though our upload speeds here are 250Mbps, while transfering a test project(60,000 files and 55GB) I cannot achieve faster upload speeds than 15Mbps.

 

How do I overcome this so that we can utilize more of our available bandwidth and decrease transfer times?  Thanks

[psantosl]

Hi!

 

You can check out this post from the folks at FacePunch Studios explaining how to configure the parallel upload/download to boost speed:

 

https://garry.tv/2016/07/13/plasticscm-speed-tip/

[ironbelly]

Awesome, thanks for digging that up for me.  Now is this same thing possible with upload?   That being said it would be nice to have a full document explaining all of the different variables which are not self-explanatory  such as DownloadPoolSize, QueuesSize, UploadCompressionSize, etc etc.

 

Oh and to quote the link above for anyone else coming in here:

Quote

 

Find your client.conf file – usually in C:\Users\username\AppData\Local\plastic4. Open it, scroll down to

<DownloadPoolSize>1</DownloadPoolSize>

and change it to

<DownloadPoolSize>16</DownloadPoolSize>

Now you’re downloadING at 200mbps. Cya.

 

 

[manu]

For the upload operation you can tweak the following values, also inside the client.conf file:

<UploadCompressionSize>
<UploadReadPoolSize>

You'll start using more threads for reading file contents and compress them, really useful when your computer has an SSD drive.

We'll try to summarize all this knowledge in a blogpost

 

 

[ironbelly]

So I'll start using more threads for reading file contents, which is great, and I will compress them more, which is also great, however as per my OP here the main bottleneck is the single-threaded upload.  This being an Unreal Project, all of the files are already pretty compressed so my pain point here is my upload speed.  As mentioned above, our TCP saturation point is about 15-20mbps per thread, so the key to improving the upload speeds is to enable multi-threaded uploads.  Is this possible?  Can I run multiple plastic clients and checkin separate sets of files concurrently?  Is there a setting where I can increase the upload threads from 1 to 4 or 8 ?  

The equivalent in Perforce is parallel sync

 

PS - Upload Read pool size isn't in the client.conf by default, so what is the default value for that?

[manu]

Hi, 

17 hours ago, ironbelly said:

so the key to improving the upload speeds is to enable multi-threaded uploads.  Is this possible?

No, sorry, it's not possible right now, but we want to support it.

17 hours ago, ironbelly said:

 Can I run multiple plastic clients and checkin separate sets of files concurrently?

Not really for the same repository, you'll end up forcing merge operations for the second concurrent checkin.

It's doable of course for independent branches/repositories.

If this is really critical for you you can upload to different branches from multiple clients and then unify the branches using the merge operation. But it sounds to me like to much for an operation you do it once (starting a new project/adding it to Plastic).

17 hours ago, ironbelly said:

PS - Upload Read pool size isn't in the client.conf by default, so what is the default value for that?

The UploadReadPoolSize default value is "1".

 

[ironbelly]

3 hours ago, manu said:

Hi, 

No, sorry, it's not possible right now, but we want to support it.

For remotely distributed teams this is the single biggest barrier to adoption for PlasticSCM in my opinion.  We have guys in Europe who reach their TCP saturation at 3-5Mbps, can you guess how long it takes to commit 15GB of data at 3Mbps?

For onsite stuff obviously this isn't an issue but this alone would be a reason for remote steams to stick with perforce over Plastic, assuming they weren't having a plethora of other issues with Perforce making their lives a living hell

[manu]

Hi!

I see your point, do you mind to schedule an online meeting with me in order to schedule a few upload tests? 

 

[psantosl]

Hi ironbelly,

Well, our customers, specially bit teams, report regularly that Plastic always outperform P4. I don't doubt we could be faster, but I'm not sure it would be the key reason for massive adoption :-). Hope it is :-)

For distributed teams we have "fully distributed development" with true push/pull (unlike others). But I bet you have artists in mind, correct?

Then, we also have a way to use a UDP based protocol for high latency. It is only for Windows so far, I'm afraid.

Finally, we are going to contact you to reproduce your results and try to figure out what is going on. Just to make it crystal clear: we consistently outperform competitors in checkin and update performance, both LAN (specially) and WAN. This is a key reason why many large companies choose Plastic. You can see some numbers here: https://www.plasticscm.com/version-control-for-games.html#performance-results

 

 

[holikri]

Hello!

I did that change:
"

Find your client.conf file – usually in C:\Users\username\AppData\Local\plastic4. Open it, scroll down to

<DownloadPoolSize>1</DownloadPoolSize>

and change it to

<DownloadPoolSize>16</DownloadPoolSize>

Now you’re downloadING at 200mbps. Cya.

"
but it's still downloading with around 1mb/s, i have 100/100 fiber.

Edit:
After restarting, now i'm getting the following in Plastic SCM
Cannot retrieve license information from the Plastic SCM server. Retrying in "counter", and button with Try again now