Jump to content
danklogic

Pending changes view never finishes loading

Recommended Posts

We're trying to create an initial commit of about 220,000 files on a windows machine. The PlasticSCM server is running Windows 7 with a MySQL back-end. When trying to commit the changes, the 'Pending Changes' view never finishes updating. It gets stuck on 'Filling grid...' and never completes, even after a few hours of waiting. This is on a pretty fast machine with 128GB of ram.

 

Is this typical behavior? How do I go about figuring out the bottleneck? Is there a better strategy for commiting this many files at once?

I've run 'cm iostats' with the following results:

Performing network tests with server: PLASTICSCM:8087. Please wait...

Upload speed   = 373.18 Mbps.   Time uploading  16MB = 343ms.      Server: PLASTICSCM:8087
Download speed = 390.24 Mbps. Time downloading  16MB = 328ms.      Server: PLASTICSCM:8087
 
Performing disk speed test on path: C:\Users\user1\AppData\Local\Temp\PlasticSCM_IOStats. Please wait...
Disk write speed = 108.68180853322 MB/s. Time writing  512MB = 4711 ms.
Disk read speed  = 1426.18384401114 MB/s. Time reading  512MB = 359 ms.
 
Any help would be greatly appreciated. Thank you.
 
 

Share this post


Link to post
Share on other sites

Hi,

 

In the pending changes view options, there are a few of them marked as * . It means that the options requieres disk search and it could take a while.  Anyway, if you are waiting so long, it doesn´t make sense for me.

 

 

Could you try to commit the changes using the command line?

 

"cm add"  +  "cm ci"  comands. 

 

PD: In case you are not able to perform the commit, please enable the client log and send it to:  support at codicesoftware dot com

https://www.plasticscm.com/documentation/technical-articles/kb-enabling-logging-for-plastic-scm-part-i.html

 

 

Regards,

Carlos.

Share this post


Link to post
Share on other sites

Thanks for the help. I was eventually able to check everything in using some command line piping I found elsewhere on this forum. The GUI just could not handle it for some reason.

I think the commands I wound up using to commit everything were:

cm findprivate -R | cm add -
cm ci

Share this post


Link to post
Share on other sites

Chiming in on this as I am having the same problem with a 74GB repo with 153,000 files.  

Has anyone found a solution to this issue?

Here are my iostats results

Upload speed   = 6.26 Mbps.    Time uploading  16MB = 20453ms.    
Download speed = 42.67 Mbps. Time downloading  16MB = 3000ms.

Performing disk speed test on path: 
Disk write speed = 345.013477088949 MB/s. Time writing  512MB = 1484 ms.
Disk read speed  = 1641.02564102564 MB/s. Time reading  512MB = 312 ms.

 

I do notice that this problem only happens on repos where I haven't committed much, ie: I have 2 out of 153,000 files committed.. On other repos where 100% of the files are committed and I add a new file for example, this issue does not happen

Share this post


Link to post
Share on other sites

An update on this as it's rearing it's head once again in a different form, this time preventing me from checking in my initial commit.  I have gone and and selected all items in the workspace, right clicked and clicked add to source control.  I go over to Pending changes to check it in and it is permanently stuck Finding Changes in the Workspace.  

I have clicked 'undo checkout' and it still is stuck at Finding changes in the workspace

After that I have tried adding just a single file to source and going to Pending Changes and again it is hung here

I have tried changing workspaces to one for another repository, and still Pending changes was stuck at finding changes.  Although this time only for 10-20 seconds

I then changed back to the workspace for the repo that I had original checkout(and then undid) the files in and once again we are stuck finding changes indefinitely.

At this point in desperation after waiting 45 minutes I killed the client through the task manager, but much to my dismay once starting it back up again it is again stuck Finding changes in the Workspace.

I noticed that it wasn't generating a log so I re-ran the client as an administrator and this time it got past Finding changes and then got stuck at Filling Grid.  This time however it did manage to write to the log and within 30 seconds I had roughly 13,000 lines like this:

2017-03-15 22:04:44,360 DEBUG BufferPool - <- Releasing with name FileTypePredictor (4)
2017-03-15 22:04:44,360 DEBUG BufferPool - -> Entering with name FileTypePredictor (5)
2017-03-15 22:04:44,360 DEBUG BufferPool - Waiting for FileTypePredictor:0 ms

 

I understand that there are some large companies using this tool on projects much larger than ours, so please tell me what they are doing differently in order to handle a 70GB+ with 150,000+ files because simply by attempting to check in the project and submit it to the server I seem to have broken the client

Update 2 - After going through this the first time, I was finally able to have it 'fill the grid' after waiting for 30-35 minutes.  I undid everything to see if I could repro.  The second time it went through and added the files in a somewhat timely fashion, only waited 10-15 minutes to fill the grid/finding changes.  So I can't repro this huge indefinite hang 100% of the time but I also can't explain why it worked the second time.  One thing learned is that plastic prefers to be run as an administrator.

During this successful last attempt I do see 35,000 lines of the following in my log file:

2017-03-15 22:37:56,980 INFO  TimerPerformance - LoadWorkspaceTree time 0.

The log file is already up to 135MB so I'll turn off logging now :)

Share this post


Link to post
Share on other sites

Yes, the problem might be the file type predictor.

I've written you from the support email requesting more info.

 

Thanks!

Share this post


Link to post
Share on other sites

Hi, can you share solution for this problem? I have infinite loadin on plastic GUI -> pending changes and log says 

2019-03-11 10:37:22,649 DEBUG BufferPool - -> Entering with name FileTypePredictor (5)
2019-03-11 10:37:22,649 DEBUG BufferPool - Waiting for FileTypePredictor:0 ms
2019-03-11 10:37:22,649 DEBUG BufferPool - <- Releasing with name FileTypePredictor (4)
2019-03-11 10:37:22,649 DEBUG BufferPool - -> Entering with name FileTypePredictor (5)
2019-03-11 10:37:22,649 DEBUG BufferPool - Waiting for FileTypePredictor:0 ms
2019-03-11 10:37:22,655 DEBUG BufferPool - <- Releasing with name FileTypePredictor (4)

(almost 1kk lines of log now).

I'm able to check-in from cmd but when I need to make more "advanced" operation than simply checkin all it's taking very long time compared with Plastic GUI client.

Share this post


Link to post
Share on other sites

It seems we have some kind of problem with the FileTypePredictor in pretty big workspaces. We're trying to understand, locate and fix the issue.

In the meantime, you can fine tune your plastic installation options to improve the pending changes performance. The config files are located in the C:\Users\<user>\AppData\Local\plastic4 folder:

* do not display private items in the pending changes view:

Edit the guiclient.conf

<CommitShowPrivates>false</CommitShowPrivates>

* do not detect manually moved files:

Edit the guiclient.conf

<CommitCalculateManuallyMoved>false</CommitCalculateManuallyMoved>

* compare file contents using the timestamp instead the file content: This may help to improve the pending changes view perf.

Open the client.conf file and setup the following line:

  <CheckFileContentForChanged>no</CheckFileContentForChanged>

Please, share with us if this configuration helps to improve the pending changes time.

Share this post


Link to post
Share on other sites

Thanks for response. Sadly, I've already did all of these and it didn't help. Its not really about "improve the pending changes time" because I've left pending changes window for night and nothing happend, just 10GB of DEBUG BufferPool. 

Share this post


Link to post
Share on other sites

That doesn't sound good 😕

Does the CLI has the same behavior? Try to execute:

$ cm status

Also, you can follow some extra steps that helped other users to solve pending changes perf issues:

There are several reasons that can probably affect to that operation:

  • The hard drive performance is not good as expected. Do you have a mechanical hard drive? Is there any other process causing a high disk usage?
  • An antivirus software is running. Disable your antivirus to reduce the disk usage. Very often antivirus software running on developer machines slow down all version control operations and turn a powerful computer into a nearly unusable state. Double check if the antivirus is eating CPU or disk IO while pending changes is working.
  • Windows superfetch service. The superfetch service has been identified as a potential cause of disk performance issues. Stopping this service may have a good effect on your computer's performance.

Share this post


Link to post
Share on other sites

Disk usage is 0%. cm status --all shows pending changes properly. Also, Gluon shows pending changes properly. It seems like only PlastucGUI has problem with pending changes tab. And this problem only occures on my biggest workspace, other ones (much smaller) works perfectly even on PlasticGUI. 

Share this post


Link to post
Share on other sites

Thank you very much @Yerendi for that information.

Can you please share your workpace size (number of files/dirs, size on disk, aprox tree deph, etc ...).

Thanks!

Share this post


Link to post
Share on other sites

It's ~45 GB, but 15GB is marked as ignored.  With ignored its ~700k files,  not ignored is ~90k. Tree is quite simple, sth like main -> task -> subtasks, so I would say deph 3/4 is max.

Share this post


Link to post
Share on other sites

@Yerendi thanks for the info. I'll try to reproduce your issue.

With the depth I meant the depth of the directory structure (distance from the root dir to the deepest leaf (aprox).

Share this post


Link to post
Share on other sites

Max 6 (in edge cases probably can be more but I didn't find any), in most cases it's folder/folder/folder/folder/target and folder/folder/folder/target. Almost all (99.9%, including ignored ones) are at least folder/target (root/"2nd root"/all files). If it makes any difference I have few big binary files (~100 mb).

Edit: If it helps - it's Unity project and Plastic is set in Project folder (so in folder with Assets/Library/ProjectSettings folders).

Share this post


Link to post
Share on other sites

@Yerendi still looking your case ...

Just for discard things,

As you have several ignored items, did you disable ignored files in the pending changes view?

  <ShowIgnoredItems>false</ShowIgnoredItems>

I still trying to reproduce your case. Does your pending changes view get stopped in the "Filling grid" phase?

Share this post


Link to post
Share on other sites

Ignored changes was set to false in config file but not in Options! Pending changes now works great! After rechecking cfg file I've checked it in PlasticGUI (Pending changes -> Options). In "What to show" tab "Show ignored changes" was checked. Dunno if it's bug that toggle in options overrides value in cfg but now its working! :D

  • Haha 1

Share this post


Link to post
Share on other sites

Great to hear that!

Anyway, it seems that it's something that we can improve when searching ignored items...

Thanks for the feedback!

  • Thanks 1

Share this post


Link to post
Share on other sites

I think that repro is simple - I've marked this toggle in options few days ago and pending changes get destroyed. Maybe that when pending changes are "destroyed" Options window values overrides cfg file or sth like that.

  • Thanks 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...