danklogic Posted April 15, 2015 Report Share Posted April 15, 2015 We're trying to create an initial commit of about 220,000 files on a windows machine. The PlasticSCM server is running Windows 7 with a MySQL back-end. When trying to commit the changes, the 'Pending Changes' view never finishes updating. It gets stuck on 'Filling grid...' and never completes, even after a few hours of waiting. This is on a pretty fast machine with 128GB of ram. Is this typical behavior? How do I go about figuring out the bottleneck? Is there a better strategy for commiting this many files at once?I've run 'cm iostats' with the following results:Performing network tests with server: PLASTICSCM:8087. Please wait... Upload speed = 373.18 Mbps. Time uploading 16MB = 343ms. Server: PLASTICSCM:8087 Download speed = 390.24 Mbps. Time downloading 16MB = 328ms. Server: PLASTICSCM:8087 Performing disk speed test on path: C:\Users\user1\AppData\Local\Temp\PlasticSCM_IOStats. Please wait... Disk write speed = 108.68180853322 MB/s. Time writing 512MB = 4711 ms. Disk read speed = 1426.18384401114 MB/s. Time reading 512MB = 359 ms. Any help would be greatly appreciated. Thank you. Link to comment Share on other sites More sharing options...
calbzam Posted April 16, 2015 Report Share Posted April 16, 2015 Hi, In the pending changes view options, there are a few of them marked as * . It means that the options requieres disk search and it could take a while. Anyway, if you are waiting so long, it doesn´t make sense for me. Could you try to commit the changes using the command line? "cm add" + "cm ci" comands. PD: In case you are not able to perform the commit, please enable the client log and send it to: support at codicesoftware dot com https://www.plasticscm.com/documentation/technical-articles/kb-enabling-logging-for-plastic-scm-part-i.html Regards, Carlos. Link to comment Share on other sites More sharing options...
danklogic Posted April 17, 2015 Author Report Share Posted April 17, 2015 Thanks for the help. I was eventually able to check everything in using some command line piping I found elsewhere on this forum. The GUI just could not handle it for some reason. I think the commands I wound up using to commit everything were:cm findprivate -R | cm add -cm ci Link to comment Share on other sites More sharing options...
calbzam Posted April 17, 2015 Report Share Posted April 17, 2015 Oks, thanks for upgrading the issue. If something similar happens again, please enable the client log. Regards, Carlos. Link to comment Share on other sites More sharing options...
ironbelly Posted March 15, 2017 Report Share Posted March 15, 2017 Chiming in on this as I am having the same problem with a 74GB repo with 153,000 files. Has anyone found a solution to this issue? Here are my iostats results Upload speed = 6.26 Mbps. Time uploading 16MB = 20453ms. Download speed = 42.67 Mbps. Time downloading 16MB = 3000ms. Performing disk speed test on path: Disk write speed = 345.013477088949 MB/s. Time writing 512MB = 1484 ms. Disk read speed = 1641.02564102564 MB/s. Time reading 512MB = 312 ms. I do notice that this problem only happens on repos where I haven't committed much, ie: I have 2 out of 153,000 files committed.. On other repos where 100% of the files are committed and I add a new file for example, this issue does not happen Link to comment Share on other sites More sharing options...
ironbelly Posted March 16, 2017 Report Share Posted March 16, 2017 An update on this as it's rearing it's head once again in a different form, this time preventing me from checking in my initial commit. I have gone and and selected all items in the workspace, right clicked and clicked add to source control. I go over to Pending changes to check it in and it is permanently stuck Finding Changes in the Workspace. I have clicked 'undo checkout' and it still is stuck at Finding changes in the workspace After that I have tried adding just a single file to source and going to Pending Changes and again it is hung here I have tried changing workspaces to one for another repository, and still Pending changes was stuck at finding changes. Although this time only for 10-20 seconds I then changed back to the workspace for the repo that I had original checkout(and then undid) the files in and once again we are stuck finding changes indefinitely. At this point in desperation after waiting 45 minutes I killed the client through the task manager, but much to my dismay once starting it back up again it is again stuck Finding changes in the Workspace. I noticed that it wasn't generating a log so I re-ran the client as an administrator and this time it got past Finding changes and then got stuck at Filling Grid. This time however it did manage to write to the log and within 30 seconds I had roughly 13,000 lines like this: 2017-03-15 22:04:44,360 DEBUG BufferPool - <- Releasing with name FileTypePredictor (4) 2017-03-15 22:04:44,360 DEBUG BufferPool - -> Entering with name FileTypePredictor (5) 2017-03-15 22:04:44,360 DEBUG BufferPool - Waiting for FileTypePredictor:0 ms I understand that there are some large companies using this tool on projects much larger than ours, so please tell me what they are doing differently in order to handle a 70GB+ with 150,000+ files because simply by attempting to check in the project and submit it to the server I seem to have broken the client Update 2 - After going through this the first time, I was finally able to have it 'fill the grid' after waiting for 30-35 minutes. I undid everything to see if I could repro. The second time it went through and added the files in a somewhat timely fashion, only waited 10-15 minutes to fill the grid/finding changes. So I can't repro this huge indefinite hang 100% of the time but I also can't explain why it worked the second time. One thing learned is that plastic prefers to be run as an administrator. During this successful last attempt I do see 35,000 lines of the following in my log file: 2017-03-15 22:37:56,980 INFO TimerPerformance - LoadWorkspaceTree time 0. The log file is already up to 135MB so I'll turn off logging now Link to comment Share on other sites More sharing options...
manu Posted March 16, 2017 Report Share Posted March 16, 2017 Yes, the problem might be the file type predictor. I've written you from the support email requesting more info. Thanks! Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 Hi, can you share solution for this problem? I have infinite loadin on plastic GUI -> pending changes and log says 2019-03-11 10:37:22,649 DEBUG BufferPool - -> Entering with name FileTypePredictor (5) 2019-03-11 10:37:22,649 DEBUG BufferPool - Waiting for FileTypePredictor:0 ms 2019-03-11 10:37:22,649 DEBUG BufferPool - <- Releasing with name FileTypePredictor (4) 2019-03-11 10:37:22,649 DEBUG BufferPool - -> Entering with name FileTypePredictor (5) 2019-03-11 10:37:22,649 DEBUG BufferPool - Waiting for FileTypePredictor:0 ms 2019-03-11 10:37:22,655 DEBUG BufferPool - <- Releasing with name FileTypePredictor (4) (almost 1kk lines of log now). I'm able to check-in from cmd but when I need to make more "advanced" operation than simply checkin all it's taking very long time compared with Plastic GUI client. Link to comment Share on other sites More sharing options...
danipen Posted March 11, 2019 Report Share Posted March 11, 2019 @Yerendi, @ironbelly can you send us your logs to try to understand you scenario? Thanks in advance. Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 @danipen, can you give me email adress? Link to comment Share on other sites More sharing options...
danipen Posted March 11, 2019 Report Share Posted March 11, 2019 7 minutes ago, Yerendi said: @danipen, can you give me email adress? Sure! You can send them to support@codicesoftware.com Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 Ok, done! Link to comment Share on other sites More sharing options...
danipen Posted March 11, 2019 Report Share Posted March 11, 2019 It seems we have some kind of problem with the FileTypePredictor in pretty big workspaces. We're trying to understand, locate and fix the issue. In the meantime, you can fine tune your plastic installation options to improve the pending changes performance. The config files are located in the C:\Users\<user>\AppData\Local\plastic4 folder: * do not display private items in the pending changes view: Edit the guiclient.conf <CommitShowPrivates>false</CommitShowPrivates> * do not detect manually moved files: Edit the guiclient.conf <CommitCalculateManuallyMoved>false</CommitCalculateManuallyMoved> * compare file contents using the timestamp instead the file content: This may help to improve the pending changes view perf. Open the client.conf file and setup the following line: <CheckFileContentForChanged>no</CheckFileContentForChanged> Please, share with us if this configuration helps to improve the pending changes time. Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 Thanks for response. Sadly, I've already did all of these and it didn't help. Its not really about "improve the pending changes time" because I've left pending changes window for night and nothing happend, just 10GB of DEBUG BufferPool. Link to comment Share on other sites More sharing options...
danipen Posted March 11, 2019 Report Share Posted March 11, 2019 That doesn't sound good 😕 Does the CLI has the same behavior? Try to execute: $ cm status Also, you can follow some extra steps that helped other users to solve pending changes perf issues: There are several reasons that can probably affect to that operation: The hard drive performance is not good as expected. Do you have a mechanical hard drive? Is there any other process causing a high disk usage? An antivirus software is running. Disable your antivirus to reduce the disk usage. Very often antivirus software running on developer machines slow down all version control operations and turn a powerful computer into a nearly unusable state. Double check if the antivirus is eating CPU or disk IO while pending changes is working. Windows superfetch service. The superfetch service has been identified as a potential cause of disk performance issues. Stopping this service may have a good effect on your computer's performance. Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 Disk usage is 0%. cm status --all shows pending changes properly. Also, Gluon shows pending changes properly. It seems like only PlastucGUI has problem with pending changes tab. And this problem only occures on my biggest workspace, other ones (much smaller) works perfectly even on PlasticGUI. Link to comment Share on other sites More sharing options...
danipen Posted March 11, 2019 Report Share Posted March 11, 2019 Thank you very much @Yerendi for that information. Can you please share your workpace size (number of files/dirs, size on disk, aprox tree deph, etc ...). Thanks! Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 It's ~45 GB, but 15GB is marked as ignored. With ignored its ~700k files, not ignored is ~90k. Tree is quite simple, sth like main -> task -> subtasks, so I would say deph 3/4 is max. Link to comment Share on other sites More sharing options...
danipen Posted March 11, 2019 Report Share Posted March 11, 2019 @Yerendi thanks for the info. I'll try to reproduce your issue. With the depth I meant the depth of the directory structure (distance from the root dir to the deepest leaf (aprox). Link to comment Share on other sites More sharing options...
Yerendi Posted March 11, 2019 Report Share Posted March 11, 2019 Max 6 (in edge cases probably can be more but I didn't find any), in most cases it's folder/folder/folder/folder/target and folder/folder/folder/target. Almost all (99.9%, including ignored ones) are at least folder/target (root/"2nd root"/all files). If it makes any difference I have few big binary files (~100 mb). Edit: If it helps - it's Unity project and Plastic is set in Project folder (so in folder with Assets/Library/ProjectSettings folders). Link to comment Share on other sites More sharing options...
danipen Posted March 12, 2019 Report Share Posted March 12, 2019 @Yerendi still looking your case ... Just for discard things, As you have several ignored items, did you disable ignored files in the pending changes view? <ShowIgnoredItems>false</ShowIgnoredItems> I still trying to reproduce your case. Does your pending changes view get stopped in the "Filling grid" phase? Link to comment Share on other sites More sharing options...
Yerendi Posted March 12, 2019 Report Share Posted March 12, 2019 Yes, ignored changes are disabled. Pending changes view get stopped in the "Finding changes in workspace..." phase - https://gyazo.com/6e49b96ed39303e33c8e7035b031d3f1 Link to comment Share on other sites More sharing options...
Yerendi Posted March 12, 2019 Report Share Posted March 12, 2019 Ignored changes was set to false in config file but not in Options! Pending changes now works great! After rechecking cfg file I've checked it in PlasticGUI (Pending changes -> Options). In "What to show" tab "Show ignored changes" was checked. Dunno if it's bug that toggle in options overrides value in cfg but now its working! Link to comment Share on other sites More sharing options...
danipen Posted March 12, 2019 Report Share Posted March 12, 2019 Great to hear that! Anyway, it seems that it's something that we can improve when searching ignored items... Thanks for the feedback! Link to comment Share on other sites More sharing options...
Yerendi Posted March 12, 2019 Report Share Posted March 12, 2019 I think that repro is simple - I've marked this toggle in options few days ago and pending changes get destroyed. Maybe that when pending changes are "destroyed" Options window values overrides cfg file or sth like that. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.