GitSync not completing

[ian.fischer.winreality]

We have been trying to use GitSync to work and it will not finish after many attempts with fast-export that kept failing.

We have a very large repository at some 400+ GB, and each time we get to the "Packaging... 0/576226" it just sits there and does not continue without error even after letting it run for a 12+ hours it remains at 0/576226 progress.

Because Git cloud providers do not have plans for repositories that are larger than 400 GB, we planned to first perform the export into a local Git repository, then prune history items, mark items for LFS, and find an external LFS hosting location like AWS to get our repository to an acceptable size.

We tried first using Git daemon to expose a Git URL for GitSync to push to 

git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose --enable=receive-pack

It would then take about 4-5 hours performing the "Exporting..." stage and then get to Packaging and not do anything else.

Looking at the logs in %LOCALAPPDATA%\plastic4\logs\cm.log.txt at the end of the Plastic GitSync "Exporting..." phase it finishes registering all the branches to push, and then the tags, and then it says it purges connection to local (which I am assuming it means it is disconnecting from the local Plastic repo to then send the packed results to the Git repo but not for sure). It then just logs info about it's "BUFFER POOL CONFIG" every 30 minutes for over 12+ hours.

We contacted support who told us that Git daemon was not as performant as a full Git server and that what was going wrong.

We then setup a local hosted GitLab server on a Linux machine with full ssh support. We experimented with smaller Plastic SCM repositories and they all completed without any issues and we were able to view them in the private hosted GitLab browser and all the appropriate history.

When attempting to now do the GitSync to this full Git server the same issue occurred. It still took the same 4-5 hours in it's "Exporting..." phase but hung at 0/576226 progress in the "Packing..." phase and could never get to the "Sending..." phase at all.

• Is there any known reasons why Plastic will not make progress during the "Packaging..." phase?
• What are some alternative solutions that might be better to export our Plastic SCM repo to a Git repo?

[calbzam]

Hi,

I can see you already opened a ticket with our support. Let's not duplicate the conversations. We can handle it wherever you prefer:

If the operation doesn't fail, it seems it should be still running. I would leave it running until there is any error or the sync continues. I'm guessing the initial changeset is very big and it will take a long time to sync it. The follwong changesets to sync should be faster.

50 minutes ago, ian.fischer.winreality said:

Because Git cloud providers do not have plans for repositories that are larger than 400 GB, we planned to first perform the export into a local Git repository, then prune history items, mark items for LFS, and find an external LFS hosting location like AWS to get our repository to an acceptable size.

Why are you migrating the repo to git? It shouldn't be a problem to handle big repos in Plastic. I don't fully understand if your plan is migrating to git or keep using the Palstic repo in the future.

Are you a current Plastic SCM customer? Are you moving to git?

Regards,

Carlos.

[ian.fischer.winreality]

Quote

If the operation doesn't fail, it seems it should be still running. I would leave it running until there is any error or the sync continues. I'm guessing the initial changeset is very big and it will take a long time to sync it. The follwong changesets to sync should be faster.

I can leave it running, but after 17+ hours when would you expect to see some progress before giving up?

Quote

Why are you migrating the repo to git?

We want to checkout and use the Git code review tools. Would you have different advice if we were planning to fully move to Git vs. utilizing both systems?

Quote

Are you a current Plastic SCM customer?

Yes, our primary account email is subscriptions@winreality.com.

Quote

Are you moving to git?

We are unsure. The front end and asset teams know and are fine with Plastic. The backend and code developers want a better code review process and are looking into ways to use other tools.

[calbzam]

On 1/27/2023 at 5:39 PM, ian.fischer.winreality said:

I can leave it running, but after 17+ hours when would you expect to see some progress before giving up?

Did you leave it running during the weekend? Is it still running?

On 1/27/2023 at 5:39 PM, ian.fischer.winreality said:

We want to checkout and use the Git code review tools. Would you have different advice if we were planning to fully move to Git vs. utilizing both systems?

What are you missing code review features? I can share your feedback with the productr team.

Regards,

Carlos.

[ian.fischer.winreality]

We did try letting the operation run. It turned out it took 4.16 hours in "Exporting...", 1 day and 6.83 hours in "Packaging..." until it started to do the "Uploading..." and then got a stdin send error.

- 41086 changesets to push

Receiving references... OK
There are changes to push.
Exporting... OK
Packaging...... OK
remote: g...... | 17.44 GB/152.98 GB
remote: ========================================================================
remote:
remote: stdin send error: EOF
remote:
remote: ========================================================================
remote:
Uploading...... | 17.44 GB/152.98 GB

From that point we have left it running for 2 days and 8 hours but it has not changed anymore.

Quote

What are you missing code review features? I can share your feedback with the productr team.

Enforcement of code review pass by a specific number of specific people/groups before allowing merges into a branch. Better code differences when viewing code reviews. Better notifications when code reviews are added.

 

At this point, what is your advice after it has not finished uploading after 2+ days and that stdin error?

[calbzam]

Hi Ian,

You finally found a workaround via GitServer, right?

Regards,

Carlos.

[Francois Bertrand]

@Carlos 

Hi there! I've been recommending Plastic SCM to all my fellow game developers as I believe it's the best out there after 25 years in the industry.

I knew Plastic had been bought by Unity for a while and I figured integration was a major part of that. However, I just noticed the latest update feature is a "Unity rebrand", and that got me a bit nervous (we use it for custom engines, Unreal and Unity), so I came to the forums.

I don't seem to be able to create a new thread (am I just missing an obvious button? Or have these boards been deprecated?) and the Plastic website is gone, and I'm wondering how much I should be freaked out regarding support for non-Unity stuff.

I might have missed blog posts etc. but would appreciate any info. Thanks!

Francois

[calbzam]

Hi,

On the forum's front page, we have added a red bar explaining that the Plastic SCM forums are now in Unity. If it's not visible enough, your feedback is welcome.

 

Plastic SCM is now Unity Version Control (part of Unity DevOps) so all the purchase process, forums, documentation... is moved to Unity.com.

But there haven't been any other changes regarding the support. We have the same team supporting Unity Version Control and we should answer your questions pretty fast.

Regards,

Carlos.

[Francois Bertrand]

Hi,

Yes I see the big red bar, no feedback there. However I'm not sure if you understand my concern as a Unreal/C++ customer that uses Plastic across his entire team, that the previous website & forums, containing Unreal, Gluon, etc. are all gone? For example:

• I don't know if it is even "legal" to post questions about Unreal in a Unity forum? Will it get moderated out? There only seems to be 1 big "Source control" category anymore.
• What is the new pricing (or the old one if that didn't change)? All I see are "Sign up for free!" or "contact sales!" buttons everywhere (with a "you will be charged xxx after 3 months"), but no clear plans for scaling, cloud vs Hosted, etc. Again, perhaps this is somewhere but MAN have they made it hard to find?!
• Given all this and that all plans are so tightly integrated with Unity, how much will Unreal plug-ins get for support? (a separate support email has been reassuring but I did want to reiterate the concern here)

I understand (and hope) it's might be just slight changes to the current "business as usual", but this customer would've appreciated more Info/reassuring upfront (or "hard truths" if that's the way it's going to go).

[calbzam]

Hi,

We will be adding more subcategories in the new Unity forums for Unreal and other integrations.

For now, it's true that there is one single forum for Unity DevOps Version Control and it involves all Plastic SCM. Including the integration with UE. As you know, we are relating more and more improvements for the UE plugin and this is not going to change.

For Plastic SCM Enterprise/on-premises pricing questions, please reach Unity sales. As far as I know, it hasn't changed for now: https://create.unity.com/contact-unity-expert

All the former Plastic SCM Cloud is now rebranded to Unity DevOps --> Version Control. The purchase process is handled via: https://unity.com/solutions/version-control

You should receive an official notification via email today explaining the rebranding and recent changes. Anyway, I will be happy to also assist here.

Regards,

Carlos.

[Francois Bertrand]

Hello,

Thank you for your follow-up. I'm glad it sounds like things will continue along the same lines they have (which is great).

So I'm not looking for any more answers, but just for the record here is my feedback so far:

• It would have been great to get an official notification/explanation BEFORE the rebranding happens
• Same for setting up forums (couldn't this have been done ahead of time?) not to spook non-Unity customers
• I still can't find any pricing information (it's not affecting us but it's not great for future customers?), the links just go in circles on Unity

Again, glad we can still use Plastic and your support is still great, and I understand Unity will be the priority, and Unity probably led this effort, but this was definitely jarring and definitely has the feel of a large corporation slamming things together.