GitSync failing at various stages

[Timo_Alta]

I am running into issues when ever I perform a GitSync for any projects larger than a few files. I have successfully pushed a test project from plastic to git which contained 1 file, but my other project which is about 500MB has failed at various stages of the sync command. 

I am trying to setup a CI pipeline similar to the one mentioned in this blog: http://blog.plasticscm.com/2019/03/connect-unity-cloud-build-to-plastic-cloud.html (except replace unity build with Gitlab CI)

The command I am using is 

cm sync PLASTIC_REPO git git@GIT_SERVER:REPO.git

It used to fail consistently on this 

Uploading... ... - 11.02 MB/117.43 MBError: The pipe has been ended.

However I managed to make it progress past that point by pushing a single file on the git side and then running the sync again, now its always failing on this

Uploading... ... / 117.43 MB/117.43 MBError: There was a problem unpacking: index-pack abnormal exit

I have attached 2 files that correspond to the error logs that i found in the cm.log.txt

 

Bonus Question:

How well does GitSync handle larger repos with a longer history? The actual repository I want to sync is above 25GB with over 15K changesets

error1.txt error2.txt

[calbzam]

1)  Have you considered using GitServer instead of GitSync? GitServer allows to configure a Plastic server so it understands the git protocol. This way, you should be able to use any git plugin connected with a Plastic repo.

https://www.plasticscm.com/documentation/gitserver/plastic-scm-version-control-gitserver-guide

2) Regarding your GitSync issue, you are not using Git LFS, right? Becayse gitsync doesn't support it. I'm guessing if GitLab is somehow preventing the sync to happen for this commit.

Does this commit involves big files? If you share the repo with us, we can run the same test in order to reproduce and debug it.

- I've found different entries when googling for the git "There was a problem unpacking: index-pack abnormal exit" error. The proposed solutions are very different but some of them could fit with your scenario:

2.1. 
https://stackoverflow.com/questions/35629118/git-push-error-unpack-failed-index-pack-abnormal-exit

I know this is an old question, but I guess many people might still experience this issue. In my case, this has happened every single time I tried to push slightly bigger commit (not just a few staged hunks, but also multiple new directories and files) to GitLab via SSH (HTTPS never allowed me to even checkout at all on GitLab, even though BitBucket, GitHub and our own GitBlit worked fine) using Windows build of SourceTree. Lost hours messing with repack and compression settings, trying different versions of GIT and SourceTree, but it never worked.

Strangely enough, all I had to do was to switch SourceTree's SSH Client from Putty to OpenSSH, run ssh-keygen in its terminal, and upload the new public key to GitLab (and probably add it back to SourceTree in the Tools menu). The push went through immediately.

2.2.
According to this other thread, it could also be a network related issue. If you repeat the operation are you getting the same error at the same point?
https://stackoverflow.com/questions/21277806/fatal-early-eof-fatal-index-pack-failed

I faced this issue for 2-3 day when i was trying to clone from VPN. in my case issue was network bandwidth. i fixed by cloning in high speed network.

2.3.
Or Memory-related issue:

This error may occur for memory needs of git. You can add these lines to your global git configuration file, which is .gitconfig in $USER_HOME, in order to fix that problem.

[core]
packedGitLimit = 512m
packedGitWindowSize = 512m
[pack]
deltaCacheSize = 2047m
packSizeLimit = 2047m
windowMemory = 2047m

 

Regards,

Carlos,

 

[Timo_Alta]

1. I have indeed been trying out gitserver. Just took forever to generate the git mappings for our main repo so was trying out the gitsync while i was waiting for that. Two questions in regards to gitserver:

1A. How does gitserver handle large files? This is a game repo and as such it has plenty of binary files, we stayed away from git as i know it inflates the repo size beyond useable very quickly (kinda the main reason we chose plastic). So will gitserver encounter the same problem or is there some magic happening?

1B. Isnt gitserver the wrong way around for a Gitlab CI? As far as I know it cant monitor a remote repo for changes to trigger a pipeline, I might give jenkins a look as Ive seen there is a plastic plugin for that, might save me the entire git hassle all together, let me know if you have any words of advice in this area.

2. Thank you for these links, I will go through these later today or tomorrow and will post a link to my repo if I cant get it to work

[calbzam]

Quote

1A. How does gitserver handle large files? This is a game repo and as such it has plenty of binary files, we stayed away from git as i know it inflates the repo size beyond useable very quickly (kinda the main reason we chose plastic). So will gitserver encounter the same problem or is there some magic happening?

From the Plastic point of view, we are not aware of any issue using Gitserver with repos with big files

Quote

1B. Isnt gitserver the wrong way around for a Gitlab CI? As far as I know it cant monitor a remote repo for changes to trigger a pipeline, I might give jenkins a look as Ive seen there is a plastic plugin for that, might save me the entire git hassle all together, let me know if you have any words of advice in this area.

- You can sync the Plastic repo via Gitsync and configure this synced repo in the GitLab CI but I think you should be able to directly point the Gitlab CI to the Platic repo via Gitserver. When you enable GitServer, the Plastic server understands the git protocol so GitLab shouldn't ever know that it's actually a Plastic repo. For instance you can run a "git clone" agains the Plastic server.

- We have multiple customers using the Jenkins plugin for Plastic. If it fits your needs, I think it's the easiest solution.

 

[Timo_Alta]

2.1 How do i change the default ssh client plastic tries to use when performing a gitsync?

2.2 I've tried this multiple times and I would consistently get the errors at the same point, so I doubt its network related

2.3 I tried again with the new global settings and unfortunately got the same issues

 

Tbh I think at this point I will just give Jenkins a go. Initially stayed clear of it, as it seems outdated compared to gitlab, but the plastic plugin would help alot.

[calbzam]

Hi,

Regarding the ssh paht, from the Plastic SCm release notes:

 

GitSync: Added support to sync with Git repositories using SSH protocol. **Important note**: To use this feature you must have an SSH command line client (openssh or a clone) in your PATH environment variable and properly configured. Check "SSH client configuration" note for more info.

SSH CLIENT CONFIGURATION
You should have the command line SSH client "ssh" in your PATH environment variable.
Copy your private key (usually the file called "id_rsa") to your ssh config dir. It should be the ".ssh" dir under your user home directory
If you are under windows, you should set the environment variable "HOME", with the path to your user directory (e.g.: C:\Users\myuser).
If you are under linux, do not forget to set proper permissions to the private key: $ chmod 600 /home/myuser/.ssh/id_rsa
Use the Gitsync in the same way as you normally do with http protocol (using plastic GUI or CLI), but specifying the SSH clone URL. (e.g.: $ cm sync rep2 git git@bitbucket.org:myuser/myrepo.git --user=myuser --pwd=mypass)
If the private key is protected by a passphrase, you will be requested to write it.
TROUBLESHOOTING
Problem: The ssh server is not in port 22, or I need to use different private keys for different servers.
Solution: You can create a "config" files under your ".ssh" directory specifying different settings for differents server. This is an example of ssh "config" file:
Host gitolite.elegosoft.com
User gitolite
Port 2222
IdentityFile ~/.ssh/id_rsa

Host github.com
User git
IdentityFile ~/.ssh/github_key

Host bitbucket.org
IdentityFile ~/.ssh/id_rsa

Problem: I get "An error occurred while connecting to Git".
Solution: It might be an error in the configuration of the ssh client. You can try gitsync using the command line in the following way to get more info:
$ cm sync rep2 git git@bitbucket.org:myuser/myrepo.git --user=myuser --pwd=mypass

Regards,

Carlos.