Jump to content

Importing TFS project into Plastic


Soho

Recommended Posts

What is the easiest way to import a TFS project into Plastic SCM 4?

I am currently experimenting with pulling a TFS project into Plastic via GIT, but perhaps there is a more direct way. E.g. making a fast-import file from a TFS project?

Link to comment
Share on other sites

So far we are doing well with "git-tfs". We had a lot of problems with a single corrupted file in an old changeset in TFS, but thanks to github and open source we were able to bypass that single file.

Currently our server is busy generating a huge git repository from an old TFS project, but I will post and update with a TFS to Plastic recipe if the final steps goes easy.

Link to comment
Share on other sites

I tried to make a fast-import with git also and it seems like git is also troubled by spaces in paths even though the fast-export is produced by git itself. There are numerous reports regarding this issue on the net. Some of these involves patching git, which is not too difficult given the nature of git being split up into small open source commands.

Since Pablo himself encountered this issue a while ago, maybe there is also a working solution for Plastic?

Link to comment
Share on other sites

It is definitely this issue. Git does not quote replacement commands in the fast-export leaving it ambiguous.

I am currently trying to quote the export with some hackish perl wizardry, but the question is, how will cm fi handle quoted replacement paths?

Link to comment
Share on other sites

Great! I will try it at once. For what it is worth here is my perl-patch:

$ perl -p -e 's/^R (.*?)\/(.*) (\1.*)/R \"$1\/$2\" \"$3\"/' MY-FAST-EXPORT-FILE > MY-PATCHED-FAST-EXPORT-FILE

It assumes that files are not moved to the root directory and that the root directory name is not repeated in a subdirectory.

Link to comment
Share on other sites

Ok new problem. Fast-import now works fine with git, but "cm fi" fails with an Illegal character error.

It appears that it doesn't like my name "Søren" or at least the encoding of the "ø".

Which character encoding does Plastic expect the fast-export file to use?

Link to comment
Share on other sites

Re-encoding didn't solve the problem. I suspect it is something else. Also, git imports the fast-export file just fine as long as the "rename" commands have been quoted. Plastic still claims that there are illegal characters.

I also tried another repository which Plastic imported without errors:

$ cm fi "PK.NO"@192.168.1.62:8084 PK.NO.fast-export-quoted
Repository PK.NO does not exist. It will be created
100 csets. 03 sec importing. 00 sec reading file. 01 sec uploading. 01 sec check
ing in.
200 csets. 09 sec importing. 00 sec reading file. 02 sec uploading. 06 sec check
ing in.
300 csets. 15 sec importing. 00 sec reading file. 05 sec uploading. 08 sec check
ing in.
400 csets. 20 sec importing. 00 sec reading file. 07 sec uploading. 12 sec check
ing in.
500 csets. 23 sec importing. 00 sec reading file. 08 sec uploading. 14 sec check
ing in.
600 csets. 27 sec importing. 00 sec reading file. 09 sec uploading. 17 sec check
ing in.
700 csets. 33 sec importing. 00 sec reading file. 10 sec uploading. 22 sec check
ing in.
PK.NO.fast-export-quoted correctly parsed. 732 commits. 34351 ms
	 master
1 branches
0 modified files found refering to an SHA instead of a mark
5408 modified files
60,6 Mb in blobs. 5306 blobs

However, when I looked at the repository in the Plastic GUI there was not a single changeset even though 732 commits have been added according to the Plastic output.

I also tried using history on a couple of files with no luck. No history.

If I look at the history in Git Extensions, it shows up just as the original TFS history.

Obviously, if I lose my history there is no point in converting the TFS repositories, I could just added the latest checkout, but it would be nice to have the history ported so we don't have to start up TFS, every time we want to go digging in the history of some project.

So right now there are two problems:

* Import fails for one repository with an "illegal character" error (it would have been nice with a more specific error message, like which character found where.

* Import seemingly works for another repository, but the changesets are simply not there.

I am beginning to worry that Plastic cannot import TFS repositories via GIT.

Are there anybody who have conducted a successful TFS project import via GIT?

Link to comment
Share on other sites

Ok, I made a stupid mistake in claiming that no changesets were visible for one of the repositories.

It turned out that the repository was so old that the changeset date-filter didn't show any changesets.

I still have the problem with "Illegal characters" in the other repository though. The problem here is that this fast-export is inconveniently large (about 8 GB), so it is hard to experiment with.

I could however not find anything wrong in the fast-export for the changeset that Plastic complains about and as said about, git doesn't mind importing it.

Link to comment
Share on other sites

* Import fails for one repository with an "illegal character" error (it would have been nice with a more specific error message, like which character found where.

Can you create a -no-data git package and send it to us, I'll try to debug the process and tell you which character is the one that is failing.

Cidico is also facing this problem the task to fix the error is already in our bug tracking system but pending to be scheduled.

* Import seemingly works for another repository, but the changesets are simply not there.

Can you review if the filter is hiding your changesets? By default only the changesets created in the last month are shown.

Link to comment
Share on other sites

New problem: When experimenting with fast-importing I tried this:

Importing a fast-export data file into Plastic with "cm fi".

Deleted the repository in Plastic using the GUI.

Attempted a new import using "cm fi".

Now cm gives me this:

Error: Item has already been added. Key in dictionary: 'tmp' Key being added: 'tmp'

I tried importing using a new and different repository name, but with the same results.

Link to comment
Share on other sites

ok the "Error: Item has already been added. Key in dictionary: 'tmp' Key being added: 'tmp'" seems to be a problem when running "cm" under cygwin. I didn't have this problem earlier today though. If I run "cm" in a "dos" prompt I do not get that error message.

Link to comment
Share on other sites

The illegal character error I get from Plastic is this:

Error processing changeset mark 1611. Parent cset mark: 38. Parent cset: 10
Illegal characters in path.
M 100644 :578 OCS Demo/PS.Billing/Test/TestApplication/Controls/SearchComboBox.resx
M 100644 :1585 OCS Demo/PS.Billing/Test/TestApplication/DataSets/PmsAddressDataSet.cs
M 100644 :174 OCS Demo/PS.Billing/Test/TestApplication/DataSets/PmsAddressDataSet.xsc
M 100644 :1586 OCS Demo/PS.Billing/Test/TestApplication/DataSets/PmsAddressDataSet.xsd
M 100644 :1587 OCS Demo/PS.Billing/Test/TestApplication/DataSets/PmsAddressDataSet.xss
M 100644 :1588 OCS Demo/PS.Billing/Test/TestApplication/DataSets/PmsAddressDataSet1.Designer.cs
M 100644 :1589 OCS Demo/PS.Billing/Test/TestApplication/Database/IRecipientDb.cs
M 100644 :1590 OCS Demo/PS.Billing/Test/TestApplication/Database/PmsRecipientSearch.cs
M 100644 :1591 OCS Demo/PS.Billing/Test/TestApplication/Form1.Designer.cs
M 100644 :1592 OCS Demo/PS.Billing/Test/TestApplication/Form1.cs
M 100644 :1593 OCS Demo/PS.Billing/Test/TestApplication/Form1.resx
M 100644 :1594 OCS Demo/PS.Billing/Test/TestApplication/PMSTransform.xslt
M 100644 :1595 OCS Demo/PS.Billing/Test/TestApplication/PmsAddress.xsd
M 100644 :1596 OCS Demo/PS.Billing/Test/TestApplication/PmsAddress.xsx
M 100644 :1597 OCS Demo/PS.Billing/Test/TestApplication/Program.cs
M 100644 :1598 OCS Demo/PS.Billing/Test/TestApplication/Properties/AssemblyInfo.cs
M 100644 :1599 OCS Demo/PS.Billing/Test/TestApplication/Properties/Resources.Designer.cs
M 100644 :17 OCS Demo/PS.Billing/Test/TestApplication/Properties/Resources.resx
M 100644 :1600 OCS Demo/PS.Billing/Test/TestApplication/Properties/Settings.Designer.cs
M 100644 :1601 OCS Demo/PS.Billing/Test/TestApplication/Properties/Settings.settings
M 100644 :1602 OCS Demo/PS.Billing/Test/TestApplication/RecipientFinding/Address.cs
M 100644 :1603 OCS Demo/PS.Billing/Test/TestApplication/RecipientFinding/District.cs
M 100644 :1604 OCS Demo/PS.Billing/Test/TestApplication/RecipientFinding/Levenshtein.cs
M 100644 :1605 OCS Demo/PS.Billing/Test/TestApplication/RecipientFinding/Recipient.cs
M 100644 :1606 OCS Demo/PS.Billing/Test/TestApplication/RecipientFinding/RecipientFinder.cs
M 100644 :1607 OCS Demo/PS.Billing/Test/TestApplication/RecipientFinding/Street.cs
M 100644 :1608 OCS Demo/PS.Billing/Test/TestApplication/RecipientSearch.cs
M 100644 :11 OCS Demo/PS.Billing/Test/TestApplication/TestApplication.csproj.vspscc
M 100644 :1609 OCS Demo/PS.Billing/Test/TestApplication/TestApplication.csproj
M 100644 :1610 OCS Demo/PS.Billing/Test/TestApplication/app.config
Error: Illegal characters in path.

Mark :1611 starts like this:


commit refs/heads/master
mark :1611
author Soren Holstebroe <sh@imagehouse.dk> 1284016721 +0000
committer Soren Holstebroe <sh@imagehouse.dk> 1284016721 +0000
data 145
Branched from $/IHP PS.Billing/Branches/PS.Billing_PostNO

git-tfs-id: [http://tfs.imagehouse.dk/tfs/IHGroup]$/IHP OCS PS.Billing Demo;C33092
from :38
M 100644 :39 OCS Demo/PS.Billing/AllDistributions/AllDistributions.sln
M 100644 :13 OCS Demo/PS.Billing/AllDistributions/AllDistributions.vssscc
M 100644 :40 OCS Demo/PS.Billing/BackupDb.sql

Link to comment
Share on other sites

Unrelated to the problem I am afraid that Plastic does not import utf-8 chars correctly.

I have encoded "author" and "committer" tabs in utf-8 rather than iso-8859-1, but the only difference is that non-English characters shows up with two ?? rather than just ? in the changeset explorer.

Link to comment
Share on other sites

I am getting closer now.

If I am not mistaken, it appears like Plastic fast-import doesn't support quoted paths for file modify "M" commands (possibly also for "D"), however git fast-export will quote M paths if the path contains special characters.

Admitted, the git documentation is unclear about quoting M commands:

"A <path> string must use UNIX-style directory separators (forward slash /), may contain any byte other than LF, and must not start with double quote (").

This could be interpreted as if quotes are banned from M commands, but what it means is that a quote must be escaped if part of the filename, since quotes are allowed to surround the whole M paths. It looks like git itself surrounds M commands with quotes if the contain escapes, and git fast-import supports quoted M commands well.

Thus, Plastic does not follow the git fast-export standard when it comes to M commands. Likely also D commands.

It is possibly that Plastic will accept the path if the surrounding quotes are stripped, but I haven't tried that yet. Also, I am not sure if Plastic supports escaped octals as special characters since this is how git fast-export represent those even though the fast-import doc recommends the use of utf8 chars.

Link to comment
Share on other sites

The perl script I wrote to convert paths:

#!/usr/bin/perl

use utf8;
no utf8;
while (<>)
{
 utf8::encode($_) if (m/^author / or m/^committer /);
# Quote replacements
s/^R (.*?)\/(.*) (\1.*)/R \"$1\/$2\" \"$3\"/;
# Quote M
# s/^M (\d+ :?\w+) ([^\"].*)/M $1 \"$2\"/; # Add quotes if not present
s/^M (\d+ :?\w+) \"(.*)\"/M $1 $2/; # Strip quotes if present
# Quote C (Copy?)
s/^C (.*?)\/(.*) (\1.*)/C \"$1\/$2\" \"$3\"/;
# Quote Deletes
# s/^D ([^\"].*)/D \"$1\"/; # Add quotes if not present
s/^D \"(.*?)\"/D $1/; # Strip quotes if present

print;
}

The git fast-export command I use:

git fast-export --all -C --tag-of-filtered-object=drop --signed-tags=strip --export-marks=fast-export-marks.git | perl QuotePatch.pl > fast-export.dat

Link to comment
Share on other sites

Current status is, that Plastic completed the import. The only problem remaining is that "git fast-export" encodes special characters using backslash-octal notation rather than using UTF8 event though UTF8 is documented in "man git-fast-import" as the preferred encoding for file paths. Actually I cannot find it documented, that the octal notation is used, but this is how git does it.

In summary, this is how git represents paths in fast-export:

* If the path does not contain special characters, it is NOT quoted. Even if the path contains spaces and command R / C is ambigious.

* If the path does contain special characters, the path is quoted and special characters are escaped with backslash. Escapes may include octal encodings like "\330" for Danish / Norwegian "Ø" in iso-8859-1 encoding.

The question is, how does Plastic expect the paths to be encoded?

I hope you find the time to fix these issues Plastic importer. I would be a good selling point for you, if people can convert old TFS repositories seamlessly to Plastic. This is not the case now.

BTW I noticed that the files are correctly encoded if they are checked out from the git repository, so git-tfs is not to be blaimed.

Link to comment
Share on other sites

After a final hack, I made Plastic import the fast-import file with correct filename encoding. Plastic expected UTF8, but git fast-export used octal iso8859.

I modified my perlscript to re-encode octal escapes in paths to UTF8.

After this hack, Plastic finally imported all changesets as expected.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...