Teamcity plugin does some crazy stuff

[Mirage]

I've been testing every new version of this plugin on Teamcity 7.x and now 8.0. It has been slowly improving, and all the issues have been fixed except for one (worked with some version half a year ago):

 

[13:45:47]Checkout directory: C:\TeamCity\buildAgent\work\740753f6110932cf

[13:45:47]Updating sources: server side checkout (18m:13s)

[13:45:48][updating sources] Will perform clean checkout. Reason: The project sources on the agent are newer than requested

[13:45:48][updating sources] Building clean patch for VCS root: Doc Alpha

[13:48:17][updating sources] Transferring repository sources: 306.54 MB so far...

[13:49:27][updating sources] Building clean patch for VCS root: PGM Alpha

[13:50:17][updating sources] Transferring repository sources: 670.89 MB so far...

[13:51:24][updating sources] Building clean patch for VCS root: Install Alpha

[13:52:17][updating sources] Transferring repository sources: 1.26 GB so far...

[13:54:19][updating sources] Building clean patch for VCS root: AutoTools Alpha

[13:54:17][updating sources] Transferring repository sources: 2.34 GB so far...

[13:54:30][updating sources] Repository sources transferred: 2.41 GB total

[13:54:30][updating sources] Average transfer speed: 5.01 MB per second

[13:54:32][updating sources] Removing C:\TeamCity\buildAgent\work\740753f6110932cf

[13:57:31][updating sources] Updating C:\TeamCity\buildAgent\work\740753f6110932cf

[14:04:01]Step 1/1: MSBuild

 

We have four repositories. Each one uses its own workspace which is entered manually in the VCS config page. One build configuration has all 4 repositories attached. The plugin updates the workspaces only if I enable VCS checkout mode = Automatically on server (I could disable this in an older version, which updated workspaces only), and we only use the workspaces as the build source. By enabling the auto checkout, Teamcity also updates its own checkout directory. This is where the fun starts:

• It deletes all previously updated sources (Will perform clean checkout)
• Downloads all the repositories into one checkout directory (which is quite useless)
• Goes ahead and deletes all the files (Removing C:\TeamCity\buildAgent\work\740753f6110932cf)
• Downloads all the files from all repositories again (Updating C:\TeamCity\buildAgent\work\740753f6110932cf)
• Does this for every build

As you can see, it all takes good 20 minutes to do, and all we need is to disable automatic checkout, and update workspaces only.

 

Is there a way to workaround this somehow? If not, please, fix this ASAP.

 

Thanks

[calbzam]

Hi @Mirage,

 

I´m going to try reproduce the issue. If the issue persists, we will try to fix it ASAP.

 

Regards,
Carlos

[Mirage]

Hi Carlos,

 

did you manage to reproduce the issue? Do you plan to fix it?

 

Thanks,

Mirage

[calbzam]

Hi,

 

I´m able to reproduce the issue, but I think there is a way to avoid the final delete and update operations.

 

If I go to  Create/Edit Build Configuration> Version Control Settings -->Do not checkout files automatically, I´m getting this log:

 

[17:51:48]Checking for changes

[17:51:51]Publishing internal artifacts

[17:51:51]Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp

[17:51:51]Clean build enabled: removing old files from C:\TeamCity\buildAgent\work\4084d6d0088a3141

[17:51:51]Checkout directory: C:\TeamCity\buildAgent\work\4084d6d0088a3141

[17:51:51]Step 1/1: Command Line (5s)

 

And if I unselect the option: Clean all files before build:

 

[17:47:53]Skip checking for changes - changes are already collected

[17:47:55]Publishing internal artifacts

[17:47:55]Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp

[17:47:55]Checkout directory: C:\TeamCity\buildAgent\work\4084d6d0088a3141

[17:47:55]Step 1/1: Command Line (3s)

 

This way I avoid the last steps:

 

2:59]Skip checking for changes - changes are already collected

[17:23:01]Publishing internal artifacts

[17:23:01]Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp

[17:23:01]Checkout directory: C:\TeamCity\buildAgent\work\747321fb9eba3242

[17:23:01]Updating sources: server side checkout

[17:23:01][updating sources] Will perform clean checkout. Reason: The project sources on the agent are newer than requested

[17:23:01][updating sources] Building clean patch for VCS root: Root2

[17:23:02][updating sources] Repository sources transferred

[17:23:02][updating sources] Removing C:\TeamCity\buildAgent\work\747321fb9eba3242

[17:23:02][updating sources] Updating C:\TeamCity\buildAgent\work\747321fb9eba3242

 

Not sure if it could be a workaround for you to save time in your build process.

 

Regards,

Carlos

[Mirage]

Yes, but this doesn't work for us, because:

- we need to update workspaces outside the checkout directory (for this, your plugin now requires to enable VCS checkout mode = Automatically on server)

- the checkout directory contains all files from all repositories merged into one directory => useless for us

- we already have Clean all files before build disabled

 

We also work with SVN, and none of these issues are present. Only the latest changesets are downloaded.

 

IMHO, this really requires a fix.

 

Thanks,

Mirage

[Mirage]

If you don't know how to fix it, all we need is a checkbox "Update workspaces only" which would work regardless of other VCS settings. That would be a perfect solution for us, and maybe other people.

 

Thanks,

Mirage

[calbzam]

Hi,

 

We have been reviewing the plugin and, as you pointed, we should add  a workaround to update workspaces only and deal with scenarios like yours. We are going to check how to implement it ASAP. Thanks for reporting.

 

Best regards,

Carlos

[Guest]

Hi, this issue is now fixed and available in the latest public release: 4.1.10.493. Get it here: https://www.plasticscm.com/download/editions.en.aspx

[Mirage]

Hi, thank you for your efforts, but it stil is NOT fixed.

 

We installed Plastic 4.1.10.493 and the corresponding Teamcity plugin running Teamcity 8.0.3. This is the result after a few builds (this happens in every build):

 

 

[01:00:33]Checking for changes

[01:01:06]Publishing internal artifacts

[01:01:06][Publishing internal artifacts] Sending build.start.properties.gz file

[01:01:06]Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp

[01:01:06]Checkout directory: C:\TeamCity\buildAgent\work\a8b9fe487aa02d43

[01:01:06]Updating sources: server side checkout (25m:18s)

[01:01:14][updating sources] Will perform clean checkout. Reason: The project sources on the agent are newer than requested

[01:01:14][updating sources] Building and caching clean patch for VCS root: PGM 5_0_02

[01:03:47][updating sources] Building clean patch for VCS root: PGM 5_0_02

[01:05:24][updating sources] Building and caching clean patch for VCS root: Doc 5_0_02

[01:07:48][updating sources] Building clean patch for VCS root: Doc 5_0_02

[01:07:59][updating sources] Transferring repository sources: 539.19 MB so far...

[01:09:05][updating sources] Building and caching clean patch for VCS root: Install 5_0_02

[01:12:12][updating sources] Building clean patch for VCS root: Install 5_0_02

[01:12:28][updating sources] Transferring repository sources: 953.85 MB so far...

[01:14:29][updating sources] Building and caching clean patch for VCS root: AutoTools 5_0_02

[01:14:28][updating sources] Transferring repository sources: 2.08 GB so far...

[01:14:49][updating sources] Building clean patch for VCS root: AutoTools 5_0_02

[01:14:55][updating sources] Repository sources transferred: 2.16 GB total

[01:14:55][updating sources] Average transfer speed: 3.38 MB per second

[01:15:53][updating sources] Removing C:\TeamCity\buildAgent\work\a8b9fe487aa02d43

[01:17:59][updating sources] Updating C:\TeamCity\buildAgent\work\a8b9fe487aa02d43

[01:26:24]Step 1/1: MSBuild (2h:35m:40s)

 

It still takes 20 - 30 minutes.

 

I hope you can reopen the issue and fix it, or provide us with the workaround to update workspaces only.

 

Thanks!

[Olaf Kober]

Hi!

 

I can confirm that it is not fixed. We still see the same problem with latest 4.1.10.493 and TeamCity 8.0.3 on all builds.

Checking for changes
[10:28:39]Publishing internal artifacts
[10:28:39][Publishing internal artifacts] Sending build.start.properties.gz file
[10:28:39]Clearing temporary directory: D:\TEAMCITY\temp\buildTmp
[10:28:39]Checkout directory: D:\TEAMCITY\work\d36c7ecd7ed2b398
[10:28:39]Updating sources: server side checkout (51s)
[10:28:39][Updating sources] Will perform clean checkout. Reason: The project sources on the agent are newer than requested
[10:28:39][Updating sources] Building clean patch for VCS root: Desktop_Platform_main
[10:29:03][Updating sources] Repository sources transferred: 73.71 MB total
[10:29:03][Updating sources] Average transfer speed: 7.27 MB per second
[10:29:03][Updating sources] Removing D:\TEAMCITY\work\d36c7ecd7ed2b398
[10:29:25][Updating sources] Updating D:\TEAMCITY\work\d36c7ecd7ed2b398

+1 Vote for fixing this

[calbzam]

Hi,

 

I´ve tested the fix and the incremental patch is working for me

 

Maybe the plugin we uploaded to our web page is not correct. This is the plugin I´ve used to test:  http://www.2shared.com/file/84HeIA9N/comcodicesoftwarepluginsteamci.html

 

Could you copy this zip file in your "C:\TeamCity\webapps\ROOT\WEB-INF\plugins" folder and restart TeamCity service. Then check if an incremental path is allowed.

[14:04:47]Checking for changes
[14:05:00]Publishing internal artifacts
[14:05:00]Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[14:05:00]Checkout directory: C:\TeamCity\buildAgent\work\a504fd63676631fa
[14:05:00]Updating sources: server side checkout (4s)
[14:05:00][Updating sources] Building incremental patch for VCS root: Plastic1
[14:05:04][Updating sources] Repository sources transferred
[14:05:04][Updating sources] Updating C:\TeamCity\buildAgent\work\a504fd63676631fa

 

PD: In case you can´t download this file, I can send you the plugin directly to your email. I would like to know if it´s working fine before uploading it to our webpage.

 

Thank you, and sorry for the inconveniences.

 

Regards,

Carlos

[Olaf Kober]

Hello!

 

The new plugin works as expected. TeamCity shows plugin version SNAPSHOT-201310221608.

 

 

Thanks for fixing that!

Olaf

[calbzam]

Hi Olaf,

 

Thanks for testing. Now, we are going to update the plugin to our web page

 

Regards,

Carlos

[Mirage]

I can confirm that it is fixed now. Thanks!

[Mirage]

I have some bad news. The reported problem is fixed, but it introduced another more serious problem.

 

It seems that workspaces are not updated anymore during incremental update, only the Teamcity's checkout dirs are updated. After a few days, we noticed that the latest changes aren't included in our builds. We tracked the problem down to your new version of Teamcity plugin (we still use Plastic 4.1).

 

Here is what happens (teamcity-vcs.log):

 

[2013-11-01 15:05:09,217] INFO [-bio-80-exec-48] - jetbrains.buildServer.VCS - Setting build revisions on agent=AUTO-BUILD-V {id=1, host=127.0.0.1:9090, agentTypeId=1, pool=Default}, buildPromotion=BuildPromotion{myId=2137, myPersonal=false, myBuildTypeId='bt2', myModificationId=40979, myRevisions=[buildRevision[myVcsRootEntry=VcsRootEntry[VcsSettings["PGM 5_0_02" {instance id=57, parent internal id=57, parent id=PGM_Alpha, properties: [cmExe=cm; selector=repository "clarity-pgm@plastic-pgm:8087"
path "/"
branch "/main"
checkout "/main"; teamcity:vcsRootName=PGM 5_0_02; vcs=PlasticSCM; workspace=C:\Csw\5_0_02\PGM; ]}; Checkout rules: ]], myRevision=RepositoryVersion{myVersion='cs:31662@rep:clarity-pgm@repserver:plastic-pgm:8087', myDisplayVersion='31662', myVcsBranch='null'}], BuildRevision[myVcsRootEntry=VcsRootEntry[VcsSettings["Doc 5_0_02" {instance id=58, parent internal id=58, parent id=Doc_Alpha, properties: [cmExe=cm; selector=repository "clarity-doc@plastic-doc:8087"
path "/"
branch "/main"
checkout "/main"; teamcity:vcsRootName=Doc 5_0_02; vcs=PlasticSCM; workspace=C:\Csw\5_0_02\Doc; ]}; Checkout rules: ]], myRevision=RepositoryVersion{myVersion='cs:4171@rep:clarity-doc@repserver:plastic-doc:8087', myDisplayVersion='4171', myVcsBranch='null'}], BuildRevision[myVcsRootEntry=VcsRootEntry[VcsSettings["Install 5_0_02" {instance id=59, parent internal id=59, parent id=Install_Alpha, properties: [cmExe=cm; selector=repository "install@plastic-pgm:8087"
path "/"
branch "/main"
checkout "/main"; teamcity:vcsRootName=Install 5_0_02; vcs=PlasticSCM; workspace=C:\Csw\5_0_02\Install; ]}; Checkout rules: ]], myRevision=RepositoryVersion{myVersion='cs:1560@rep:install@repserver:plastic-pgm:8087', myDisplayVersion='1560', myVcsBranch='null'}], BuildRevision[myVcsRootEntry=VcsRootEntry[VcsSettings["AutoTools 5_0_02" {instance id=60, parent internal id=60, parent id=AutoTools_Alpha, properties: [cmExe=cm; selector=repository "autotools@plastic-pgm:8087"
path "/"
branch "/main"
checkout "/main"; teamcity:vcsRootName=AutoTools 5_0_02; vcs=PlasticSCM; workspace=C:\Csw\5_0_02\AutoTools; ]}; Checkout rules: ]], myRevision=RepositoryVersion{myVersion='cs:1259@rep:autotools@repserver:plastic-pgm:8087', myDisplayVersion='1259', myVcsBranch='null'}]], myAssociatedBuildId=1447, myBranch=}, revisions={"Install 5_0_02" {instance id=59, parent internal id=59, parent id=Install_Alpha, description: "repository "install@plastic-pgm:8087"
path "/"
branch "/main"
checkout "/main""}=null, "Doc 5_0_02" {instance id=58, parent internal id=58, parent id=Doc_Alpha, description: "repository "clarity-doc@plastic-doc:8087"
path "/"
branch "/main"
checkout "/main""}=null, "PGM 5_0_02" {instance id=57, parent internal id=57, parent id=PGM_Alpha, description: "repository "clarity-pgm@plastic-pgm:8087"
path "/"
branch "/main"
checkout "/main""}=null, "AutoTools 5_0_02" {instance id=60, parent internal id=60, parent id=AutoTools_Alpha, description: "repository "autotools@plastic-pgm:8087"
path "/"
branch "/main"
checkout "/main""}=null} 

 

Here's one line that interests us (there are 3 more):

 

checkout "/main"; teamcity:vcsRootName=PGM 5_0_02; vcs=PlasticSCM; workspace=C:\Csw\5_0_02\PGM; ]}; Checkout rules: ]], myRevision=RepositoryVersion{myVersion='cs:31662@rep:clarity-pgm@repserver:plastic-pgm:8087', myDisplayVersion='31662', myVcsBranch='null'}], BuildRevision[myVcsRootEntry=VcsRootEntry[VcsSettings["Doc 5_0_02" {instance id=58, parent internal id=58, parent id=Doc_Alpha, properties: [cmExe=cm; selector=repository "clarity-doc@plastic-doc:8087"

 

If you study the log closely, you'll find out that it tries to update a workspace using settings for a different workspace (clarity-pgm vs clarity-doc). The settings seem to be shifted for all repositories. Almost as if the workspaces were indexed and you started indexing from 0 instead of 1 or vice versa (wild guess). The log could be wrong too, of course.

 

All I know is that after the fix, no workspace is updated (we do not care about Teamcity's checkout dirs).

 

Therefore, we reverted to the old plugin and 2-3 people wasted half a day with this.

 

Cheers,

Mirage

[calbzam]

Hi,

 

Wow, I´m going to try reproduce the issue.

 

Regards,

Carlos

[calbzam]

Hi,

 

Regarding the VCS settings, could you tell me your configuration to reproduce the issue?

 

 

Regards,

Carlos

[Mirage]

These are our settings:

 

[Mirage]

Howdy there,

 

was there any progress on the issue? Did you manage to reproduce it or even fix it? Thanks!