Access/updating problems and inconsistencies when using cloaked.conf

[Wolfram]

Hiho.

We're trying to use a workflow where users should start off with a workspace containing only parts of the repository, but can update to a full copy if they wish.

From what I found out, using Gluon is probably the way to go - although there's still a severe usability issue in our case. But I'll post that in a separate thread.

 

However, I did a few tests to see if cloaked.conf could help us achieve what we want.

But it seems it cannot, as using cloaked.conf appears to have several severe drawbacks, which prevents it from being of any use in a working environment in most cases.

 

In our scenario we have a Unity project with many (>=100) 3D models, which usually consist of the model itself (an .fbx file), and several/many texture files for each one. The problem with the textures is, it can take VERY long for Unity to import them after an Update. So the idea is to have a workspace that only shows/contains the .fbx files, to be able to get a quick overview, but if the user wants to he can download the associated textures from the repository into his workspace.

I tried to accomplish this using cloaked.conf, but any variant I tried seems unusable:

- Just adding "*.jpg" to cloaked.conf will work for existing assets, BUT it will prevent any future *.jpg that is added from showing up in Pending Changes (so it can never be committed!). As I understand, the workaround is to temporarily remove the cloaked.conf entry, which makes new (and changed) files appear in the Pending Changes. However, even then it's rather complicated to have our modellers open a text editor to manually aund temporarily modify this file (or remember to do this via the right-click context menu). In the real world, this will get forgotten (which results in textures not being committed, and this missing from the repo), or the modified cloaked.conf will be accidentally be committed as well without noticing, forcing all users that update to involuntarily download all textures - which is exactly what we're trying to prevent. Or they will forget they have a locally modified cloaked.conf - which also forces a download of all previously cloaked textures on the next update. With Unity, having the *.meta-Files in sync with the main assets is also vital. In the real world, people will forget to also add/remove the *.meta-Files from cloak, leading to problems that are difficult to trace, and which can cause dangerous inconsistencies (=multiple IDs for the same object, lost references, ...)
- Not adding a catch-all "*.jpg" to cloaked.conf, but instead commit the textures normally, and only then add them explicitly (or their directory, which is usually unique per model and contains all relevant textures) seems another way to go. However, again, people will - per definition - forget they have to modify cloaked.conf as well after changing/adding a model. Also, this approach seems to require TWO commits per change: first, commit the model with all textures, THEN modify cloaked.conf (as the new textures cannot be committed if they are already covered by cloaked.conf).
- More importantly, it seems impossible to locally update or modify any assets that are listed in cloaked.conf - at all. I couldn't even get the "cm.exe upd --cloaked" workaround to do that. And here, even the "locally modified cloaked.conf" is not working - even when removing these files from cloaked.conf, they are still invisible for updates.

There were additional problems with the cloaked.conf workflow, as for example that suddenly the built-in diff was unable to correctly display certain differences, even when diffing existing changesets in the branch explorer.

Are we missing something? Is there a workflow using cloaked.conf where we can achieve what we want? And what use is the claoking option if those files can never, ever, be updated again?

Ideally, a global catch-all should prevent downloading/updating the specified files, but should of course enable us to ADD more files of that type to the repository, and FORCE an update of any repository changes to get the local workspace...well, up-to-date.

EDIT:

The docs at https://www.plasticscm.com/documentation/gui/plastic-scm-version-control-gui-guide.shtml#ColumnsintheItemsView say "An Update command bypasses the item (unless you explicitly selected it when issuing the update)."

How/where can these "explicitly selected"? I was not able to do that. And as mentioned above, the CLI-workaround "cm.exe upd --cloaked myDirectory" doesn't work (as in, it doesn't do anything different than issueing that command without the "--cloaked" parameter).

Thanks a lot!
Cheers,
Wolfram

[psantosl]

> Just adding "*.jpg" to cloaked.conf will work for existing assets,
> BUT it will prevent any future *.jpg that is added from showing up
> in Pending Changes (so it can never be committed!). 

[pablo] Well, yes, just to make it crystal clear, the goal of cloaked is:

* Don't download stuff in cloaked.conf (great for avoiding big stuff).
* If you have something in your workspace already, don't bother updating it.

So, it is not a bug or anything, it is *meant* to work this way.

> As I understand, the workaround is to temporarily remove the cloaked.conf
> entry, which makes new (and changed) files appear in the Pending Changes.
> However, even then it's rather complicated to have our modellers open a
> text editor to manually aund temporarily modify this file (or remember to
> do this via the right-click context menu).

[pablo] Yes, editing cloaked.conf this way is definitely crazy for modellers (or anyone). Not meant to be used this way, definitely.

> In the real world, this will get forgotten (which results
> in textures not being committed, and this missing from the repo),
> or the modified cloaked.conf will be accidentally be committed
> as well without noticing, forcing all users that update to
> involuntarily download all textures - which is exactly what
> we're trying to prevent. Or they will forget they have a locally
> modified cloaked.conf - which also forces a download of all
> previously cloaked textures on the next update.

[pablo] Yes, I agree.

> With Unity, having the *.meta-Files in sync with the main
> assets is also vital.

[pablo] Not sure if you knew this, but when you checkin, Gluon always warns you if the associated meta is not selected. Not sure if it is related, but thought it was good to mention. Unity plugin also does it, but we built this into Gluon too.

> In the real world, people will forget to also add/remove
> the *.meta-Files from cloak, leading to problems that are
> difficult to trace, and which can cause dangerous
> inconsistencies (=multiple IDs for the same object,
> lost references, ...)

[pablo] Agreed.

> - Not adding a catch-all "*.jpg" to cloaked.conf, but instead
> commit the textures normally, and only then add them explicitly
> (or their directory, which is usually unique per model and
> contains all relevant textures) seems another way to go.
> However, again, people will - per definition - forget they have
> to modify cloaked.conf as well after changing/adding a model.
> Also, this approach seems to require TWO commits per change:
> first, commit the model with all textures, THEN modify cloaked.conf
> (as the new textures cannot be committed if they are already covered by cloaked.conf).

[pablo] Yes.

> - More importantly, it seems impossible to locally update or modify
> any assets that are listed in cloaked.conf - at all. I couldn't
> even get the "cm.exe upd --cloaked" workaround to do that.
> And here, even the "locally modified cloaked.conf" is not
> working - even when removing these files from cloaked.conf,
> they are still invisible for updates.

[pablo] I don't follow this last part: I will double check if cm update --cloaked is broken, but it sounds weird.

Finally:

* Definitely cloaked.conf is not meant to be continuously edited this way.
* It "could" be somehow automated with triggers... but if you really need that, you are better served building your own tool on top of Plastic and doing all the cloaking/uncloaking behind the scenes (we have seen this several times by tools teams in big game studios, building their on thing on top of the plastic "cm partial" - telltale did that, to mention one).
* We also have something called "transformable workspaces" https://www.plasticscm.com/documentation/administration/plastic-scm-version-control-administrator-guide.shtml?term=transformable#Chapter17:Transformableworkspaces which could help, but again, it is more something you configure once (or often, but not continuously).
* So, I think you are better served by Gluon + configuration - we can probably add what you need (what we mentioned in http://www.plasticscm.net/topic/20563-selectingdeselecting-many-directories-with-gluon/?tab=comments#comment-38151) or, at last resort, some sort of script on top of "cm partial" to do what you need (like a small program or something).

Thanks!

pablo

[Wolfram]

Hi pablo!

Thank you for your detailed answers!

> [pablo] Not sure if you knew this, but when you checkin, Gluon always warns you if the associated meta is not selected. Not sure if it is related, but thought it was good to mention. Unity plugin also does it, but we built this into Gluon too.

Ah, that's good to know. So far we never really used Gluon, and we do have the plugin active, but do everything else directly in the client.

>> - More importantly, it seems impossible to locally update or modify
>> any assets that are listed in cloaked.conf - at all. I couldn't
>> even get the "cm.exe upd --cloaked" workaround to do that.
>> And here, even the "locally modified cloaked.conf" is not
>> working - even when removing these files from cloaked.conf,
>> they are still invisible for updates.
> 
> [pablo] I don't follow this last part: I will double check if cm update --cloaked is broken, but it sounds weird.

Hm, maybe I'm using it wrong. I tested it with files added to a cloaked directory, as well as explicitly adding a file from a 2nd workspace copy, and only then setting it to cloaked via that workspace, and committing this modified cloaked.conf
Then I used the client to update the first workspace. As expected, the file was not downloaded, as it's cloaked.
When I now issue a "cm update ." in the first workspace, I expect nothing to change. But when I say "cm update --cloaked .", I'd expect the new-but-cloaked file to be force-downloaded - which does not happen.

Or does this only work with CHANGED files, not with new files? I do get a "Could not get revision information" error when I explicitly specify the file (which does not exist in workspace 1) by "cm update --cloaked Assets/Textures/Smileys/smiley_cool_test.png"

> * It "could" be somehow automated with triggers... but if you really need that, you are better served building your own tool on top of Plastic and doing all the cloaking/uncloaking behind the scenes (we have seen this several times by tools teams in big game studios, building their on thing on top of the plastic "cm partial" - telltale did that, to mention one).

Ah, interesting idea! But I guess it's too much hassle in our case - it sounds simpler to rather have the artists live with the longer (and generally one-time, anyway) import times in this case. Especially from a developer point of view... x-D *cough*

> * We also have something called "transformable workspaces" https://www.plasticscm.com/documentation/administration/plastic-scm-version-control-administrator-guide.shtml?term=transformable#Chapter17:Transformableworkspaces which could help, but again, it is more something you configure once (or often, but not continuously).

Very nice, I wasn't aware of this feature!
I just tested it and it seems it, in principle, actually does what we want, and even seems to check "rm" commands BEFORE downloading anything from the server.
Unfortunately, it's not really usable for our artists at this time, as it must be edited by hand, is not in source control, and can't do wildcards or relative pathes (so every path for every model would have to be added manually).

> * So, I think you are better served by Gluon + configuration - we can probably add what you need (what we mentioned in http://www.plasticscm.net/topic/20563-selectingdeselecting-many-directories-with-gluon/?tab=comments#comment-38151) or, at last resort, some sort of script on top of "cm partial" to do what you need (like a small program or something).

That sounds very promising, thanks!
And I might also have a closer look at "cm partial" as well. So far we didn't have the need to use the CLI, but it's of course much more powerful than "just" the GUI/client.
 

[calbzam]

Hi, We have fixed the following bug:

[Bug] Plastic SCM 7.0.16.2242
The cm update --cloaked was not working. The --cloaked option was ignored. Fixed.

Regards,

Carlos