How to do "sparse checkouts" (to ignore art folders)

[Jashan Chittesh]

I have read Git sparse checkouts and partial clones, nodata: the secret sauce of lighter clones, Xlinks: Reusable components and Xlinks Guide ... but I'm afraid none of these have answered my question in a satisfying manner.

What we're trying to solve: We have a Unity project repository (centralized, so no replicas or anything distributed and we'd prefer to keep it that way) and want to add a folder for the art team. For the art team, this needs to be one workspace / repository with everything because we don't want to make things complicated for them by requiring two different workspaces or even worse, two different repositories. Some other team members may want to opt in to the art-folder but by default, they don't want to get those updates.

So this very much sounds like what Git sparse checkouts and partial clones wants to offer (and it is in fact trivial to set up with Git sparse checkouts) - except we don't care about changesets at all for this use case. We only want to say "include this folder for me" or "ignore this folder for me" (basically, a workspace setting that locally ignores folders and pretends that they don't exist in the repository). So Xlinks kind of looked like it may solve the issue, even if in a more complicated way than we'd like: We could make a UnityAndArt repository, Xlink into the Unity repository from there and also have the art folder in there. Most people would just use the Unity repository, art would use UnityAndArt.

My concern is that Xlinks seem to be designed to link to a specific changeset in a different repository (which makes sense for the use case it was designed for) but our workflow needs to be "Get latest for everything" or "switch to that other branch of everything" or "switch to that specific changeset for everything" (again, we don't want to have art to jump through additional hoops). In other words, we can't have separate changeset histories and from what I've seen, this doesn't seem to be possible with Xlinks because it's different repositories (because it's actually designed for a different use case). Even worse: It looks like every time art would want to change the version of the Unity project, they'd have to change the Xlink, which is too cumbersome to work.

In Git sparse checkouts and partial clones, there's one interesting paragraph:

Quote

There's one great thing about Git partial clones that we don't have yet, though; they can hydrate a given path inside the repo, while we operate on a changeset basis. This is definitely something we have to add because it is very useful. Instead of hydrating from the root path, you can do it just from a sub path.

While from what I understand, using this (if it is already implemented) would require switching over to a distributed model, it could solve the actual use case we have in a comparatively straightforward manner. That posting is more than 2 years old, so maybe we're lucky?

Or is there another way to solve this in an easy-to-use way?

[calbzam]

Hi,

The most natural way for the artists to work in Plastic if you don't want to use Xlinks is using Gluon GUI. They will be configuring the workspace to download the files they actually need and the rest of the project will be transparent to them. They normally use file locks so they don't need to worry about branching and merging. They can just focus on lock the file and checkin the changes keeping the workflow very simple.

Regards,

Carlos.

[Jashan Chittesh]

Hi Carlos,

I'm sorry but that doesn't even address our use case. Our issue isn't that artists need sparse checkout - because they actually do need to work in the Unity project. Our issue is that we don't want the rest of the team to have to download each binary art update when artists work on their source files. Artists get everything, the rest of the team only gets the Unity project because they don't need to work on the art files (most of the time, some of us may occasionally want those files). So, in our case, everyone would have to use Gluon except for the artists, lol.

I'm honestly a little shocked that something this basic doesn't seem to be implemented, especially when there's a dedicated blog posting claiming that Plastic does it all better than Git when Git sparse checkout does exactly what we need and Plastic apparently just can't do it at all (except with rather cumbersome hacks that require changing our workflows).

Sunny regards,
Jashan

[calbzam]

Hi,

Plastic supports work distributed (using replicas as git does) and also centralized.

- If you work distributed, you can use replica with --nodata to partially clone the repo. You can clone the metadata avoiding to clone the data. This avoids to download big amount of data during the replicas. Git needs to hydrate the local repo to use it; Plastic can work without any local repo data, just retrieving the required blobs on demand from the original server (or a designated alternative). 

- Working centralized also solves the problem, but it is out of scope for Git. What I mean is that it is perfectly fine to work with a working copy directly connected to the central repo. Checkin (commit in Git jargon) and done. No need for the extra push/pull or having a loca repository. And that's solves the problem because you don't need the local or partial clone, and you can tweak the working copy to download only what you need. You can use Gluon to natively configure the files you want to download but you can also use "cloaked.conf" to do the same in the Plastic GUI.

Regards,

Carlos.

[Jashan Chittesh]

Hi, working in Plastic, we use centralized for sake of simplicity. It looks like cloaked.conf actually does what we need, thank you! Maybe you could mention that file in the Git sparse checkouts and partial clones blog posting because that's where people looking for that functionality will usually end up.

[calbzam]

Thanks for the update! We appreaciate your feedback.

[Jashan Chittesh]

On 10/13/2022 at 6:23 PM, calbzam said:

You can use Gluon to natively configure the files you want to download but you can also use "cloaked.conf" to do the same in the Plastic GUI.

I have just tried using the cloaked.conf approach (which seems to be exactly what we needed all along) and it does seem to work well as long as there are no changes in the cloaked files. In other words, once I have set this up, Plastic doesn't download any of the files in the folder that I have added to cloaked.conf. I can switch branches without getting those files, so that's all perfect and as expected.

But as soon as I make a change to one of the files in that folder, Plastic downloads that file the next time I use Update workspace (in pending changes => "1 new changeset available" => View => Update workspace). I have also tried a changeset with changes both in cloaked and non-cloaked folders but I still get the files from the cloaked-folder that way, unfortunately.

I can work around that by using "Switch workspace to this branch" in the branch explorer instead but this basically means that nobody can use Update workspace if we want to use this.

[Jashan Chittesh]

Another issue that I just saw: When removing the folder from the cloaked.conf file, instead of downloading the files next time I refresh in Plastic, the folder is shown as deleted. I can see why this happens and I can get what I want by using Undo ... but it's not a great UX because there is quite a risk that someone accidentally checks in deleting the art folder.

The way we'd use this is to have cloaked.conf having the art folder included by default, so the first thing an artist would do when creating a new workspace is get the workspace, remove that one line from cloaked.conf, then update.

It looks like again, the workaround would be to get the workspace, remove the line from cloaked.conf, then "Switch workspace to this branch" even though we already are on that branch.

[calbzam]

Hi,

This recent fix shouid be useful, right? https://www.plasticscm.com/download/releasenotes/11.0.16.7665

Quote

 

All clients: Incoming Changes respects the cloaked rules.

Before, the Incoming Changes operation downloaded the files/directories although they were under cloaked paths. This was inconsistent with the update operation behavior and with the cloaked semantics themselves.

Now, the changes under cloaked paths are skipped like the update operation does.

The Gluon Incoming changes is not affected by this behavior (cloaked rules don't make sense in Gluon) since the operation works in a different way.

 

 

On 12/20/2022 at 2:18 PM, Jashan Chittesh said:

Another issue that I just saw: When removing the folder from the cloaked.conf file, instead of downloading the files next time I refresh in Plastic, the folder is shown as deleted. I can see why this happens and I can get what I want by using Undo ... but it's not a great UX because there is quite a risk that someone accidentally checks in deleting the art folder.

Can you confirm if this behavior still happens after upgrading to the latest Plastic version? We will evaluate it.

Regards,

Carlos.

[Jashan Chittesh]

Sorry for not getting back to you earlier - after all the troubles with this, we eventually gave up and have kept using Google Drive for the art assets. But it's good to know that this should now be fixed, so we might give it another try at some later point in time.