Problem committed when one folder in repo is hidden to some users

[ironbelly]

We have a repo that is divided up via 2 folder trees for Unity and Unreal something like this

 ROOT

     Unity           Unreal 4

1 2 3              1 2 3 

11 222 33             11 222 33

 

In this repo we have 2 groups of users - Unity and Unreal

The Unity group only has read/write perms to the Unity side of the tree while the Unreal group only to the Unreal side of the tree.  This is so our Unreal devs don't have to download 30GB of Unity content and vice versa.  We have a single repo because our QA people and our managers need access to both sides of the tree to do their job and it is much more convenient for them to go to one repo, do one update and be able to push to one place.

 

Here's the issue at hand:

1. Dev in Group Unity pushes a bunch of updates
2. Dev in Unreal group does a bunch of pulls, because he has no read access to the Unity side it updates him to the latest changeset but doesn't pull anything down(this is the desired behavior)
3. Dev in Unreal group does a bunch of work and then goes to commit.  While he was working a dev in the Unity group pushed 2 more changes
4. Dev in Unreal group goes to push and is told he is out of date and has to do a merge before he can commit.
5. Dev in Unreal group doesn't have read access to Unity side of things so he can't merge or do anything
6. Only option we found to allow the Unreal dev to do anything was to cherry pick those 2 changesets for what he does have acess to and then merge
7. This results in those 2 changes being erased essentially as he can only cherry pick what he can read

My question is how do we overcome this challenge in this situation or any situation where one person might not have read access to somethingr that someone else commits but still needs to perform a merge to get their stuff in?

[psantosl]

Hi,

I was about to answer but I need some clarification first:

* Pull => you mean update, right? Pull in plastic involves multiple repos and pushing/pulling from one another. Correct?

* Push => you mean checkin, right? (Again, push is about distributed operation).

Thanks,

 

pablo

[psantosl]

Hi,

The best solution for this problem is:

1. Split the project in two repositories. Repos in Plastic are light and easy to use.

2. Managers and QA: third repo with Xlinks to the other two.

Problem solved.

 

== Second solution ==

* Each team works in branches.

* An integrator with access to the entire repo merges to main.

 

== Third solution ==

* Teams work in branches.

* The team is responsible to merge their own branches => just have to switch to main and merge from their branches.

 

== Long explanation ==

Now, what can we do to solve a problem like the one you posted originally without having to ask you to split in 2 repos: the main issue here is merge tracking.

=== Merge tracking in Plastic ===

Merge tracking in Plastic is changeset based, not file based. This is similar to what Git does, but radically different than what Perforce does.

What does it mean?

When you merge a changeset, you merge all or nothing. You can't merge just a few files and left others for a future merge.

It wasn't like that for years. Major versions 1, 2 and 3 had per-file merge tracking. It was more flexible (to some extent) but it had a major drawback: performance and complexity.

With per-file merge tracking, each time you merge two branches, Plastic had to retrieve and walk the version tree of each file, finding the path between the two contributors involved in merge, calculating the common ancestor (graph walking, pondering multiple possible paths, etc) and finally launching the merge. It means that if you had 1000 files involved in merge, Plastic had to calculate the common ancestors 1000 times. It didn't scale.

Later, around 2011, in version 4, we changed merge tracking to be changeset oriented. Since then, each time you merge you calculate the common ancestor just once walking the cset tree, independently of the number of files involved. No worries if you had 500k files involved in the merge, constant time, which was huge compared to 3.0 where you had to do ops for each of the 500k files involved, and that took forever (despite of the many optimizations we made).

We lost some flexibility but we gained performance and simplicity, and then Plastic learned to handle much more complex cases well, like divergent moves, cyclic moves and many others. And, to be honest, all the flexibility provided by the old 3.0 wasn't really widely appreciated since it came with greater complexity on the user side too.

=== Ok, but what has all that to do with your path-based security? ==

Again, is all about the merge tracking. You are about to do a checkin, but someone else already checked in, so you need to solve a changeset level conflict. Most likely the files are not even in conflict at all, but the tree structure needs to be merged.

You can see /unity but you can't see /unreal. Ok, but in order to complete the new "tree" you need to add changes from both places, even if you don't see them.

Merge, at this point, is made on the client side, and there are lots of complex cases with directory moves that it has to handle. So, at this point, if part of the tree that needs to be "merged" (added, deleted, changes inside it, whatever) can't be loaded... the merge is aborted.

 

== Enter Gluon ==

We created Gluon precisely to handle cases like this: you work on a single branch, you touch just certain parts of the tree, Gluon lets you load just those parts and checkin with ease.

Gluon is created for artists in game development, and the 3 key principles we always considered during development were: I don't want to see a merge or a branch at all, I need to load just part of the repo and checkin correctly, I can't download the whole tree because it is huge.

Gluon works single branch, no merges, but it can solve scenarios like the ones you mentioned.

 

== The road ahead ==

It seems there is no hard limit to actually be able to merge changesets when their underlying trees do not collide. We don't do it today because it wasn't super high prio so far, and because it means some complex changes in the merge process that right now heavily relies on the client side to solve super complex directory move scenarios.

But we could certainly add something to simply solve the conflicts, download the tree metadata (even without data because you can't see the full tree due to permission restrictions), and submit it so that the new tree can be created without loosing parts of its directory structure (right now the reason to avoid merge is that part of the tree will be missing on the client side, so there is no way to load the new metadata underneath.

 

In short: we will eventually improve the ability to merge "subtrees" provided the final one is coherent, but we are not yet there. Now we need access to the full origin-of-the-merge tree to do the merge.

[ironbelly]

Thanks for the thorough answer, I will be coming back and re-reading this and sending our admins here for some background info.. It is very interesting to know.  

Now let me complicate the situation even more.  In the situation above:

 ROOT

     Unity           Unreal 4

1 2 3              1 2 3 

11 222 33             11 222 33

You notice that we have the 3rd layer of folders 1, 2 3 in the example.  As we've grown we end up with separate teams working on each of those.  So each of those represents a different and slightly independently developed, but still related, Unity project that fall under the large umbrella.. There are cases where assets will be shared between 1 and 3 but maybe not between 2 and 3, or sometimes between 4 and 12, etc etc, so it is important that they are kept together.

Reading through your reply I thought that I might need to change the way I look at version control and be far more liberal with creating repos.  They are light and easy as you mentioned so why not have a separate repo for each project but then I went through the list of projects in my head and wanted to come back here to voice my concern.  In our current setup we have a Unity and Unreal version of all of our internal projects.  We have 50-60 internal projects being worked on by 5-10 teams, meaning we would need 120 repos, all of which would get xlinked up.  Overall the setup and maintenance of 120 repos, xlinking them for our team of 4-5 managers, adding and removing people, etc etc seems overwhelming when compared to the way we are doing things now

The next way is that each project is it's own branch, but again with 120 branches, staying on top of and merging changes becomes a full time job and the chances of updates falling through the cracks goes up.

At the end of the day from what I know about Gluon, that seems to be our ticket and not just for artists.. We have programmers who work on their own in their own corner of a project that would appreciate not going through all of the hoops described above. I will task our team with switching as many people over to gluon this week and if I have any more questions will come back..  Thanks again for the reply