Jump to content
Marc S

Data usage optimisation.

Recommended Posts

Hi.

When I started using Plastic few weeks ago, I was pleased to see my usage on the cloud server was less that the assets on my hard drive but this has shifted the other way since. I understand plastic also stores previous versions of my files: if I save often update a 1GB psd file for example, it's fair that it uses more than 1GB on the cloud. That said, I know I don't need to keep previous versions of files like lighmaps data and was wondering if there are some optimizations I could use so I don't end up using 3x my local data on the cloud.

Is that possible?

Thanks

Share this post


Link to post
Share on other sites

Hi Marc,

 

Thanks for the suggestion.

 

What you says is:

  • Your working copy size is smaller than repo size. Well, this is normal, as you pointed, the repo contains ALL revisions, while you only have a working copy. Repo stores revisions compressed, that's why initially it weighted less.
  • You'd like to have a way to "discard history and keep only last N revisions" for certain files, correct?

Share this post


Link to post
Share on other sites

Hi Psantosl and thanks for that quick reply.

Yes that is correct.

A coworker and I are currently working on a large game level. The scene file is 130MB and it's lightmap 1.7GB. We re-bake the lighmap once or twice a week and it's only on the cloud server so it's shared on our computers. We don't need versioning on it since it can always be re-baked. The scenes are sent back and forth to each other all the time so they can be uploaded +5 times a day which is more than we need for long term backups. As you can see, this can pile up quickly for minor changes.

The lightmap is easy to deal with. Unity creates a file always called "LightingData.asset" so all we would need is a rule for these files to only keep N revisions as you suggested.

For files like my scene, above a certain weight, it's a bit trickier. A solution would be to have an other rule to discard chosen assets after x days, but because we might need to get it back to a previous version after that date, we could set an exception with a milestone. A milestone would be a "super-changeset" where that rule doesn't apply.

In practice here is how it would go. We're working as usual and set an occasional milestone each week when we're satisfied with our work. At this point we can always go back to any changeset because recent data is kept on it as long as it is not a lightmap. Six month later, a client asks us to revert some changes on that scene we were done with: most of the scenes data was erased after x days to save space but we have the milestones which is enough for what we need.

 

I understand my request might be unusual. I'm working on a big unity project to make multiple mini games that share the same assets so I'm not creating a new repository each time a create a new one. That means the data I'm currently pilling up could follow me for the next years and would cost me a lot on the long run. So while it's not urgent, I hope we can find a solution to avoid hoarding too much data.

Thanks

Share this post


Link to post
Share on other sites
7 hours ago, Marc S said:

A coworker and I are currently working on a large game level. The scene file is 130MB and it's lightmap 1.7GB. We re-bake the lighmap once or twice a week and it's only on the cloud server so it's shared on our computers. We don't need versioning on it since it can always be re-baked. The scenes are sent back and forth to each other all the time so they can be uploaded +5 times a day which is more than we need for long term backups. As you can see, this can pile up quickly for minor changes.

My question is: why do you checkin the lightmap if it can be rebaked? Because it is easier to share it through Plastic SCM than actually using other mechanism, correct?

 

7 hours ago, Marc S said:

The lightmap is easy to deal with. Unity creates a file always called "LightingData.asset" so all we would need is a rule for these files to only keep N revisions as you suggested.

Understood.

 

7 hours ago, Marc S said:

For files like my scene, above a certain weight, it's a bit trickier. A solution would be to have an other rule to discard chosen assets after x days, but because we might need to get it back to a previous version after that date, we could set an exception with a milestone. A milestone would be a "super-changeset" where that rule doesn't apply.

Of course this means you are in single branch, right. How would this work with multiple branches?

The "super changeset" can be simply a labelled one. You know, you can label changesets.

So the rule would be something like:

Keep only 3 versions of LightingData.asset.

Keep only N versions of files > that size except if they are labelled.

 

7 hours ago, Marc S said:

I understand my request might be unusual. I'm working on a big unity project to make multiple mini games that share the same assets so I'm not creating a new repository each time a create a new one. That means the data I'm currently pilling up could follow me for the next years and would cost me a lot on the long run. So while it's not urgent, I hope we can find a solution to avoid hoarding too much data.

I'm actually very interested on this topic, because it makes a lot of sense for us to add extra features for teams using Unity.

 

I'll add more people to the conversation.

Share this post


Link to post
Share on other sites

I think it would be great to be able to optimize for storage, but I think it's also important that the history of a project remains stable. If you remove the data from a changeset, it would be impossible to switch to that changeset and view the project as it was at that point in time. Not all changesets are that useful, though, as evidenced by uploading several alterations of a photoshop document in a single day, so as long as it's done in an orderly fashion, I'm happy. :)

 

On 3/20/2019 at 9:50 AM, psantosl said:
On 3/20/2019 at 2:23 AM, Marc S said:

For files like my scene, above a certain weight, it's a bit trickier. A solution would be to have an other rule to discard chosen assets after x days, but because we might need to get it back to a previous version after that date, we could set an exception with a milestone. A milestone would be a "super-changeset" where that rule doesn't apply.

Of course this means you are in single branch, right. How would this work with multiple branches?

The "super changeset" can be simply a labelled one. You know, you can label changesets.

So the rule would be something like:

If you look in the branch explorer and view "only relevant" changesets, all the changesets that disappears I'd say are eligible for removing the data from. So all inbetween-y changesets after some point should have their data deleted can be a general rule.

Share this post


Link to post
Share on other sites
On 3/20/2019 at 9:50 AM, psantosl said:

My question is: why do you checkin the lightmap if it can be rebaked? Because it is easier to share it through Plastic SCM than actually using other mechanism, correct?

Yes. I come from Unity's "Collaborate", a pseudo VCS tool used to work as a team. If I'm not mistaken, it only kept 10 last versions of each files and that was enough for us.

Understood.

 

Of course this means you are in single branch, right. How would this work with multiple branches?

I'm not sure about that. I don't have an extensive branches experience.

The "super changeset" can be simply a labelled one. You know, you can label changesets.

I did not but I'm checking this out. Thanks

So the rule would be something like:


Keep only 3 versions of LightingData.asset.

Keep only N versions of files > that size except if they are labelled.

Yes, that would work. Though we would need a label preset option to avoid human error.

I'm actually very interested on this topic, because it makes a lot of sense for us to add extra features for teams using Unity.

I'll add more people to the conversation.

 

I'm glad this is welcome. I think you have some business to gain from unity's users since they screwed up badly with their own service (Collaborate). It was so neglected and full of bugs that it became unusable. I subscribed to your service in an emergency and believe you should offer a unity preset plan because that's what I was looking for.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...