Making DVC more feature-complete for Game Development

[Hisamera]

Greetings

First things first, I want everybody to know what this discussion doesn't entail, it is NOT a "Someone, somewhere, please code this features!!!". I want this discussion to be a start of open-minded cooperation with the help of DVC maintainers and DVC community, to establish how to approach this problem, and progress with implementation of those features. We (Due to contract I cannot disclose, what I mean by "We"), are fully capable of just taking DVC and implementing these features on our backend ourselves, but we would like for it to be mainlined into main, open-source repo, so we won't have to actively maintain it.

We are fully aware that DVC is more geared towards "Machine Learning Projects", but Game Development projects have a lot in common. After taking a look at DVC features, we have pinpointed those three must-have

1. File locking (And possibility to customize how it is done; for example, exclusive lock, or read-only lock - to help with "swarm" reviews, I don't want to go into details here, to not limit possible implementations)
2. User-level permissions (Due to other tools being made for Git like Android-repo, DVC would at most need branch-level per user permissions, but we fell like a general per-repo, User-level permissions would be a good start, how to do it, to have in DVC-repo or maybe a server program like a github for git? That's what we want to discuss)
3. Hooks (Let's be real, all companies need to get their backend, just right; hooks would allow this customization)
   With good implementation of Hooks, there could be a possibility to solve 1. and 2. but maybe DVC community would have other better ideas. If we switch and try out a green project we would need a DVC in a state that makes it BETTER than perforce is for US currently, otherwise why switch?

We applaud - for example - "Split data into blocks #829", as we think that Data Version System can compress that data far better than a general compression algorithm ever could; with openZFS possibly becoming multi-OS we can compress and dedup data that way.

Game dev makes a lot of artifacts; maybe with #829 a new daily build would not weigh another 10GB, but maybe 100MB? This would allow us to focus more on delivering value for our customers, than on our backend.

We sacrifice today for tommorow, and are firm believers in tools, and even though there are other projects that are more feature-complete for US... Let's just say that we don't like the way they are maintained and/or where they are headed.

We would like for this discussion to end with an Epic (issue) that entails what needs to be done, in what way and that an effort could be made in an orderly and efficient manner to implement those features.

With regards
Undisclosed company.

[pared]

I guess, in the background, there always has been this idea that a lot of different use cases could be covered by DVC. Not only ML use code + binaries = something approach.

File locking (And the possibility to customize how it is done; for example, exclusive lock, or read-only lock - to help with "swarm" reviews, I don't want to go into details here, to not limit possible implementations)

Sorry, swarm review is something new for me, tried watching the video on perforce but I do not understand what are its advantages over "usual" review offered, for example by Github. Also - are we talking here about code review or also other assets reviews? Like would you need the ability to review changes of, for example, images?

User-level permissions (Due to other tools being made for Git like Android-repo, DVC would at most need branch-level per user permissions, but we feel like a general per-repo, User-level permissions would be a good start, how to do it, to have in DVC-repo or maybe a server program like a github for git? That's what we want to discuss)

Since DVC leverages git for project code and its structure versioning, all the tools created for git should be working fine with it. For example Gitlab/Github. But that regards to code. I guess that you would also probably need to control for data? Ex - userX gets permissions for branchY, so he can only access data related to branchY? Am I right here?

Hooks (Let's be real, all companies need to get their backend, just right; hooks would allow this customization)
With good implementation of Hooks, there could be a possibility to solve 1. and 2. but maybe DVC community would have other better ideas. If we switch and try out a green project we would need a DVC in a state that makes it BETTER than perforce is for US currently, otherwise why switch?

Are we talking about git-like hooks? They should be working just fine with DVC, if used with git repo.

I think this is a great idea, the question is whether we can incorporate those requirements to our plans of development but I think this requires some discussion.
cc @dberenbaum

[Hisamera]

Sorry, swarm review is something new for me, tried watching the video on perforce but I do not understand what are its advantages over "usual" review offered, for example by Github. Also - are we talking here about code review or also other assets reviews? Like would you need the ability to review changes of, for example, images?

With Perforce when you lock a file, it is what is called an "Exclusive lock". With this type of lock, no one else is even permitted to download a file you are working on if they don't have it on their drive already. With swarm reviews, it is problematic because sometimes, to review changes fully, you NEED to have both the source file and the binary file (to see the finished result). With those exclusive locks, reviews with multiple people have to happen sequentially, not in parallel. 

I mean that those types of locks also have their uses but this could be resolved with a read-only lock, that DVC could use to reject a PUSH from someone that doesn't have a lock, but not a PULL. So both exclusive and some sort of "read-only" lock would be preferred. Exclusive lock is a must.

To be precise, not exactly images, but sometimes compressed database files and other such things. Believe me, a LOT of things. That's why this exclusive lock is sometimes a **** ** *** ***. But overall, it saves time. 

Swarm review is just a marketing term from Perforce.

File locking is overall very needed to not double work. With hundreds of people working on a project, it is a quick and reliable way of saying "Hey, I'm working on this."

Since DVC leverages git for project code and its structure versioning, all the tools created for git should be working fine with it. For example Gitlab/Github. But that regards to code. I guess that you would also probably need to control for data? Ex - userX gets permissions for branchY, so he can only access data related to branchY? Am I right here?

I mean, there could be a front-end for a DVC, sort of like a Gitlab for Git, and it could have access to a single repository a'la Data Vault, and manage Authorization and Access. It was a sort of throw-in to list all the requirements. However, Gitlab, aside from its authorization system, appears to make extensive use of hooks, which DVC does not.Now that I'm thinking about it (while writing this), maybe a DVC-GIT pair could be a good start... I'm looking at it from a perforce point of perspective (single massive repository). I need to think about it. This is a bigger problem. How would we decide who gets access to what data if there were three DVC in one Git repo, one for art, one for programmers, and one for writers?

Are we talking about git-like hooks? They should be working just fine with DVC, if used with git repo.

I dug a little further into this. When you change data, the git repo also changes (meta-files), so git-hooks will also just work(?). Please correct me if I'm wrong, and we could use server-instance DVC to pull the data and do something with it?

[dberenbaum]

• File locking (And possibility to customize how it is done; for example, exclusive lock, or read-only lock - to help with "swarm" reviews, I don't want to go into details here, to not limit possible implementations)

This is my biggest question. It seems like it fundamentally violates the principles of distributed version control on which git and dvc are built. However, maybe there's a way to support it at least as an optional feature for a specified remote 🤔 .

• User-level permissions (Due to other tools being made for Git like Android-repo, DVC would at most need branch-level per user permissions, but we fell like a general per-repo, User-level permissions would be a good start, how to do it, to have in DVC-repo or maybe a server program like a github for git? That's what we want to discuss)

👍 We don't have this prioritized yet but it is something that we would be open to adding in the future.

• Hooks (Let's be real, all companies need to get their backend, just right; hooks would allow this customization)
  With good implementation of Hooks, there could be a possibility to solve 1. and 2. but maybe DVC community would have other better ideas. If we switch and try out a green project we would need a DVC in a state that makes it BETTER than perforce is for US currently, otherwise why switch?

As you mentioned later, Git hooks should "just work," but feel free to follow up if you need something specific.

[pared]

File locking is overall very needed to not double work. With hundreds of people working on a project, it is a quick and reliable way of saying "Hey, I'm working on this."

Hmm, but that kind of looks like we are talking here not about versioning system, but rather an UI that would prevent other users from updating until current reviewer is done with it, right?

maybe a DVC-GIT pair could be a good start

This is recommended way of working with DVC :) Git is for text files, DVC takes care of managing pipeline and storing data. The connecting bridge between the two is dvc.lock file which contains checksums for data, hence allowing versioning the datasets in git.

How would we decide who gets access to what data if there were three DVC in one Git repo, one for art, one for programmers, and one for writers?

This is a very valid point, and as I recall we had some discussions about providing access control to DVC - though again, I think that would have to be a feature of hypothetical UI rather than DVC itself.

so git-hooks will also just work(?). Please correct me if I'm wrong, and we could use server-instance DVC to pull the data and do something with it?

Yep, dvc.lock will get updated - this can be used to trigger the hooks.

Also, regarding the swarm review - Hypotheticaly if two people are reviewing same change and making some updates, I guess the push rejection from git server should indicate that someone is already working on that (the first person push-ing will create additional commits and git server will reject other changes due to commit history mismatch). I mean Its definitely not as convenient as being able to lock file the file, but at least requires the users to rebase their copy. Worth remembering that in this case force push should be heavily restricted.

[Hisamera]

Hmmm...

tl;dr I came to the conclusion that having a good API (I didn't dive into this topic yet in DVC) to ease implementing the back-end, supporting locking files as an option from the client-side, and standardizing the HTTP path to request a list of currently locked files (details below) should be sufficient.

I've been reading a little more about the state of DVC. 3. seems to be possible with a combination of git hooks and git custom commands (we ran a few tests, but until we deploy it in a greenfield project nothing can be said for 100%).

User-Level permissions; this could be solved by first hitting a HTTP server with custom logic with your HTTP "dvc remote modify myremote url https://example.com/path." 

But there are also client-side issues. How does the DVC client react to access denial? Can it retrieve information like git with "remote: "?

Because the server could just provide (or deny) different data to different users based on what they are allowed to access, the client would just need to act accordingly. Making as much of the DVC operations available from some-kind of API (which you currently have), would be indispensable to make it work seamlessly and for a long time (especially after adding #829 ). Without this API, one would have to develop a backend from scratch for DVC repositories. 

(Example: client-DVC makes a pull request to an intermediate HTTP server for the content folder — which holds data for programmers, writers, and artists — this server processes the request, makes an API request for the list of files to send to the on-server DVC instance, finds out that he has access to art-related resources, filters out this list of programmers and writers' files, and then sends only those files to a user).

Hmm, but that kind of looks like we are talking here not about versioning system, but rather an UI that would prevent other users from updating until current reviewer is done with it, right?

I think we focused too much on the perforce reviewing system (gitlab has similar functionality in this regard). If you knew about it, sorry, but I'll clarify by pasting in a very succinct comment about what file locking is.

Locks address a very different problem. Binary assets usually cannot be merged, and often involve large amounts of work in changing them. This is especially true of large assets. If two developers start making changes to the same file, one set of changes will have to be discarded, and then recreated starting with the other change as a base. Locking prevents this from happening by accident. If you encounter a lock, you either wait until later to make your changes, or you go talk to the person who has that lock. Either they can make the requested change, or they can release the lock and allow you to make your change on top of their changes so far. Then when you're done, you can push your change and release the lock, allowing them to continue with your change. Thus, it allows changes (the entire developer process, not file-writing) for specific files to be serialized, rather than the parallel-then-merge paradigm used for textual source files. Source: https://stackoverflow.com/users/556742/bob-kerns

As stated above, file locking makes changes sequential and saves a TON of time overall. It's a must for game-dev, where six people may be working on one asset (level, character, etc). DVC would need to support (even only as an option) file-locking on the client side (which boils down to making everything that the user specifies as non-mergable, for example with *.jpg mask read-only and giving write access after successfully locking a file), and make an effort to standardize a folder on FTP repos and URI path on HTTP to access server-side information about what files are locked.

We don't expect the DVC core team to maintain a server-implementation. The standardization would save the DVC ecosystem from a free-for-all with dozens of incompatible implementations of server-side locking. It would also make it possible for the DVC team to make changes underneath without everything suddenly breaking for customers.

I've seen a few different solutions. Gitlab just made their own tool for their premium members before git LFS came, I don't have the time to look into the source code of how it was made, but this is stated on their website.

How it works
File Locking is included in GitLab Enterprise Edition Premium.
Anyone who has push access to a repository can lock a file or a directory.
Once a file is locked, anyone else trying to push changes to it will get a pre-receive hook or a notification in the UI that will reject the changes.
Teams can view and manage locks directly from the web interface of each of their projects.

So they utilised git hooks. For which DVC hooks may be redundant, but as it was a premium feature, I guess it wasn't easy to add this.

Git LFS went a different way. They made ".git\lfs\cache\locks\refs\heads\main\verifiable", on the client machine and hard-coded a path to retrieve that information from the server. Gitlab implemented it and stored all the information in a PostgreSQL database.

Ultimately, Git LFS on a supporting hosting machine does a GET request from reponame.git/info/lfs/locks and it (server) responds as follows (sample) {"locks":[{"path":"path/from/root/of/repo/test.gif","id":"17","locked_at":"2022-02-14T18:18:54Z","owner":{"name":"Testing"}}]}

I honestly think this discussion is going in the right direction, so I would like to continue it until the DVC core team is satisfied.

With regards

[pared]

standardizing the HTTP path to request a list of currently locked files (details below) should be sufficient

DVC by itself does not have a server, aren't we talking here about actually creating GitHub-like UI that would expand DVC capabilities?

User-Level permissions; this could be solved by first hitting a HTTP server with custom logic with your HTTP "dvc remote modify myremote url https://example.com/path."

As above.

But there are also client-side issues. How does the DVC client react to access denial? Can it retrieve information like git with "remote: "?

DVC by itself does not react - GIT handles versioning so every problem with GIT repo (ex conflict) will be handled as in the usual git workflow. In conflict example, GitHub will reject the push if the upstream log differs from our change log.

We don't expect the DVC core team to maintain a server-implementation.

Got it. I guess I am still a bit confused about this functionality and provided GitLab docs helps me understand this a bit. My understanding of it as of now is that in the ideal scenario we have some kind of a server that is capable of handling GIT+DVC repos. This server contains one source of truth regarding which files are locked, and doesn't allow the users to modify a particular asset unless the lock is taken down, even if GIT would allow that in a usual scenario (eg someone adds a new commit, with history matching the target branch up to HEAD~1). Do I get it right so far?

[Hisamera]

First, let's make sure we're all on the same page.

I hope we agree that 2. (User-level permissions) is possible with a HTTP server, and 3. (Hooks) is possible with a combination of git hooks and custom git commands. And we will do it ourselves.

What is not possible in any way or form is 1. (File locking), which we operate under the definition that we provided.

DVC by itself does not have a server, aren't we talking here about actually creating GitHub-like UI that would expand DVC capabilities?

Maybe I wrote it in a confusing manner. I meant that DVC in the main client-side repo should have a default path to a file containing all locks, and by whom they were locked (It could require git for this to work, and just steal their username or email as a default, or maybe DVC maintainers have a better idea).

So maybe on a ftp server it will access ./locks.dvc (by ./ i mean root folder) and on a http repo it will try to access by DEFAULT (it doesn't have to be hardcoded, like you can already specify the method by which to call HTTP remote) www.dvcreposite.com/locks. It is as simple as that. 

By providing this (even as an overridable default), it'll make major strides to standardize custom server-side implementations, and when maybe one day Gitlab or some other company invests in supporting you, believe me, it will (like I'm 99% sure) use this default. It'll just make for a greater experience for all. 

So this is purely a client-side implementation, to which a server-side implementation will just bend.

User-Level permissions; this could be solved by first hitting a HTTP server with custom logic with your HTTP "dvc remote modify myremote url https://example.com/path."
As above.

I just wrote it to be pedantic and disambigious. We already found a way to implement User-Level permissions through a custom HTTP server as a proof internally. We don't require further action from the DVC community.

But there are also client-side issues. How does the DVC client react to access denial? Can it retrieve information like git with "remote: "?
DVC by itself does not react - GIT handles versioning so every problem with GIT repo (ex conflict) will be handled as in the usual git workflow. In conflict example, GitHub will reject the push if the upstream log differs from our change log.

We found out that in the case of an error, DVC just chugs along and then tells how many files it couldn't download. This behaviour is sufficient for us.

This server contains one source of truth regarding which files are locked, and doesn't allow the users to modify a particular asset unless the lock is taken down, even if GIT would allow that in a usual scenario

Aside from someone holding the lock.

tl;dr
If I am again saying too much, file locking needs a server-side (we don't expect the DVC core team to do this part) implementation and a client-side (which we want to help design, mainline it and change our focus to other things) implementation, which requires setting a file as read-only on the local file-system and the ability to retrieve that information from a cloud drive (to not say server). That's pretty much all.

Yes, in a situation in which someone pushes a commit containing a locked file - which he didn't lock himself - the Git+DVC server should reject the whole push. If someone does hold a lock for that file and pushes a commit that doesn't contain files locked by another user, even though this commit contains a locked file, it should be accepted.
There is a question of whether or not the file stays locked or unlocked after being pushed, but this is such meager overhead that I'm leaning on the side of unlocking it locally and querying the server for locks and seeing how it handles that situation. If it still holds a lock for that user, just lock it locally as well (so allow writes), and if it is not locked, nothing has to be done.

That's why DVC in such a situation would NEED to maintain on the local drive which files are read-only and which are not, because without it, two people could still be modifing the same file, and just notice it while one tries to push and that push gets rejected.

We still want this discussion to end with an Epic (issue) that entails what needs to be done, but the scope got way smaller, and now encompasses only file-locking. But at least you don't hit us with "It can't be done" at the very start...

If you still weren't able to understand me, just ask further questions. I'll be here until we can develop a common position.

[pared]

Aside from someone holding the lock.

Yes, sorry for not clarifying that.

---

Maybe I wrote it in a confusing manner. I meant that DVC in the main client-side repo should have a default path to a file containing all locks, and by whom they were locked (It could require git for this to work, and just steal their username or email as a default, or maybe DVC maintainers have a better idea).
So maybe on a ftp server it will access ./locks.dvc (by ./ i mean root folder) and on a http repo it will try to access by DEFAULT (it doesn't have to be hardcoded, like you can already specify the method by which to call HTTP remote) www.dvcreposite.com/locks. It is as simple as that.
By providing this (even as an overridable default), it'll make major strides to standardize custom server-side implementations, and when maybe one day Gitlab or some other company invests in supporting you, believe me, it will (like I'm 99% sure) use this default. It'll just make for a greater experience for all.
So this is purely a client-side implementation, to which a server-side implementation will just bend.

Ok, I think I start to get the gist of it.
Few notes for myself and future readers here:

1. DVC by itself does not have any integrations with some centralized server. The actual versioning is done by GIT. That way we don't have to create our own versioning system server. But that also means that if we want to implement this it will require us to integrate it into existing platforms handling GIT.

2. The bridge between versioning (GIT) and DVC is dvc.lock file, which contains checksums for the binary assets that users don't store in GIT.

3. In order to provide locking functionality, we would have to have an entity on a centralized server that would keep information on what is locked. This entity should be able to "lock" the file. By "lock" I would say that we store information on which file path, on which revisions are locked, and reject incoming changes if they try to override them. Looking at 2 that would require some logic parsing dvc.lock and making sure that changed md5's are not "locked". (Problem 1). I guess the first point for starting research on it would be the beforementioned GitLab and how exactly they lock files. This seems like an absolute basic necessity to support this scenario.

4. Local file locking - not sure that would be required in the initial version - some supporting command or workflow allowing the user easy understanding of what is locked could be enough (If the user doesn't check it, he/she will get slapped with push rejection). Not ideal but I am trying to find absolute necessities that would make this scenario work. Also, even if we support locking, we would require some user activity (like dvc locks update) to make sure we work on up-to-date information.
   A typical workflow in this case - check if the file you want to work on is locked, if not - lock it, start working.

Problems:
1 - what to do with dirs? We should be able to choose whether to "lock" the whole dir or granularly "lock" a particular file inside - the logic for locking in the case of dir would have to be a bit more elaborate to verify, that if we change dir content, we do not change the "locked" file.

---

Now, hypothetical question - is it possible to do it with existing, available tools?

I think that would be complicated by possible (Disclaimer: it's my wild imagination, it might be too much for a workaround):

1. Separate git repo as our source of "locks" truth
2. In order to lock a file, one needs to put the path to it in our locks repo
3. If the user wants to start working on a particular file, he/she needs to create a commit for it (possible lock file should be a YAML dict, that way we can parse a file, and YAML will throw an error with duplicated keys)
4. Main repo has a workflow (like GitHub Action or GitLab CI/CD) that will run a custom "lock checking" code. This lock checking code would require the most work here as it requires knowledge of DVC insides. The workflow doesn't pass if PR modifies the locked file.

---

We still want this discussion to end with an Epic (issue) that entails what needs to be done, but the scope got way smaller, and now encompasses only file-locking. But at least you don't hit us with "It can't be done" at the very start...

I cannot guarantee that we will be able to pick it up anytime soon as we have a lot of work as it is, but it's always worth talking about it :) . I will bring this subject up with the team once we have the requirements clarified, and discuss how it looks with our plans for the future.

[pared]

@Hisamera Have you tried evaluating GitLab for your needs? It seems that it has a file locking capabilities.