Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix race condition when trying to fetch the same project from the registry more than once #131

Merged
merged 2 commits into from
Sep 27, 2024

Conversation

kylewlacy
Copy link
Member

Closes #128

This PR updates the fetch_project_from_registry function to use a mutex to ensure that the same project doesn't get cloned more than once at a time. This works by using a BTreeMap (wrapped in a Mutex), where each key-value pair is a project hash and another mutex

As far as implementation, there are a few other places in the code where we have a similar problem of wanting to ensure that a piece of code only runs once for a particular key: baking recipes, and creating outputs. For baking, we use a map that contains channels, which allows parallel calls to get the result immediately without doing more work. For creating outputs, we use a lazier option of using a single mutex to guard the whole section-- it's not (currently) fine-grained. This new implementation is kind of like a mix between these two other implementations, where we do have fine-grained locking, but each invocation ends up doing a bit of work to read the filesystem to figure out where the output goes

One downside with this implementation is that the mutex map uses Arcs, and it never gets keys removed. We could switch to using Weak values then add a "remove" phase, but I thought it'd be unlikely that enough keys could ever be added to the map to become an issue

@kylewlacy kylewlacy changed the title Fix move temp project error Fix race condition when trying to fetch the same project from the registry more than once Sep 27, 2024
@kylewlacy kylewlacy enabled auto-merge (squash) September 27, 2024 08:50
@kylewlacy kylewlacy merged commit fce1529 into main Sep 27, 2024
5 checks passed
@kylewlacy kylewlacy deleted the fix-move-temp-project-error branch September 27, 2024 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant