Fix directory creation race condition in Folder and SandboxFolder
#4912
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `Folder` and `SandboxFolder` classes of `aiida.common.folders` used
the following paradigm to create the required folders:
if not os.path.exists(filepath):
os.makedirs(filepath)
However, this is susceptible to a race condition. If two processes call
the same piece of code almost at the same time, they may both evaluate
the conditional to be True if the filepath does not yet exist, but one
of the two will actually get to the creation first, causing the second
process to except with a `FileExistsError`.
The solution is to replace it with `os.makedirs(filepath, exist_ok=True)`
which will swallow the exception if the path already exists.
Codecov Report
@@ Coverage Diff @@
## develop #4912 +/- ##
===========================================
+ Coverage 80.07% 80.07% +0.01%
===========================================
Files 518 518
Lines 36683 36680 -3
===========================================
- Hits 29371 29369 -2
+ Misses 7312 7311 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
giovannipizzi
approved these changes
May 5, 2021
sphuber
added a commit
that referenced
this pull request
Aug 8, 2021
…#4912) The `Folder` and `SandboxFolder` classes of `aiida.common.folders` used the following paradigm to create the required folders: if not os.path.exists(filepath): os.makedirs(filepath) However, this is susceptible to a race condition. If two processes call the same piece of code almost at the same time, they may both evaluate the conditional to be True if the filepath does not yet exist, but one of the two will actually get to the creation first, causing the second process to except with a `FileExistsError`. The solution is to replace it with `os.makedirs(filepath, exist_ok=True)` which will swallow the exception if the path already exists. Cherry-pick: dc686c5
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #4911
The
FolderandSandboxFolderclasses ofaiida.common.foldersusedthe following paradigm to create the required folders:
However, this is susceptible to a race condition. If two processes call
the same piece of code almost at the same time, they may both evaluate
the conditional to be True if the filepath does not yet exist, but one
of the two will actually get to the creation first, causing the second
process to except with a
FileExistsError.The solution is to replace it with
os.makedirs(filepath, exist_ok=True)which will swallow the exception if the path already exists.