Reference the shared memory object in a worker process when using spawn multiprocessing method #106
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
num_workers
processes.fork
and when a process is forked the child process inherits all the same variables in the same state as they were in the parent.spawn
and when a process is spawned, it begins by starting a new Python interpreter. The current module is reimported and new versions of all the variables are created.spawn
processes, since the resources are not shared, any changes made innext_epoch
variable by thenum_workers
will not be reflected in the main process and this results in thetest_streamingdataloader_mid_epoch_resumption
test failures whennum_workers
> 0.num_workers
processes which was created in the main process. This would ensure that the main process or a parent process would see the updated values when changes made by thenum_workers
processes.Issue #, if available:
Merge Checklist:
Put an
x
without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.General
Tests
pre-commit
on my change. (check out thepre-commit
section of prerequisites)