Cleanup internal data-structures when process has been forked

[jleibs]

Closes #1921

What

The crux of the problem is the following:

The child process is created with a single thread—the one that called fork(). The entire virtual address space of the parent is replicated in the child, ...

The major consequence of this is that our global RecordingStream context is duplicated into the child memory space but none of the threads (batcher, tcp-sender, dropper, etc.) are duplicated. When we go to call connect() inside the forked process, we try to replace the global recording-stream, which subsequently tries to call drop on the forked copy of RecordingStreamInner . However, without any existing threads to process the flush, things just hang inside that flush call.

We take a few actions to alleviate this problem:

1. Introduce a new SDK function: cleanup_if_forked which compares the process-ids on existing globals and forgets them as necessary.
2. In python, use os.register_at_fork to proactively call cleanup_if_forked in any forked child processes.
3. Also add a call to cleanup_if_forked inside of init() in case we're forking through a more exotic mechanism.
4. Check for the forked state anywhere we potentially flush to avoid deadlocks and produce a visible user-error.

Additionally, it turns out that forked processes bypass the normal python atexit handler which means we don't get proper shutdown/flush behavior when the forked processes terminate. To help users workaround this, we introduce a @shutdown_at_exit decorator which can be used to decorate functions launched via multiprocessing.

Testing

On linux:

$ python examples/python/multiprocessing/main.py

observe demo exits cleanly and all data shows in viewer.

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested demo.rerun.io (if applicable)

• PR Build Summary
• Docs preview
• Examples preview

[teh-cmc]

Looks fantastic 👌

[teh-cmc]

That comment (which predates the PR) doesn't make sense to me and I vote to remote it.

All processes spawned with multiprocessing will automatically be assigned the same default recording_id.

No they won't, that's why we need to call init in all children processes too 🤔

[jleibs]

They do have the same default recording-id... that's why calling init works instead of creating a new recording. But we still have to call init to create new sinks.

[teh-cmc]

Oh right I confused the RecordingId with the app name... If i'm getting confused by this comment, I don't imagine it's better for end users though :|

[emilk]

it's worth clarifying here if we are detecting the fork as the parent or as the child, here and elsewhere