TLS destructors on the main thread are not always run

[alexcrichton]

There are some platforms where TLS destructors are run when the main thread exits, and there are some platforms where this does not happen,~~ and there are some platforms where things just go crazy ~~. For example, testing this program:

struct Foo;

impl Drop for Foo {
    fn drop(&mut self) {
        println!("wut");
    }
}

thread_local!(static FOO: Foo = Foo);

fn main() {
    FOO.with(|_| {});
}

This should print Foo dtor, but prints nothing on some targets:

• musl (e.g. i686-unknown-linux-musl, x86_64-unknown-linux-musl)
• android (e.g. arm-linux-androideabi)
• Linux with glibc before 2.18
• Maybe more?

Here's a more complicated testcase that also involves initializing a destructor while destructors are being run; ideally this will be added to the test suite at some point:

struct Bar;

impl Drop for Bar {
    fn drop(&mut self) {
        println!("Bar dtor");
    }
}

struct Foo;

impl Drop for Foo {
    fn drop(&mut self) {
        println!("Foo dtor");
        // We initialize another thread-local inside the dtor, which is an interesting corner case.
        thread_local!(static BAR: Bar = Bar);
        BAR.with(|_| {});
    }
}

thread_local!(static FOO: Foo = Foo);

fn main() {
    FOO.with(|_| {});
}

some older notes

• Linux ELF TLS - appears to work
• Linux pthread destructors - appear to not work (valgrind shows memory leak for a simple print statement #19776)
• OSX - appears to call destructors, but the program above specifically causes some form of memory corrupting, triggering an assert in malloc fixed by OSX: fix #57534 registering thread dtors while running thread dtors #57655
• Windows GNU/MSVC - appears to not work. We're listening for DLL_PROCESS_DETACH but for some reason we're not getting that notification.

[ranma42]

The memory corruption in MacOSX seems to be related to the TLS usage during Drop of a TLS value.
If the print! invocation is replaced with stdout().write, no crash occurs.
However, if stdout is used within main, the process dies with the error

thread '<main>' panicked at 'cannot access stdout during shutdown', ../src/libcore/option.rs:333

both if drop calls print! and if it calls write.

[ranma42]

pthread destructors seem to have the same behaviour on MacOS X as on Linux (i.e. they are not run from when the main thread terminates). Googling around seems to suggest that this is a known (expected?) fact. Specifically, TLS destructors are only run when pthread_exit is invoked, which for non-main threads happens implicitly.
The only authoritative source I was able to find is http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_exit.html

[alexcrichton]

Yeah I think solving this in the case of pthreads will either require us documenting "dtors may not run" or perhaps adding our own atexit handler or something like that to handle this (e.g. some custom code on our end). Not entirely sure what that would look like though!

[ranma42]

In order to make the atexit handler practical we would need to keep explicitly track the active destructors (at least on the main thread), i.e. basically keep the list of DTORS as in the pthread fallback implementation. This might not be as bad as it looks; in fact it might allow us to reuse some code (currently the Linux fallback dtor and the normal Windows dtor are basically doing the same thing in a slightly different way).

Another option would be to start the main function in a new thread and immediately joining it in lang_start. This would be about as effective as hooking the destructors in the Rust atexit, because for Rust application either way would work just fine and for Rust libraries used by non-Rust applications TLS would not be destroyed in either case.

The C atexit function would suffer from the same limitations as the Rust atexit (need to track dtors), but it might help in non-Rust applications, assuming that go through the normal termination (exit, not _Exit nor quick_exit nor an abort of any kind) as expected by the C runtime.

Documenting "dtors may not run" looks like a reasonable compromise to me, especially if we can guarantee that this will at most happen for the main thread (and, ideally, only on non-Rust apps). I believe that this guarantee is especially important if threads are spawned and joined (instead of managed as a thread pool) to ensure that leaks are bounded.

[alexcrichton]

we would need to keep explicitly track the active destructors

Yeah this probably isn't so bad as we already do it in some cases, so it'd just be a matter of shuffling things around.

Another option would be to start the main function in a new thread and immediately joining it in lang_start

Unfortunately I think this won't work because there's a number of GUI frameworks (or something like that) which only work on the main thread (I think on Windows in particular), so running off-the-main-thread by default may be a bit of a heavy hammer to fix this!

Documenting "dtors may not run" looks like a reasonable compromise to me

I agree it's probably not that bad, but if we could get OSX and Windows working reliably, it's more pressure for us to get pthreads working reliably :). I feel like Windows is pretty easy to fix, it seems like some small error is just being missed there. OSX also feels the same way to me in terms of weird things happening. The pthreads case also isn't super critical on Linux because it's only used on older linuxes.

Overall I think we may still have enough rope left to climb out and close this issue, so I wouldn't be quite willing just yet to close it out by documenting things may not run.

[retep998]

there's a number of GUI frameworks (or something like that) which only work on the main thread (I think on Windows in particular)

Windows itself does not actually care which thread you do the UI on, as long as you're consistent. Once you create a window on a given thread, that thread now has a message queue which you're obligated to pump forever.

[mtak-]

Just confirmed the memory corruption in the OP on OSX occurs for the same reason listed here #57534 (comment). It is not specific to the main thread. AFAICT it is UB on macos to register destructors during thread cleanup via _tlv_atexit,

[RalfJung]

So from the 4 items listed in the OP, the first two are fine, and the third was a separate bug that got fixed. EDIT: oh, the 2nd one is also not fine, I see.

What is the situation on Windows nowadays? (Cc @retep998)

[workingjubilee]

Looks like the only failing one left then is "Linux pthread destructors - appear to not work" -- and where do we even use that? On playground, it also prints wut.

It seems we do not, but rather, we currently effectively require libc support to include thread-local support.

[workingjubilee]

Wait.

Android and OpenBSD.

cc @quininer Sorry to bother, but you made this PR to add "emulated TLS" support: #117873

Can you confirm if this is still an issue on the appropriate targets?

[quininer]

@workingjubilee I confirmed that emutls on android will print "wut".

[workingjubilee]

Thank you! It seems this is resolved then? Hopefully?

[RalfJung]

I am a bit confused by the macos sitaution: I don't see how the simple program above could be affected by #57655. Very strange. 🤷

Do we have a test checking that main thread destructors run? Reopening as needs-test.

[RalfJung]

PR with a test is up: #126829

(I couldn't find an existing test.)

[RalfJung]

Turns out musl is still broken, I opened a new issue for that: #126858.

[RalfJung]

I gave up on the PR since even some glibc targets are effected (including the one we test on CI), so now we have two issues tracking the same thing -- this and #126858.

EDIT: I have closed the other issue.

[surban]

Testing on FreeBSD 14 shows that TLS destructors are not run for the main thread.