Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fsevent hangs on Mac during shutdown #208

Open
matklad opened this issue Jul 22, 2019 · 4 comments
Open

fsevent hangs on Mac during shutdown #208

matklad opened this issue Jul 22, 2019 · 4 comments

Comments

@matklad
Copy link

matklad commented Jul 22, 2019

System details

  • OS/Platform name and version: macOS 10.13.6 on a MacBook Pro (15 Zoll, 2016)
  • Rust version (if building from source): rustc --version: rustc 1.38.0-nightly (07e0c3651 2019-07-16)
  • Notify version (or commit hash if building from git): 4.0.12

Hi! Users report that rust-analyzer sometimes hangs during shutdown. The stack trace points to this code:

https://github.com/passcod/notify/blob/2b1f1d4d1acc8b9738ffbe41bfe6043ba37f9431/src/fsevent.rs#L109-L111

Downstream issue (with captured stack trace): rust-lang/rust-analyzer#1541

cc @killercup

@killercup
Copy link

  • OS/Platform name and version: macOS 10.13.6 on a MacBook Pro (15 Zoll, 2016)
  • Rust version: rustc 1.38.0-nightly (07e0c3651 2019-07-16)

@passcod passcod changed the title fsevent hands on Mac during shutdown fsevent hangs on Mac during shutdown Jul 22, 2019
@passcod
Copy link
Member

passcod commented Jul 22, 2019

I'm thinking this is likely because of #118. The original race was that the fsevent loop wasn't yet running, so ending it wasn't doing anything. The workaround was therefore to wait until it tells us it's waiting. Here, the loop is running (probably, given how this happens), but at the point of dropping, the fsevent loop isn't waiting for an event, so we yield until it does, but it's shutting down, so we're never going to get there.

#118 had a better solution: to use loop observers. Unfortunately I'm not a mac developer and don't really know how best to use those for this purpose. My guess is:

At startup:

  1. Create a loop observer firing on runloop exit that, idk, sets a "i'm dead" flag or sends to a channel or something.
  2. Save that in our struct.
  3. Start the thread:
    1. Check that the observer isn't invalidated. If it is, return. (If the observer has already been invalidated, we're stopping before we're starting, which was the cause of the initial deadlock.)
    2. Add the observer to the loop.
    3. Check again.
    4. Run the loop.

To stop the loop:

  1. Check if the observer is valid. If it's not, return. (An invalid observer would mean it's fired already, that is, that the loop has already exited.)
  2. Check if the runloop contains the observer. If it doesn't, return. (If the observer isn't present, we're exiting either before the loop has been initialised, before the thread has run, or while it's exiting but before the observer has been invalidated. Or, because this is multi-threaded and anything is possible, the observer was invalidated in between steps 1 and 2.)
  3. Invalidate the observer. This is to prevent the loop from starting if we're stopping before starting.
  4. Check the flag/channel that the observer sets/sends. If observer has run, return.
  5. Tell the runloop to stop.

This has no infinite wait loops, so no hangs and no 100% CPU usage. I think it covers all most cases. To be extra careful, there may be cases for:

  • starting the runloop for a few milliseconds only, rechecking the observers and flags, then running indefinitely.
  • adding another observer on enter that seeds the is_running flag so we have a better indication of whether the loop is running.
  • storing the thread handle and killing it after a timeout after we tell the runloop to stop.

This isn't especially hard to implement, so I can do that fairly soon. However, beforehand I'd want some kind of review of the above (or someone to say this is ridiculous and '''[[[this]]] is how to do that''') by a Rust developer familiar with CFRunLoop and mac programming (or a mac developer familiar with Rust, whichever). If you know someone... ;)

Pinging @cmyr in case they can take a look

@0xpr03
Copy link
Member

0xpr03 commented Oct 16, 2019

#210 is merged and fixes at least several deadlocks that also happens on linux, so my take would be to release a new version and see whether this resolves it ?

@0xpr03
Copy link
Member

0xpr03 commented Oct 17, 2019

4.0.14 is released, I'd appreciate feedback if this fixes the problem (or others..) as I can't test it on mac

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants