Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rustdoc does not wait for DocFS writes to complete before exiting on Windows #109060

Closed
willcrichton opened this issue Mar 12, 2023 · 0 comments · Fixed by #109139
Closed

Rustdoc does not wait for DocFS writes to complete before exiting on Windows #109060

willcrichton opened this issue Mar 12, 2023 · 0 comments · Fixed by #109139
Labels
C-bug Category: This is a bug. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.

Comments

@willcrichton
Copy link
Contributor

willcrichton commented Mar 12, 2023

As discovered by @ehuss in rust-lang/cargo#10120, a Cargo test is intermittently failing on Windows because a source file generated by Rustdoc is not being written before the process exits. The suspected issue is that Rustdoc spawns asynchronous writes here:

rayon::spawn(move || {
fs::write(&path, contents).unwrap_or_else(|e| {
sender.send(format!("\"{}\": {}", path.display(), e)).unwrap_or_else(|_| {
panic!("failed to send error on \"{}\"", path.display())
})
});
});

However, Rustdoc never checks that the writes are completed. We should add a mechanism that prevents the process from exiting before these write tasks are completed.

One possible solution:

  • Add a field outstanding_writes: Arc<AtomicUsize> to DocFS.
  • Increment outstanding_writes at the start of DocFS::write, and decrement at the end of the spawned Rayon task.
  • Add a method DocFS::wait_for_outstanding_writes(&self) that spin-loops until outstanding_writes is zero. (Alternatively: use a condition variable, and have write notify the condvar when outstanding_writes is decremented.)
  • Either call wait_for_outstanding_writes explicitly in the end of the Rustdoc process, or add it as a Drop implementation to DocFS.

cc @GuillaumeGomez

@willcrichton willcrichton added the C-bug Category: This is a bug. label Mar 12, 2023
@workingjubilee workingjubilee added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. A-doctests Area: Documentation tests, run by rustdoc labels Mar 12, 2023
@jyn514 jyn514 removed the A-doctests Area: Documentation tests, run by rustdoc label Mar 13, 2023
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Mar 15, 2023
…t-for-write, r=notriddle

rustdoc: DocFS: Replace rayon with threadpool and enable it for all targets

Fixes rust-lang#109060.

Switching to `threadpool` makes it a bit simpler for us to wait for all tasks in `DocFS` directly in the `Drop` implementation. I'm also curious if making all the writes into a thread pool could improve run time for rustdoc on all other platforms than Windows as well.

I'll run a perf check to see.

cc `@ehuss`
r? `@notriddle`
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Mar 16, 2023
…t-for-write, r=notriddle

rustdoc: DocFS: Replace rayon with threadpool and enable it for all targets

Fixes rust-lang#109060.

Switching to `threadpool` makes it a bit simpler for us to wait for all tasks in `DocFS` directly in the `Drop` implementation. I'm also curious if making all the writes into a thread pool could improve run time for rustdoc on all other platforms than Windows as well.

I'll run a perf check to see.

cc ``@ehuss``
r? ``@notriddle``
@bors bors closed this as completed in 6cf2f47 Mar 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
Projects
None yet
3 participants