Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

only use recursive watchers on macOS and windows #4100

Merged
merged 2 commits into from
Mar 7, 2023

Conversation

sokra
Copy link
Member

@sokra sokra commented Mar 7, 2023

Description

Only windows and macOS support real recursive file watchers, other OS emulate it by walking the directory structure and watching all directories. But that might be really slow and use up a lot of watchers, so we don't want that. Instead we know exactly which directories are used and can watch selectively directories when they are accessed.

  • only use recursive watchers on macOS and windows
  • fallback to non-recursively watch read directories on other OS

Note that this implementation still has some bugs when renaming folders, but we can probably figure out these edge cases later...

fallback to non-recursively watch read directories on other OS
@vercel
Copy link

vercel bot commented Mar 7, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated
examples-cra-web 🔄 Building (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-kitchensink-blog 🔄 Building (Inspect) Mar 7, 2023 at 4:15PM (UTC)
8 Ignored Deployments
Name Status Preview Comments Updated
examples-basic-web ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-designsystem-docs ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-native-web ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-nonmonorepo ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-svelte-web ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-tailwind-web ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
examples-vite-web ⬜️ Ignored (Inspect) Mar 7, 2023 at 4:15PM (UTC)
turbo-site ⬜️ Ignored (Inspect) Visit Preview Mar 7, 2023 at 4:15PM (UTC)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 7, 2023

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Go e2e tests

See workflow summary for details

@github-actions
Copy link
Contributor

github-actions bot commented Mar 7, 2023

Benchmark for e2a9b99

Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8757.31µs ± 113.36µs N/A N/A N/A
bench_hmr_to_commit/Turbopack RCC/1000 modules 13.40ms ± 0.19ms N/A N/A N/A
bench_hmr_to_commit/Turbopack RSC/1000 modules 496.45ms ± 1.53ms N/A N/A N/A
bench_hmr_to_commit/Turbopack SSR/1000 modules 8907.09µs ± 49.51µs N/A N/A N/A
bench_hmr_to_eval/Turbopack CSR/1000 modules 8151.33µs ± 62.83µs N/A N/A N/A
bench_hmr_to_eval/Turbopack RCC/1000 modules 11.29ms ± 0.27ms N/A N/A N/A
bench_hmr_to_eval/Turbopack SSR/1000 modules 8022.69µs ± 56.56µs N/A N/A N/A
bench_hydration/Turbopack RCC/1000 modules 3577.22ms ± 9.43ms N/A N/A N/A
bench_hydration/Turbopack RSC/1000 modules 3239.94ms ± 13.06ms N/A N/A N/A
bench_hydration/Turbopack SSR/1000 modules 3305.74ms ± 8.17ms N/A N/A N/A
bench_startup/Turbopack CSR/1000 modules 2553.60ms ± 6.03ms N/A N/A N/A
bench_startup/Turbopack RCC/1000 modules 2126.94ms ± 4.02ms 85.26ms ± 0.14ms -95.99% -95.96%
bench_startup/Turbopack RSC/1000 modules 2058.49ms ± 6.56ms 84.93ms ± 0.18ms -95.87% -95.83%
bench_startup/Turbopack SSR/1000 modules 2034.94ms ± 5.00ms 84.50ms ± 0.11ms -95.85% -95.82%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8757.31µs ± 113.36µs N/A N/A N/A
bench_hmr_to_commit/Turbopack RCC/1000 modules 13.40ms ± 0.19ms N/A N/A N/A
bench_hmr_to_commit/Turbopack RSC/1000 modules 496.45ms ± 1.53ms N/A N/A N/A
bench_hmr_to_commit/Turbopack SSR/1000 modules 8907.09µs ± 49.51µs N/A N/A N/A
bench_hmr_to_eval/Turbopack CSR/1000 modules 8151.33µs ± 62.83µs N/A N/A N/A
bench_hmr_to_eval/Turbopack RCC/1000 modules 11.29ms ± 0.27ms N/A N/A N/A
bench_hmr_to_eval/Turbopack SSR/1000 modules 8022.69µs ± 56.56µs N/A N/A N/A
bench_hydration/Turbopack RCC/1000 modules 3577.22ms ± 9.43ms N/A N/A N/A
bench_hydration/Turbopack RSC/1000 modules 3239.94ms ± 13.06ms N/A N/A N/A
bench_hydration/Turbopack SSR/1000 modules 3305.74ms ± 8.17ms N/A N/A N/A
bench_startup/Turbopack CSR/1000 modules 2553.60ms ± 6.03ms N/A N/A N/A
bench_startup/Turbopack RCC/1000 modules 2126.94ms ± 4.02ms 85.26ms ± 0.14ms -95.99% -95.96%
bench_startup/Turbopack RSC/1000 modules 2058.49ms ± 6.56ms 84.93ms ± 0.18ms -95.87% -95.83%
bench_startup/Turbopack SSR/1000 modules 2034.94ms ± 5.00ms 84.50ms ± 0.11ms -95.85% -95.82%

@github-actions
Copy link
Contributor

github-actions bot commented Mar 7, 2023

Benchmark for 01255ab

Test Base PR % Significant %
bench_hmr_to_eval/Turbopack RCC/1000 modules 10.95ms ± 0.15ms 11.57ms ± 0.14ms +5.63% +0.24%
bench_hydration/Turbopack RCC/1000 modules 3582.60ms ± 6.75ms 3539.83ms ± 12.85ms -1.19% -0.10%
bench_startup/Turbopack CSR/1000 modules 2584.51ms ± 9.56ms 2532.50ms ± 7.79ms -2.01% -0.67%
bench_startup/Turbopack RCC/1000 modules 2105.95ms ± 4.84ms 2090.89ms ± 2.24ms -0.72% -0.04%
bench_startup/Turbopack RSC/1000 modules 2047.48ms ± 3.57ms 2027.99ms ± 5.70ms -0.95% -0.05%
bench_startup/Turbopack SSR/1000 modules 2049.95ms ± 3.68ms 2011.25ms ± 4.64ms -1.89% -1.08%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8762.34µs ± 67.82µs 8860.73µs ± 91.16µs +1.12%
bench_hmr_to_commit/Turbopack RCC/1000 modules 12.71ms ± 0.31ms 14.11ms ± 0.98ms +11.06%
bench_hmr_to_commit/Turbopack RSC/1000 modules 496.16ms ± 1.42ms 494.88ms ± 0.97ms -0.26%
bench_hmr_to_commit/Turbopack SSR/1000 modules 8972.20µs ± 95.38µs 9102.91µs ± 87.03µs +1.46%
bench_hmr_to_eval/Turbopack CSR/1000 modules 7630.68µs ± 62.44µs 7661.41µs ± 54.40µs +0.40%
bench_hmr_to_eval/Turbopack RCC/1000 modules 10.95ms ± 0.15ms 11.57ms ± 0.14ms +5.63% +0.24%
bench_hmr_to_eval/Turbopack SSR/1000 modules 7875.04µs ± 32.10µs 7916.72µs ± 59.09µs +0.53%
bench_hydration/Turbopack RCC/1000 modules 3582.60ms ± 6.75ms 3539.83ms ± 12.85ms -1.19% -0.10%
bench_hydration/Turbopack RSC/1000 modules 3221.66ms ± 16.54ms 3202.72ms ± 9.89ms -0.59%
bench_hydration/Turbopack SSR/1000 modules 3286.11ms ± 6.03ms 3270.45ms ± 6.45ms -0.48%
bench_startup/Turbopack CSR/1000 modules 2584.51ms ± 9.56ms 2532.50ms ± 7.79ms -2.01% -0.67%
bench_startup/Turbopack RCC/1000 modules 2105.95ms ± 4.84ms 2090.89ms ± 2.24ms -0.72% -0.04%
bench_startup/Turbopack RSC/1000 modules 2047.48ms ± 3.57ms 2027.99ms ± 5.70ms -0.95% -0.05%
bench_startup/Turbopack SSR/1000 modules 2049.95ms ± 3.68ms 2011.25ms ± 4.64ms -1.89% -1.08%

@sokra sokra marked this pull request as ready for review March 7, 2023 17:54
@sokra sokra requested a review from a team as a code owner March 7, 2023 17:54
@sokra sokra requested a review from kwonoj March 7, 2023 17:54
@sokra sokra added the pr: automerge Kodiak will merge these automatically after checks pass label Mar 7, 2023
@sokra sokra merged commit c0bb681 into main Mar 7, 2023
@sokra sokra deleted the sokra/web-692-recursive-watcher-is-slow-on-os-that branch March 7, 2023 18:32
@@ -29,7 +29,7 @@ use std::{
path::{Path, PathBuf, MAIN_SEPARATOR},
sync::{
mpsc::{channel, RecvError, TryRecvError},
Arc, Mutex,
Arc, Mutex, MutexGuard,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MutexGuard is unused on Mac/Windows, giving a clippy warning.

#[cfg(not(any(target_os = "macos", target_os = "windows")))]
fn restore_if_watching(&self, dir_path: &Path, root_path: &Path) -> Result<()> {
if self.watching.contains(dir_path) {
let mut watcher = self.watcher.lock().unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a lock that's only used inside a loop. If we passed the HashSet to this method, we could lock once and iterate

Comment on lines +229 to +230
let disk_watcher = self.watcher.clone();
let root_path = self.root_path().to_path_buf();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto unused. Could we make a consistent interface between Mac and Linux so we can just make calls to empty methods?

#[cfg(not(any(target_os = "macos", target_os = "windows")))]
batched_new_paths.insert(path.clone());
}
Ok(DebouncedEvent::Remove(path)) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to remove a from the DiskWatcher in this case?

jridgewell added a commit that referenced this pull request Mar 7, 2023
mehulkar pushed a commit that referenced this pull request Mar 7, 2023
### Description

Only windows and macOS support real recursive file watchers, other OS
emulate it by walking the directory structure and watching all
directories. But that might be really slow and use up a lot of watchers,
so we don't want that. Instead we know exactly which directories are
used and can watch selectively directories when they are accessed.

* only use recursive watchers on macOS and windows
* fallback to non-recursively watch read directories on other OS

Note that this implementation still has some bugs when renaming folders,
but we can probably figure out these edge cases later...
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 25, 2024
### Description

Only windows and macOS support real recursive file watchers, other OS
emulate it by walking the directory structure and watching all
directories. But that might be really slow and use up a lot of watchers,
so we don't want that. Instead we know exactly which directories are
used and can watch selectively directories when they are accessed.

* only use recursive watchers on macOS and windows
* fallback to non-recursively watch read directories on other OS

Note that this implementation still has some bugs when renaming folders,
but we can probably figure out these edge cases later...
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

Only windows and macOS support real recursive file watchers, other OS
emulate it by walking the directory structure and watching all
directories. But that might be really slow and use up a lot of watchers,
so we don't want that. Instead we know exactly which directories are
used and can watch selectively directories when they are accessed.

* only use recursive watchers on macOS and windows
* fallback to non-recursively watch read directories on other OS

Note that this implementation still has some bugs when renaming folders,
but we can probably figure out these edge cases later...
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 1, 2024
### Description

Only windows and macOS support real recursive file watchers, other OS
emulate it by walking the directory structure and watching all
directories. But that might be really slow and use up a lot of watchers,
so we don't want that. Instead we know exactly which directories are
used and can watch selectively directories when they are accessed.

* only use recursive watchers on macOS and windows
* fallback to non-recursively watch read directories on other OS

Note that this implementation still has some bugs when renaming folders,
but we can probably figure out these edge cases later...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr: automerge Kodiak will merge these automatically after checks pass
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants