fix: fix "too many files open" error when watching with-antd example #1022

zhangpanweb · 2024-04-05T13:31:19Z

close: #991

出现的原因： with-antd 是 monorepo 中的一包，其 node_modules 会 link 到最外层 node_modules，之前的判断中，判断的是，执行目录的 node_modules 会被 ignore，但是上层并不会，导致需要 watch 的文件过多
解决方式：将上层目录的 node_modules 也加入到 ignore 范畴

Summary by CodeRabbit

新功能
- 功能更新：根据预定义模式和编译器输出路径动态生成忽略列表。
- watch_dir_recursive方法现在支持接受要忽略的目录列表。
- watch_file_or_dir方法现在接受路径数组来忽略指定路径。
- should_ignore_watch方法增强，检查忽略列表中的路径前缀。
- 优化watch_dir_recursive方法，引入新的get_ignore_list方法，根据特定条件动态构建忽略列表。
- watch_dir_recursive方法中的ignore_list参数从&str数组更改为PathBuf数组。
- 更新忽略目录的逻辑，以处理与在monorepo设置中解决“打开的文件太多”问题相关的特定情况。

coderabbitai · 2024-04-05T13:31:38Z

漫游

整体变更的高层摘要是：watch.rs 功能已更新，现在根据预定义模式和编译器输出路径动态生成忽略列表。watch_dir_recursive 方法现在接受用于忽略目录的 PathBuf 列表，而 watch_file_or_dir 方法现在接受用于忽略路径的 PathBuf 数组。此外，should_ignore_watch 方法现在检查忽略列表中的路径前缀而不是完全匹配。watch_dir_recursive 方法已重构，使用新的 get_ignore_list 方法根据特定条件动态构建忽略列表。在 watch_dir_recursive 中，ignore_list 参数已从 &str 数组更改为 PathBuf 数组。此外，忽略目录的逻辑已更新，以处理与在 monorepo 设置中解决“打开的文件过多”问题相关的特定情况。

更改

文件	摘要
crates/mako/src/watch.rs	更新以根据模式和编译器输出路径动态生成忽略列表。

诗

在代码的舞蹈中，路径现在跳跃摇摆，
被巧妙地模式所忽略。
兔子跳动，变化展现，
动态列表中的路径未知。
眼睛警惕地寻找路径前缀，
在代码的领域里，一次奇妙的调整。

Note

Pull Request Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://coderabbit.ai

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

zhangpanweb · 2024-04-05T13:42:52Z

目前的解决方式：

在非 monorepo + link npm 和 monorepo 下都有效
- 非 monorepo + link npm，link npm 源地址不在项目中，不会被 ignore，会watch到
- monorepo 下，项目在 packages 间 link，源地址都不在 node_modules下，也不会被 ignore
缺点是 node_modules 不会被 watch，webpack 下，部分 node_modules 会加到 watch 列表，如果要达到相同效果，还需要再研究一下

sorrycc · 2024-04-10T05:48:12Z

crates/mako/src/watch.rs

+            "node_modules",
+            ".DS_Store",
+            ".node",
+            self.compiler.context.config.output.path.to_str().unwrap(),


这行应该只在 root 下生效，ancestors 也加会有副作用，如果上级目录下真有用到 dist 就有问题了。

sorrycc · 2024-04-10T05:48:56Z

crates/mako/src/watch.rs

-        ignore_list.iter().any(|ignored| path.ends_with(ignored))
+        ignore_list
+            .iter()
+            .any(|ignored| path.strip_prefix(ignored.to_str().unwrap()).is_some())


之前的逻辑是判断 ends_with，为啥这里是 strip_prefix（前缀）？没看懂。

之前传入的是类似 dist 的字符串，所以判断是以 dist 结尾，现在传入的是整个路径，比如 /a/b/node_modules，所以判断是以这个开头的就忽略

sorrycc · 2024-04-10T05:49:24Z

crates/mako/src/watch.rs

+        ];
+
+        let mut dirs = vec![];
+        self.root.ancestors().for_each(|path| {


这里补个 why 的注释。

…atch

sorrycc · 2024-05-20T08:01:28Z

补充下 dumi 的场景。

xxx (e.g. example)
  node_modules
node_modules
src

sorrycc · 2024-05-20T08:58:01Z

Notes:

watch 最早的逻辑是，read_dir root，每个文件或目录基于 should_ignore_watch 忽略。

#864 为了支持 linked 的场景，更改后的逻辑是，在之前的基础上，会收集所有 module_graph 中带 package.json 模块的路径，且这些路径不能是 root 下，或者 root 的上层目录。那么会额外监听这些目录。

然后遇到的一个场景是 with-antd 的 monorepo 场景，其 node_modules 大量文件在上层，所以会 watch 到太多文件，导致「too many files open」报错，解法是忽略所有上层目录，见 PR #1022

…1022) * fix: fix "too many files open" error when watching with-antd example * fix: include dist dictionary of root's parent dictionary when watching --------- Co-authored-by: zp365238 <zp365238@antgroup.com> Co-authored-by: pshu <pishu.spf@antfin.com>

* feat: ✨ add [d]ynamic [r]equire runtime function * refactor: 🎨 dynamic import require process with interop * test: ✅ add test case * test: ✅ import context module adjust to dynamic import * release: @umijs/mako@0.4.18-canary.20240517.1 * chore: ⬆️ mako 0.4.18-canary.20240517.1 * chore: ⬆️ update pnpm-lock * typo * fix: fix "too many files open" error when watching with-antd example (#1022) * fix: fix "too many files open" error when watching with-antd example * fix: include dist dictionary of root's parent dictionary when watching --------- Co-authored-by: zp365238 <zp365238@antgroup.com> Co-authored-by: pshu <pishu.spf@antfin.com> * chore: 🔧 adjust puffin for easy profile for both long time compiling and watch mode (#1168) * chore: 🔧 adjust puffin for easy profile for long time compiling * refactor: 🎨 move profile logic to profile app * fix: decorator visitor should run before preset env (#1176) * test: ✅ add runtime assert in decorator and target to chrome 40 * fix: 🐛 move decorator before preset_env * add comment on decorators * feat: add watch.ignorePaths config (#1179) * release: @umijs/mako@0.4.18-canary.20240520.1 * chore: ⬆️ v0.4.18-canary.20240520.1 * chore: ⬆️ update pnpm-lock --------- Co-authored-by: zhangpanweb <37805064+zhangpanweb@users.noreply.github.com> Co-authored-by: zp365238 <zp365238@antgroup.com> Co-authored-by: chencheng (云谦) <sorrycc@gmail.com>

fix: fix "too many files open" error when watching with-antd example

1235453

sorrycc requested changes Apr 10, 2024

View reviewed changes

stormslowly and others added 3 commits April 15, 2024 18:05

Merge remote-tracking branch 'origin/master' into fix/fix-with-antd-w…

3589d0f

…atch

fix: include dist dictionary of root's parent dictionary when watching

df5f2f2

Merge branch 'master' into fix/fix-with-antd-watch

461d775

sorrycc approved these changes May 20, 2024

View reviewed changes

sorrycc merged commit 7b8c887 into master May 20, 2024
8 checks passed

delete-merged-branch bot deleted the fix/fix-with-antd-watch branch May 20, 2024 08:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fix "too many files open" error when watching with-antd example #1022

fix: fix "too many files open" error when watching with-antd example #1022

zhangpanweb commented Apr 5, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 5, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

zhangpanweb commented Apr 5, 2024

sorrycc Apr 10, 2024

zhangpanweb May 5, 2024

sorrycc Apr 10, 2024

zhangpanweb May 5, 2024

sorrycc Apr 10, 2024

zhangpanweb May 5, 2024

sorrycc commented May 20, 2024

sorrycc commented May 20, 2024 •

edited

Loading

fix: fix "too many files open" error when watching with-antd example #1022

fix: fix "too many files open" error when watching with-antd example #1022

Conversation

zhangpanweb commented Apr 5, 2024 • edited by coderabbitai bot Loading

Summary by CodeRabbit

coderabbitai bot commented Apr 5, 2024 • edited Loading

漫游

更改

诗

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

zhangpanweb commented Apr 5, 2024

sorrycc Apr 10, 2024

Choose a reason for hiding this comment

zhangpanweb May 5, 2024

Choose a reason for hiding this comment

sorrycc Apr 10, 2024

Choose a reason for hiding this comment

zhangpanweb May 5, 2024

Choose a reason for hiding this comment

sorrycc Apr 10, 2024

Choose a reason for hiding this comment

zhangpanweb May 5, 2024

Choose a reason for hiding this comment

sorrycc commented May 20, 2024

sorrycc commented May 20, 2024 • edited Loading

zhangpanweb commented Apr 5, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Apr 5, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

sorrycc commented May 20, 2024 •

edited

Loading