Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fs: optimize fs.cpSync js calls #53614

Merged
merged 1 commit into from
Jul 22, 2024
Merged

fs: optimize fs.cpSync js calls #53614

merged 1 commit into from
Jul 22, 2024

Conversation

anonrig
Copy link
Member

@anonrig anonrig commented Jun 27, 2024 •
edited
Loading

#53612 is required to do a benchmark

A different approach to #53541

cc @nodejs/performance @nodejs/cpp-reviewers @billywhizz @jasnell @nodejs/fs

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1573/

Small win on happy path, and probably high win on error path:

                           confidence improvement accuracy (*)   (**)   (***)
fs/bench-cpSync.js n=1            ***     21.96 %       ±6.85% ±9.14% ±11.97%
fs/bench-cpSync.js n=100          ***     14.91 %       ±2.86% ±3.81%  ±4.97%
fs/bench-cpSync.js n=10000        ***      9.74 %       ±1.28% ±1.71%  ±2.23%

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 27, 2024
@anonrig anonrig mentioned this pull request Jun 27, 2024
Open
@anonrig anonrig force-pushed the optimize-cp-sync branch from 02e0883 to d7d4ae8 Compare June 27, 2024 18:44
@anonrig anonrig requested a review from lemire June 27, 2024 18:44
@anonrig anonrig force-pushed the optimize-cp-sync branch 2 times, most recently from bebf131 to 2d5fb50 Compare June 27, 2024 21:23
@anonrig
Copy link
Member Author

anonrig commented Jun 27, 2024

cc @nodejs/tsc @nodejs/fs appreciate your input on which path to take.

previous pr:

  • is a sem-ver major
  • introduces 2 different implementations (js only and cpp)
  • had limitations due to std::filesystem::copy

this pr:

  • is not a sem-ver major
  • optimizes all flags passed to cpSync
  • optimizes error cases as same as the previous pr
  • has a lot more optimization to do

@anonrig anonrig force-pushed the optimize-cp-sync branch from 2d5fb50 to 6a216bd Compare June 27, 2024 21:33
@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
mcollina
mcollina approved these changes Jun 28, 2024
View reviewed changes
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60175/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60191/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60190/

@anonrig anonrig force-pushed the optimize-cp-sync branch from 6a216bd to 371d7a7 Compare July 11, 2024 18:54
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60257/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60274/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60277/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60366/

@aduh95
Copy link
Contributor

aduh95 commented Jul 16, 2024

Windows compilation is failing:

C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\mksnapshot.vcxproj]
  embedded-file-writer.cc
  mksnapshot.cc
  embedded-empty.cc
  platform-embedded-file-writer-base.cc
  platform-embedded-file-writer-generic.cc
  platform-embedded-file-writer-win.cc
  platform-embedded-file-writer-aix.cc
  platform-embedded-file-writer-mac.cc
  snapshot-empty.cc
  static-roots-gen.cc
     Creating library ..\..\out\Debug\mksnapshot.lib and object ..\..\out\Debug\mksnapshot.exp
  mksnapshot.vcxproj -> ..\..\out\Debug\\mksnapshot.exe
  generating: "..\..\out\Debug\obj\v8_snapshot\/snapshot.cc" "..\..\out\Debug\obj\v8_snapshot\/embedded.S"
  Assembling ..\..\out\Debug\obj\v8_snapshot\\embedded.S...
  setup-isolate-deserialize.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
  snapshot.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
  v8_snapshot.vcxproj -> ..\..\out\Debug\lib\v8_snapshot.lib

@anonrig
Copy link
Member Author

anonrig commented Jul 17, 2024

@nodejs/platform-windows any idea why this build is failing?

@H4ad
Copy link
Member

H4ad commented Jul 17, 2024

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
 'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

@StefanStojanovic
Copy link
Contributor

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
 'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

@anonrig anonrig force-pushed the optimize-cp-sync branch from 371d7a7 to e2c27ae Compare July 21, 2024 21:16
@anonrig anonrig requested a review from jasnell July 21, 2024 21:17
@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
@anonrig
Copy link
Member Author

anonrig commented Jul 21, 2024

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

It seems I forgot a src_path.c_str() there. Thank you! @StefanStojanovic @H4ad

@anonrig anonrig added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 21, 2024
@anonrig anonrig force-pushed the optimize-cp-sync branch from e2c27ae to a590e84 Compare July 21, 2024 23:40
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60519/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/60528/

jasnell
jasnell approved these changes Jul 22, 2024
View reviewed changes
Copy link
Member

@jasnell jasnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still LGTM

@anonrig anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024
@nodejs-github-bot nodejs-github-bot merged commit 88027e8 into nodejs:main Jul 22, 2024
53 checks passed
@nodejs-github-bot
Copy link
Collaborator

Landed in 88027e8

targos pushed a commit that referenced this pull request Jul 30, 2024
@anonrig @targos
fs: optimize fs.cpSync js calls
1b0d5d4 
PR-URL: #53614
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@RafaelGSS RafaelGSS mentioned this pull request Jul 30, 2024
Merged
RafaelGSS pushed a commit that referenced this pull request Aug 5, 2024
@anonrig @RafaelGSS
fs: optimize fs.cpSync js calls
5e03c17 
PR-URL: #53614
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
joyeecheung
joyeecheung reviewed Aug 22, 2024
View reviewed changes
src/node_file.cc
? std::filesystem::symlink_status(src_path, error_code)
: std::filesystem::status(src_path, error_code);
if (error_code) {
return env->ThrowUVException(EEXIST, "lstat", nullptr, src.out());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks wrong, the error code should be properly translated instead of being directly mapped to EEXIST, there are many other reasons why this could fail.

src/node_file.cc
ToNamespacedPath(env, &src);
THROW_IF_INSUFFICIENT_PERMISSIONS(
env, permission::PermissionScope::kFileSystemRead, src.ToStringView());
auto src_path = std::filesystem::path(src.ToStringView());
Copy link
Member

@joyeecheung joyeecheung Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BufferValue converts the string to UTF8, on Windows this requires automatic conversion for char8_t to work which may not be supported by the version of MSVC we use, I think that may be the source of bugs like #54476

Copy link
Member

@joyeecheung joyeecheung Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually no, this is simply broken on Windows, because this just gets returned in char. It needs to be returned as std::u8string/std::u8string_view instead.

@joyeecheung joyeecheung mentioned this pull request Aug 22, 2024
Closed
@markandrus markandrus mentioned this pull request Sep 11, 2024
Open
1 task
@jazelly jazelly mentioned this pull request Sep 12, 2024
Merged
@joyeecheung joyeecheung mentioned this pull request Sep 20, 2024
Merged
@targos targos added the dont-land-on-v20.x PRs that should not land on the v20.x-staging branch and should not be released in v20.x. label Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joyeecheung joyeecheung joyeecheung left review comments

@benjamingr benjamingr benjamingr approved these changes

@jasnell jasnell jasnell approved these changes

@mcollina mcollina mcollina approved these changes

@lemire lemire Awaiting requested review from lemire

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. c++ Issues and PRs that require attention from people who are familiar with C++. dont-land-on-v20.x PRs that should not land on the v20.x-staging branch and should not be released in v20.x. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@anonrig @nodejs-github-bot @aduh95 @H4ad @StefanStojanovic @mcollina @jasnell @benjamingr @joyeecheung @targos