-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fs: add promises API #18297
fs: add promises API #18297
Conversation
@addaleax ... because I know you'll ask.. this intentionally does not do the |
My intent would be to land this as an experimental feature |
Just fyi.. benchmark comparison for bench-stat vs bench-stat-promises (running in a vm on a laptop...)
|
doc/api/fs.md
Outdated
`fs.promises.open()` method. | ||
|
||
Unlike the callback-based `fs.fstat()`, `fs.fchown()`, `fs.fchmod()`, | ||
`fs.ftruncate()`, `fs.read()`, and `fs.write()`, operations -- all of which |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it supposed to be an exhaustive list? If so, maybe fs.appendFile()
, fs.fdatasync()
, fs.fsync()
, fs.readFile()
, fs.futimes()
, and fs.writeFile()
should be added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not exhaustive, will add some clarifying language
doc/api/fs.md
Outdated
--> | ||
|
||
* Returns: {Promise} A `Promise` that will be resolved once the underlying | ||
file descriptor is closed, or will reject if an error occurs while closing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will reject -> will be rejected?
doc/api/fs.md
Outdated
* `fs.constants.X_OK` - `path` can be executed by the calling process. This has | ||
no effect on Windows (will behave like `fs.constants.F_OK`). | ||
|
||
If the accessible check is successful, the `Promise` is resolved with no |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
accessible check -> access check / accessibility check ?
doc/api/fs.md
Outdated
@@ -3446,6 +4203,7 @@ The following constants are meant for use with the [`fs.Stats`][] object's | |||
[`AHAFS`]: https://www.ibm.com/developerworks/aix/library/au-aix_event_infrastructure/ | |||
[`Buffer.byteLength`]: buffer.html#buffer_class_method_buffer_bytelength_string_encoding | |||
[`Buffer`]: buffer.html#buffer_buffer | |||
[`FileHandle`]: #fs_class_filehandle |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is [FileHandle]
(without backticks) in the signatures, so this needs to be synced.
Or this type can be added here to spare some doc bytes)
doc/api/fs.md
Outdated
|
||
The `file` may be specified as a `FileHandle` that has been opened | ||
for appending (using `fs.promises.open()`). The `FileHandle` will | ||
not be closed automatically. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean that any FileHandle
opened for appending will not be closed automatically? If so, should this be added to FileHandle
description as an exception?
doc/api/fs.md
Outdated
`bytesWritten` specifies how many _bytes_ were written from `buffer`. | ||
|
||
Note that it is unsafe to use `fs.write` multiple times on the same file | ||
without waiting for the `Promise` to resolve. For this scenario, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"to resolve" -> "to be resolved (or rejected)"?
doc/api/fs.md
Outdated
* `encoding` {string|null} **Default:** `'utf8'` | ||
* `mode` {integer} **Default:** `0o666` | ||
* `flag` {string} **Default:** `'w'` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing "Returns: ..."?
doc/api/fs.md
Outdated
The `encoding` option is ignored if `data` is a buffer. It defaults | ||
to `'utf8'`. | ||
|
||
If `options` is a string, then it specifies the encoding. Example: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Obsolete " Example:"
doc/api/fs.md
Outdated
|
||
Any specified `FileHandle` has to support writing. | ||
|
||
It is unsafe to use `fs.writeFile` multiple times on the same file without |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fs.writeFile
-> fs.promises.writeFile()
doc/api/fs.md
Outdated
waiting for the `Promise` to be resolved (or rejected). | ||
|
||
If a `FileHandle` is specified as the `file`, it will not be closed | ||
automatically. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"it will not be closed automatically" — if this is true, is it worth to be mentioned as an exception in the FileHandle
description?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point, will need to figure out the different wording here. The FileHandle
will be closed if it is allowed to gc .... but if a FileHandle
is provided to this function, the function itself will not call filehandle.close()
, it's still up to the user to do that.
|
@vsemozhetbyt ... as always I love your reviews of my doc commits. Thank you for being so thorough. Yes, the omission of the I plan to go back and add better examples either in an iteration of this PR or a future one. |
lib/fs.js
Outdated
} | ||
|
||
fs.promises = { | ||
async access(path, mode = fs.F_OK) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that you never await
anything, does this need to be async
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To catch errors thrown, yes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jasnell, can you elaborate? For my education, how does making the function async help with catching thrown errors? My understanding is that async
merely enables the use of the await
keyword.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any error thrown within an async function
is automatically caught and causes rejection of the promise returned by the function. With a regular function you could have two cases: synchronous throw or rejection depending on where the error happens.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, what @targos said... for instance, try the following cases:
function foo() {
throw new Error('foo'); // error is thrown synchronously
return new Promise(() => {});
}
async function bar() {
throw new Error('bar'); // error causes promise rejection
return new Promise(() => {});
}
Thanks for this. This is great. Can we have a shorter namespace though? Something like |
@jasnell Your benchmark run seems to suggest that the promise performance is already quite good compared to the callback version. Although that is of course a micro-benchmark. Do you have a representative (macro-)benchmark? |
cc @psmarshall |
Yep and rightfully so given that they mostly use exactly the same internal mechanisms. The differences are quite minimal thankfully. Where they are different is in the creation of the additional promise and the additional overhead of the promise hook, both of which are fairly minimal. I do not yet have a macro benchmark in place. It'll likely take bit to get that. One thing I did notice is that there are about twice as many GC runs in the promises microbenchmark, which is not at all surprising but is certainly worth noting. |
@jasnell Did you see #17739 (comment) from @gsathya? That might be worth investigating. |
src/env.cc
Outdated
cb.cb_(this, cb.data_); | ||
if (try_catch.HasCaught()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is worth updating the comment of SetImmediate
that throwing errors in a native immediate callback would crash the process.
int ret = uv_fs_close(env()->event_loop(), &req, fd_, nullptr); | ||
uv_fs_req_cleanup(&req); | ||
|
||
struct err_detail { int ret; int fd; }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not move this outside?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because it's only used here. If it turns out to be useful elsewhere later, moving it is easy :-)
src/node_file.cc
Outdated
AsyncCall(env, args, "open", UTF8, AfterOpenFileHandle, | ||
uv_fs_open, *path, flags, mode); | ||
} else { | ||
SYNC_CALL(open, *path, *path, flags, mode) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you use SyncCall
here?
src/node_file.h
Outdated
|
||
private: | ||
bool finished_ = false; | ||
double statFields_[14] {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why making it an array here though? Maybe use a unique_ptr and only allocates memory and reset it when FillStatsArray
is called?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to avoid an extra allocation. should be pretty cheap doing it this way overall.
test/parallel/test-fs-promises.js
Outdated
} | ||
|
||
const base = path.resolve(common.tmpDir, 'FOO'); | ||
assert( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this work on windows? common.tmpDir
would contain \
so it should not be able to be passed into new RegExp
like that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
heh, good point. will have to double check
I don't know if we should discuss this here but I have a concern with the interaction of fs.promises and ESM (related to #18131). |
@bmeurer ... Yeah, I'm working that way but wanted to get all of the js functions impl'd to make sure we would be able to completely eliminate creating the FSReqPromise object in js. I've got a way forward on that now that should work nicely, tho I'll have to test that it does not negatively impact the perf of the callback approach first. |
@targos ... definitely a valid concern, will have to think about that a bit more. FWIW, we would have precisely the same issue with |
@bmeurer ... updated to eliminate the creation of
|
@targos makes a great point in #18297 (comment). I'd be a definite +1 on |
@jasnell if we were implementing |
PR-URL: nodejs#18297 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Notable Changes: * crypto: - add cert.fingerprint256 as SHA256 fingerprint (Hannes Magnusson) #17690 * lib: - v8_prof_processor works again 🎉 (Anna Henningsen) #19059 * loader: - --inspect-brk now works properly for esmodules (Gus Caplan) #18949 * src: - handle exceptions in env-\>SetImmediates (James M Snell) #18297 - make process.dlopen() load well-known symbol (Ben Noordhuis) #18934 * trace_events: - add file pattern cli option (Andreas Madsen) #18480 PR-URL: #19181
Notable Changes: * crypto: - add cert.fingerprint256 as SHA256 fingerprint (Hannes Magnusson) #17690 * lib: - v8_prof_processor works again 🎉 (Anna Henningsen) #19059 * loader: - --inspect-brk now works properly for esmodules (Gus Caplan) #18949 * src: - handle exceptions in env-\>SetImmediates (James M Snell) #18297 - make process.dlopen() load well-known symbol (Ben Noordhuis) #18934 * trace_events: - add file pattern cli option (Andreas Madsen) #18480 PR-URL: #19181
SYNC_REQ has been removed. See nodejs/node#18297 nodejs/node#19041
SYNC_REQ has been removed. See nodejs/node#18297 nodejs/node#19041
PR-URL: nodejs#18297 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
PR-URL: nodejs#18297 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
The `node::fs::FileHandle` object wraps a file descriptor and will close it on garbage collection along with a process warning. The intent is to prevent (as much as possible) file descriptors from being leaked if the user does not close them explicitly. PR-URL: nodejs#18297 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Initial set of fs.promises APIs with documentation and one benchmark. PR-URL: nodejs#18297 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This seems to be slipped in nodejs#18297 PR-URL: nodejs#18599 Refs: nodejs#18309 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Khaidi Chu <i@2333.moe>
fs.Promises, the actual PR (replaces #17739)
/cc @mcollina @addaleax
/cc @bmeurer and @nodejs/v8 @nodejs/chakracore ... it would be excellent if we could have y'all take a look at this and do some performance testing.
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passesAffected core subsystem(s)
fs