-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
worker: improve integration with native addons (take II) #26175
Conversation
Should have fixed the linter and Windows issues. New CI: https://ci.nodejs.org/job/node-test-pull-request/20895/ This might require some debugging on AIX. |
I'll try to look into AIX, but I might have to request ssh access for that. @gireeshpunathil Do you know anything off the top of your head that would lead to AIX-specific here? Maybe |
@addaleax: yes, AIX returns a unique value each time excerpt from the manual:
cat dl.cc #include <dlfcn.h>
#include <stdio.h>
int main(int argc, char *argv[]) {
for(int i = 0; i < 10; i++) {
void* addr = dlopen(argv[1], RTLD_NOW | RTLD_GLOBAL);
if (addr == NULL) {
fprintf(stderr, "error loading the lib\n");
return -1;
}
fprintf(stderr, "address %d: %p\n", i, addr);
}
} Linux: ./a.out foo.so address 0: 0x1ede040
address 1: 0x1ede040
address 2: 0x1ede040
address 3: 0x1ede040
address 4: 0x1ede040
address 5: 0x1ede040
address 6: 0x1ede040
address 7: 0x1ede040
address 8: 0x1ede040
address 9: 0x1ede040 AIX: address 0: 3
address 1: 4
address 2: 5
address 3: 6
address 4: 7
address 5: 8
address 6: 9
address 7: a
address 8: b
address 9: c Let me know if you want any other specific info, or assistance. |
@gireeshpunathil Thanks, that is indeed very helpful. :) I think what I’ll attempt to do for AIX is to implement wrappers for If you have other ideas, I’d be happy to hear them. |
@addaleax - I think that is a good approach. Other way is to use |
@gireeshpunathil I’ve implemented that … it still does look a bit hacky (although probably less than a Either way, if you have the bandwidth I would definitely appreciate a review from you on this PR :) |
7dae4e8
to
2a474aa
Compare
CI on musl should be fixed now, too. 🤞 |
@addaleax - sure, will do. Need some time to go through the linked issues and previous PRs. |
Allow loading add-ons from multiple Node.js instances if they are declared context-aware; in particular, this applies to N-API addons. Also, plug a memory leak that occurred when registering N-API addons. Refs: nodejs#23319
d1c6258
to
9c0f9d3
Compare
Once more: https://ci.nodejs.org/job/node-test-pull-request/20917/ |
@addaleax - I am unable to connect some links, so thought I will ask:
We are now managing the referencing count of loaded libraries. Are these maintained across:
|
@gireeshpunathil Thanks for taking a look!
Yes, that’s the error message that people currently get when they try to load a Node.js addon that uses the
For AIX, yes. Other operating systems/standard libraries already implement
That’s correct. (As mentioned above, that’s already the case on almost all platforms). The core of this patch is the fact that, based on the handle returned from
That’s true, although we can’t realistically unload addons that were loaded on the main thread for backwards compatibility (that’s what #25577 fixed).
We should not be attempting to unload addons more times than we’ve loaded them, so I’m not sure if I’m understanding this correctly?
The reference counts, both for the shared object handle (whether we maintain that ourselves on AIX, or libc does that for us) and for the
Hm – I’m not sure I understand what you’re asking here? |
I mean to say, a dll was loaded by a worker, but a second worker that comes up starts loading the same. I guess I got the answer as, it is nothing different than a main - worker scenario. Am I right? re: AIX, does this fact make anything better for you? i.e, AIX knows (internally) how many times the library load was attempted, and the unload unrolls that count. I closed more than I opened just to show that it actually remembers what it opened. $ cat main.cc #include <dlfcn.h>
#include <stdio.h>
int main(int argc, char *argv[]) {
int* cache[3];
for(int i = 0; i < 3; i++) {
cache[i] = (int*) dlopen(argv[1], RTLD_NOW | RTLD_GLOBAL);
if (cache[i]== NULL) {
fprintf(stderr, "error loading the lib\n");
return -1;
}
fprintf(stderr, "index: %d, handle: %p\n", i, (void*) cache[i]);
}
for(int i = 0; i < 5; i++) {
fprintf(stderr, "%d\n", dlclose((void*) cache[i % 10]));
}
} $ ./a.out ./libfoo.so
|
Yeah, the only difference that Worker threads vs the main thread makes is that the main thread doesn’t unload addons. So that’s why in the test here, we’re seeing 2 full load + unload cycles when using 2 Workers, but only 1 call to static constructors/destructors when using the main thread and a Worker thread.
I’m not sure… what we need is a fixed key for caching the |
@addaleax - agreed. we can't rely on undocumented opaque variables. And those handles are not even native addresses, even to attempt one. |
@gireeshpunathil Thank you! @jasnell Does your approval still stand? The PR has changed a bit since you approved, but otherwise it should be ready. :) |
In case anyone wants to reference the AIX documentation for |
Landed in 8ebd339 🎉 |
Allow loading add-ons from multiple Node.js instances if they are declared context-aware; in particular, this applies to N-API addons. Also, plug a memory leak that occurred when registering N-API addons. Refs: #23319 PR-URL: #26175 Fixes: #21481 Fixes: #21783 Fixes: #25662 Fixes: #20239 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Allow loading add-ons from multiple Node.js instances if they are declared context-aware; in particular, this applies to N-API addons. Also, plug a memory leak that occurred when registering N-API addons. Refs: #23319 PR-URL: #26175 Fixes: #21481 Fixes: #21783 Fixes: #25662 Fixes: #20239 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Notable changes: * worker: * Improve integration with native addons (#26175)
Allow loading add-ons from multiple Node.js instances if they are declared context-aware; in particular, this applies to N-API addons. Also, plug a memory leak that occurred when registering N-API addons. Refs: #23319 PR-URL: #26175 Fixes: #21481 Fixes: #21783 Fixes: #25662 Fixes: #20239 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Notable Changes * n-api: * Implement date object (Jarrod Connolly) #25917 * util: * Add compact depth mode for `util.inspect()` (Ruben Bridgewater) #26269 * worker: * Improve integration with native addons (Anna Henningsen) #26175 * MessagePort.prototype.onmessage takes arguments closer to the Web specification now (Anna Henningsen) #26082
Notable Changes * n-api: * Implement date object (Jarrod Connolly) nodejs#25917 * util: * Add compact depth mode for `util.inspect()` (Ruben Bridgewater) nodejs#26269 * worker: * Improve integration with native addons (Anna Henningsen) nodejs#26175 * MessagePort.prototype.onmessage takes arguments closer to the Web specification now (Anna Henningsen) nodejs#26082
Allow loading add-ons from multiple Node.js instances if they are
declared context-aware; in particular, this applies to N-API addons.
Also, plug a memory leak that occurred when registering N-API addons.
Refs: #23319
Fixes: #21481
Fixes: #21783
Fixes: #25662
Fixes: #20239
/cc @nodejs/workers @nodejs/n-api
Checklist
make -j4 test
(UNIX), orvcbuild test
(Windows) passes