Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add minutes for meeting 5 Sep 2022 #31

Merged
merged 3 commits into from
Sep 9, 2022
Merged

doc: add minutes for meeting 5 Sep 2022 #31

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8c16e2a
doc: add minutes for meeting 5 Sep 2022
RaisinTen Sep 5, 2022
fb5f50f
Add notes
jviotti Sep 5, 2022
27c69a3
fixup! Add notes
RaisinTen Sep 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions meetings/2022-09-05.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Node.js Single-Executable Meeting 2022-09-05

## Links

* **Recording**: https://youtu.be/93cIMtkQaak
* **GitHub Issue**: https://github.com/nodejs/single-executable/discussions/19

## Present

* Anna Henningsen @addaleax
* Darshan Sen @RaisinTen
* Jean Burellier @sheplu
* Juan Cruz Viotti @jviotti
* Maël Nison @arcanis
* Tobias Nießen @tniessen

## Notes

During this call, most of the discussions revolved around the problem of
designing a general Virtual File System that would fit every necessary use
case, ranging from compression support, native add-on interoperability, etc.

The key conclusion from the call is that designing a truly universal VFS that
checks all the boxes that we care about and the ones that are probably unknown
so far is not trivial. For this reason, we will attempt to make VFS
user-defined for the SEA MVP. That way, we will have a better sandbox to play
with VFS', see what people come up with and eventually bring back these
learnings to the SEA project without getting indefinitely stuck at the VFS
phase.

To accomplish this decoupling, we hope to:

- Provide hooking capabilities to Node.js to allow developers to extend
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a really problematic thing to consider (require especially) if it will work outside a SEA.

`require()`, `fs` and related key components of Node.js to support arbitrary
virtual file systems without monkey-patching these modules like i.e.
Electron, Yarn and PKG do at the moment.
Comment on lines +33 to +36
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be clearer that "Provide hooking capabilities to Node.js" doesn't actually involve changing Node.js in any way at this time - even if non-official, we already have the "hooking capabilities" we need (monkey-patching) to implement a userland SEA.

Of course it won't the best in terms of quality / perfs, but since I think we agreed this userland idea was only to get a couple prototypes that could be used as basis for the discussion (and that it wasn't intended to be the final "deliverable"), it's good enough:

Once we have a better grasp on what the ideal VFS should look like, we will attempt to ship the SEA project with a blessed VFS implementation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. However, we would still need to hook into these things indefinitely (see the Bootstrapper component discussed in https://github.com/nodejs/single-executable/blob/main/blog/2022-08-05-an-overview-of-the-current-state.md#bootstrapper), even with a blessed implementation. And if that's the case, should we just solve the entire thing for once and for all?

As we have seen with Yarn, PKG, Electron, etc a very common reason for monkey patching fs and require is packaged Node.js applications, so in a way, we could argue this falls within our scope.

As far as I understand, even require internally uses fs (or its native bindings?) and fs relies on libuv for most things, so I wonder if hooking into the lower-level layers of fs is actually much simpler than monkey patching all the upper level functions.

I guess @addaleax and @RaisinTen might be able to confirm or debunk :P Feels at least worth a look IMHO

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. However, we would still need to hook into these things indefinitely, even with a blessed implementation

The way I imagine it, once a format is selected, the support for reading it from a VFS would be integrated inside Node (similar to how Python supports importing modules from eggs without users having to hook anywhere). We'd then get the bootstrap "for free", since by running node ./path/to/app.sea Node (which uses fs, except for a couple of places which are being worked on) would transparently treat app.sea as a folder, and execute its index.js.

As we have seen with Yarn, PKG, Electron, etc a very common reason for monkey patching fs and require is packaged Node.js applications, so in a way, we could argue this falls within our scope.

I'd say it falls within the scope of https://github.com/nodejs/loaders, a separate WG working to add customization support to import/require.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say it falls within the scope of https://github.com/nodejs/loaders, a separate WG working to add customization support to import/require.

That seems like a big stretch? By itself, fs hooking has nothing to do with module loading.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The loaders team has talked about renaming to the "customization" team and expanding their purview. Might make sense to double-check with them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arcanis

The way I imagine it, once a format is selected, the support for reading it from a VFS would be integrated inside Node (similar to how Python supports importing modules from eggs without users having to hook anywhere)

Yeah, I think we can eventually aim for that, once we have a VFS that we are confident to support long term and that works for all the scenarios we care about. Still, the "native" implementation on Node.js does require some hooking (even if it's internal hooking).


- Fix remaining issues on the Resource Injector current implementation
(https://github.com/postmanlabs/postject) for injecting multiple sections of
data. One of these sections will contain the VFS byte-stream, and the other
one will contain the bootstrap script that can be injected to `teach` Node.js
about the VFS of choice.

Once we have a better grasp on what the ideal VFS should look like, we will
attempt to ship the SEA project with a blessed VFS implementation.

We also touched on what the definition of "Single Executable Application"
means. Does it mean we will go out of our way to package every Node.js
application as a single file (embedding native add-ons, etc on the executable
itself)? Does an app that consists of the main executable along with sibling
".node" shared libraries still qualifies as a SEA? Clearly defining what an SEA
is (and is not) is key for identifying the core constrains for the ideal VFS
implementation later down the road.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the “s” is for “single”, so imo unless it results in just one file, it doesn’t qualify.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was the discussion about. The project name doesn't say "single file applications", but "single executable applications". Therefore, the question is: does a Node.js CLI application compiled to a binary + a couple of .node shared libraries qualifies? Technically yes, as it still consists of a single "executable."

Some files must live outside of the main binary to be used in all operating systems (like native add-ons), and as @zcbenz and @arcanis pointed out in other discussions, extracting them on the fly every single time can be both problematic and slow. If an app consisting of an executable + shared objects still qualifies as a SEA, then that can solve a lot of problems.

Or to ask the reverse question: what do we gain from having a single file containing the entire thing vs a single executable + shared objects?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's much easier to distribute and manage.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true that a single file its easier to distribute, however people rarely distribute standalone binary executables in the internet. Instead, even single executables are usually distributed on a ZIP, tarball, etc. One of the core reasons is that on i.e. macOS, the browser would strip the executable bit from the binary and users would need to be taught how to chmod +x.

So my thinking is: if you would zip/tar the single binary anyway, then having a few files rather than a single one doesn't really make distribution much harder.

Furthermore, having a single file representing an app with native add-ons on Windows is a no-go (#30 (reply in thread)), and it might be a hard constraint outside of our control.

The current status is: if a Node.js app has native add-ons and targets Windows, shipping a single file is not possible. So we either accommodate our definition or say that apps with native add-ons cannot be archived as an SEA.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ljharb @jviotti I've created a separate discussion for this in #34 because this is a very important topic and it might easily get lost here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!


## Action Items

- [ ] Make defining user-land VFS' feasible (given the plan outlined above)
- [ ] Define and document what SEAs really mean

## Upcoming Meetings

* **Node.js Project Calendar**: <https://nodejs.org/calendar>

Click `+GoogleCalendar` at the bottom right to add to your own Google calendar.