Replace pkg with Node SEA

[jannis-baum]

Close #60

[jannis-baum]

To-Do Done!

This still needs to include the assets (static/**/*) which is not as trivial as it was with pkg. The approach that currently seems the best bet is the following:

• create an archive of all assets at build time in the makefile
• set the archive to be an asset of the SEA
• make some helpers that abstract loading the files to be the same: at run time, check if we are in SEA or not (https://nodejs.org/api/single-executable-applications.html#seaissea)
  • if not, we use the filesystem to retrieve the files
  • if we are, we retrieve the archive as a blob (https://nodejs.org/api/single-executable-applications.html#seagetassetasblobkey-options) and then load the files from there
• create express routes that use the helpers to serve the files

[jannis-baum]

Hello @tuurep! This PR replaces the deprecated pkg which we have been using so far to package Vivify into a single executable file with the new native Node SEA (Single Executable Application) which is still in beta. I have been wanting to do this because of the deprecation and a security alert that has been hanging around for pkg. As a bonus, it seems that vivify-server is smaller now (≈50MB vs ≈70MB before).

Anyways, I would appreciate if you could verify that this also works on your machine! You can check out this branch and then run make linux and you should get your vivify-server and viv executables in a build/linux directory.

In case you have the time I would also really appreciate if you could take a look at the code and general infrastructure. But no worries if not. Let me know if you want to look into it, then I can give you a quick run-down of the ideas and considerations I had here :)

PS: I invited you as a collaborator on Vivify so if you accept you should be able to leave a "real" review on this PR😊

[tuurep]

Hi, thanks

I can look at this over the weekend!

If you wanna give a run-down, I'm interested

[jannis-baum]

I can look at this over the weekend!

Thank you!

If you wanna give a run-down, I'm interested

Okay so first of all I switched out the little build script we had for a Makefile since Node SEA is a bit more work to build at the moment:

1. build/bundle.js: Node SEA ironically doesn't seem to like node_modules ATM so I added webpack to bundle everything into 1 single .js file
2. build/<platform>/vivify-server: Here we basically follow the documentation. We use node --experimental-sea-config to create a blob of our bundled code, copy the node executable from our system, and then inject the blob into the executable

Doing just this leaves us with all the code working, but the assets in the static/ directory inaccessible. Node SEA does not give us a fancy virtualized file system like pkg, instead we have to create string keys mapped to each file we want to access from the executable later, and then use sea.getAsset to retrieve it at runtime. Here, I decided to

1. Grab all files in the static/ tree and archive them in the make process
2. Include this archive as a single asset in our sea-config.json
3. At runtime, retrieve the archive as a streamable buffer and get the files from there

Since this is very different from getting the files directly from the disk, we can't use express's convenient static router anymore, so I reimplemented it manually: The StaticProvider in src/routes/static.ts abstracts the different file retrieval process between development vs. production so that we can always run StaticProvider.content(path) to serve the static file.

This concludes the main gist of the changes☺️

---

A couple of other small details:

• CI
  • ubuntu-latest uses a Node version by default that is from before SEA was a thing so I had to select Node 20 explicitly
  • Up until now, we hadn't taken care of macOS's required code signing, meaning macOS users weren't able to just download and run Vivify. This should be fixed now: We sign the executable in the Makefile. For this to work our CI has to build the macOS version on a macOS machine which makes the CI a bit more complex/lengthy (we will see if this actually works when we make a release, haha)
• I didn't find a good NodeJS library to stream the zip archive directly from a buffer so what we actually do is retrieve the archive with sea.getAsset, then save it to a temporary file on disk, and then stream it from there. When the server shuts down it is cleaned up

[tuurep]

Yeah this works on Linux with absolutely no issues! edit: wrong

Quite a lot of setup involved with the Node SEA I see - I can't really judge it other than looks good based on your explanations as well

[tuurep]

Now that we have a Makefile, do you think we could add install and uninstall targets to just copy/remove the executables on the path? :) Would be nice for development and testing.

Also I'm prepared to update the AUR PKGBUILD, but I have a feeling this will be an improvement for it/easier than when the pkg was used.

About Arch Linux build dependencies, gotta note that zip isn't installed by default.

npx comes from package npm, in case this matters.

[jannis-baum]

Yeah this works on Linux with absolutely no issues!

Quite a lot of setup involved with the Node SEA I see - I can't really judge it other than looks good based on your explanations as well

Awesome, thanks for the review!

Now that we have a Makefile, do you think we could add install and uninstall targets to just copy/remove the executables on the path? :) Would be nice for development and testing.

Hm, not sure, what directory exactly would you install it to? I put everything I have to install manually into ~/.bin which is conventionally probably not even on the $PATH. Would be great if there was some convention for having an environment variable set to the directory where you want things installed but I don't know of any, do you?

I agree that it's nice for development and testing though and while I haven't committed anything to the repo that installs Vivify because I don't know where exactly to install it to, I do have a script that does it for me. My .gitignore_global has ignoreme* on it and I keep the script ignoreme-install.sh in the repo's root directory with

#!/bin/sh

make macos && cp build/macos/* ~/.bin

and just run this to install. Unless you have an idea for where exactly to install for everyone maybe something like this could be an option for you as well :)

Also I'm prepared to update the AUR PKGBUILD, but I have a feeling this will be an improvement for it/easier than when the pkg was used.

Ah nice, thanks! I forgot, is there something you have to do on this repo to update it or is it entirely somewhere else? In case there is, you want to do that on this branch before we merge it or do you want to make a separate PR? I don't mind, feel free to decide :)

About Arch Linux build dependencies, gotta note that zip isn't installed by default.

npx comes from package npm, in case this matters.

Oh, I didn't think about this at all! I guess npx is not relevant because npm was already a dependency before, right? Concerning zip, is there another archiving util that is installed by default that you think would be worth investigating to replace zip with? I don't really mind what we use here, just needs a replacement for node-stream-zip as well if we want to change it.

[tuurep]

I believe it's (at least in a general sense) standard on linux to have these types of executables in either:

• ~/.local/bin for single user
• /usr/bin /usr/local/bin for system-wide

Though I have heard about ~/.bin as an alternative for ~/.local/bin too

But it's a good question whether this works indeed for everyone, I'm thinking I could take a look at a bunch of projects' make install and where it's conventionally put.

Yeah I've also been using a script for this:

#!/bin/bash

cp -f ./bin/linux/viv ./bin/linux/vivify-server ~/.local/bin/ \
        && echo Installed

The AUR stuff yeah it's on its own little repo and will be only relevant after the next release, so it's fine :D

Oh, I didn't think about this at all! I guess npx is not relevant because npm was already a dependency before, right?

Actually in the PKGBUILD there's only yarn as a build dependency atm so that's exactly why I was wondering. But anything can be added in there, as well.

is there another archiving util that is installed by default that you think would be worth investigating to replace zip with?

I think tar is the one, but is it applicable here?

[jannis-baum]

I believe it's (at least in a general sense) standard on linux to have these types of executables in either:

• ~/.local/bin for single user
• /usr/bin for system-wide

Though I have heard about ~/.bin as an alternative for ~/.local/bin too

But it's a good question whether this works indeed for everyone, I'm thinking I could take a look at a bunch of projects' make install and where it's conventionally put.

I checked and on macOS the only directories that are on PATH by default are the following

/usr/local/bin
/System/Cryptexes/App/usr/bin
/usr/bin
/bin
/usr/sbin
/sbin

So not really any option to install for a single user and (at least on macOS) all of those options would require sudo. If you want we can add an install for Linux, but I think we won't find a good generalizable solution for macOS.

The AUR stuff yeah it's on its own little repo and will be only relevant after the next release, so it's fine :D

Nice!

Actually in the PKGBUILD there's only yarn as a build dependency atm so that's exactly why I was wondering. But anything can be added in there, as well.

Okay, I got rid of the npx, it was just there because I copied code from the Node SEA documentation and didn't think about it. We really don't need it since it just "helps find the executables" in this case but because they are in node_modules/.bin/ anyways I hardcoded the paths now :)

I think tar is the one, but is it applicable here?

So I checked and while there are modules for it (tar-stream or tar), the APIs don't look nearly as nice as the one of node-stream-zip which we are currently using. Another option we could probably do is just use a script based on node-stream-zip to zip the archive and not rely on the zip tool itself. I don't really have a feeling for how much effort it is worth to get rid of a build dependency like zip. If you think it would be good I can look into making a node-based script to replace zip :)

[tuurep]

Hmm. Googling around seems like /usr/local/bin and sudo make install could work but yes stuff under $HOME isn't generalizable

https://unix.stackexchange.com/questions/8656/usr-bin-vs-usr-local-bin-on-linux

/usr/bin is for distribution-managed normal user programs.

/usr/local/bin is for normal user programs not managed by the distribution package manager, e.g. locally compiled packages. You should not install them into /usr/bin because future distribution upgrades may modify or delete them without warning.

This was a pretty helpful & clear read too, about bins under $HOME:

https://unix.stackexchange.com/questions/36871/where-should-a-local-user-executable-be-placed-under-home/36874#36874

I'm not sure, we could add install with /usr/local/bin, or keep using our scripts, or both? :D

I don't really have a feeling for how much effort it is worth to get rid of a build dependency like zip

Yeah, I don't have an opinion whether it's bad to have zip as a build dependency, it's probably fine. Especially if the alternative is a problem.

[jannis-baum]

I went through the Node SEA documentation again and realized I incorrectly copied an argument that they only list for macOS to the Linux injection as well. Can you try building it again like this?

[tuurep]

Nice find!

Sadly it's segfaulting the same way with this change

[jannis-baum]

Okay, I hoped this would be it, haha. Maybe you could try following the example they have on the documentation and see if that works?

[tuurep]

Yeah! I have to get back to this tomorrow. Thanks for the big efforts today.

[jannis-baum]

Thanks to you too! Having fun working on this together!

[tuurep]

Hey, I didn't have time yesterday, but going to start looking at it now.

If you've got any leads lmk, but I can start with the Node SEA docs for sure.

btw, there is this warning in the build log:

Start injection of NODE_SEA_BLOB in build/linux/vivify-server...
warning: Can't find string offset for section name '.note'
💉 Injection done!

Does that look suspicious?

[jannis-baum]

No worries! Yes, I think the docs are a good starting point. They have a full and simple example and we can see what we do next based on whether that works/how it fails :)

I haven't found any new leads unfortunately, it's hard since I haven't managed to reproduce it even on (Docker) Linux.

Hm, that warning does indeed seem suspicious. I checked and the CI-build also prints it. On macOS it doesn't happen, neither locally for me nor in the CI. However running the CI-build worked for me in the Linux container so I am not sure if that could actually be it

[jannis-baum]

Ah, speaking of Docker, in case we don't find any new leads based on you trying the documentation example another option would be for you to try out if you can run the build in the Docker setup that I used the other day

[tuurep]

Ok important lead here:

I just followed the tutorial with the hello.js example, and the resulting executable segfaults just the same way, also there's the Can't find string offset for section name '.note' error (I thought .note would have something to do with a CSS class heh)

So the problem can be entirely separated from vivify, I guess?

[tuurep]

I'm not experienced using gdb, but in case this can tell us something, i'll dump this here :D

(gdb) run
Starting program: /home/tuure/sea-test/hello
warning: section  not found in /home/tuure/.cache/debuginfod_client/a2426a7a21ac2b8be3a7ef5bf4d45535ad998cc3/debuginfo

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fd736f in _dl_relocate_object (l=l@entry=0x7ffff7ffe2e0, scope=<optimized out>,
    reloc_mode=<optimized out>, consider_profiling=<optimized out>,
    consider_profiling@entry=0) at dl-reloc.c:301
301	    ELF_DYNAMIC_RELOCATE (l, scope, lazy, consider_profiling, skip_ifunc);
(gdb) bt
#0  0x00007ffff7fd736f in _dl_relocate_object (l=l@entry=0x7ffff7ffe2e0,
    scope=<optimized out>, reloc_mode=<optimized out>, consider_profiling=<optimized out>,
    consider_profiling@entry=0) at dl-reloc.c:301
#1  0x00007ffff7fe8891 in dl_main (phdr=<optimized out>, phnum=<optimized out>,
    user_entry=<optimized out>, auxv=<optimized out>) at rtld.c:2311
#2  0x00007ffff7fe5146 in _dl_sysdep_start (start_argptr=start_argptr@entry=0x7fffffffdcb0,
    dl_main=dl_main@entry=0x7ffff7fe6cc0 <dl_main>)
    at ../sysdeps/unix/sysv/linux/dl-sysdep.c:140
#3  0x00007ffff7fe69be in _dl_start_final (arg=0x7fffffffdcb0) at rtld.c:494
#4  _dl_start (arg=0x7fffffffdcb0) at rtld.c:581
#5  0x00007ffff7fe5748 in _start () from /lib64/ld-linux-x86-64.so.2
#6  0x0000000000000001 in ?? ()
#7  0x00007fffffffe062 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb) l
296	    }
297	
298	  {
299	    /* Do the actual relocation of the object's GOT and other data.  */
300	
301	    ELF_DYNAMIC_RELOCATE (l, scope, lazy, consider_profiling, skip_ifunc);
302	
303	#ifndef PROF
304	    if ((consider_profiling || consider_symbind)
305		&& l->l_info[DT_PLTRELSZ] != NULL)
(gdb)

[jannis-baum]

Hm, interesting. I am unfortunately also not experienced with low level debugging. The only thing I get from that is that it seems it has something to do with shared libraries that are trying to be loaded and the _dl_relocate_object function. Googling that didn't get me anywhere.

nodejs/single-executable#46 (comment) says that

Node.js links to GLIBC. Depending on the official release configuration, building against too new GLIBC versions reduces Linux compatibility

Could this be the issue? maybe your GLIBC is not compatible with whatever would be required by the Node version(s) we have tried?

Other than that the only thing I could think of would be opening an issue on nodejs/single-executable.

[jannis-baum]

I looked around the shared library thing a bit more, seems like Vivify depends on these:

libdl.so.2
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libpthread.so.0
libc.so.6
ld-linux-x86-64.so.2

With ldconfig -p | grep <library> you can check where they link on your system. Maybe you can get somewhere with that, I don't know🙈

Edit You can get the required shared libraries with objdump -p <path/to/executable> | grep NEEDED

[tuurep]

Eh, I was about to test running in docker as you suggested, but found something in the meanwhile:

The CI-built executables do actually work for me (as linked before: https://github.com/jannis-baum/vivify/actions/runs/9841072109)

Not sure why I thought earlier they didn't.

Is this with the latest commit, and how's it built compared to manual?

As for the linker related stuff, when I run objdump -p ./vivify-server | grep NEEDED on the CI built bin, I get this:

$ objdump -p ./vivify-server | grep NEEDED
  NEEDED               libdl.so.2
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6
  NEEDED               ld-linux-x86-64.so.2

But same on make linux built bin:

objdump -p ./vivify-server | grep NEEDED
  NEEDED               libz.so.1
  NEEDED               libuv.so.1
  NEEDED               libbrotlidec.so.1
  NEEDED               libbrotlienc.so.1
  NEEDED               libcares.so.2
  NEEDED               libnghttp2.so.14
  NEEDED               libcrypto.so.3
  NEEDED               libssl.so.3
  NEEDED               libicui18n.so.75
  NEEDED               libicuuc.so.75
  NEEDED               libstdc++.so.6
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  NEEDED               ld-linux-x86-64.so.2

Seems wildly different. What could that tell us?

[jannis-baum]

The CI-built executables do actually work for me (as linked before: https://github.com/jannis-baum/vivify/actions/runs/9841072109)

Not sure why I thought earlier they didn't.

Awesome! I don't know how you tried it the first time but maybe somehow it still tried starting a version you compiled on your machine instead? E.g. if you used the viv executable from the CI-build but still had your own compilation on your PATH?

Is this with the latest commit, and how's it built compared to manual?

Not sure which commit exactly that was from but I think this should mean that they probably all work☺️ The build is the same; it just runs make linux in the CI. I guess the thing that is different though is the node executable of the CI container vs. the one you have.

As for the linker related stuff, when I run objdump -p ./vivify-server | grep NEEDED on the CI built bin, I get this:

...

But same on make linux built bin:

...

Seems wildly different. What could that tell us?

I think this should mean that (for whatever reason) your node executable links to different shared libraries, which is probably also what breaks the SEA in the end. I would assume that if you ran the objdump command on your raw node executable you will also get something different from what you get if you run it on the node executable the CI container has. I have no idea why it's like this, my only guess is that it has something to do with your GLIBC versions and/or your node executable was built in a different way (it seems to not depend on libpthread.so.0 and libdl.so.2 which may be necessary for SEA).

All of this was just my wild speculation though ^^

---

Anyways, since we have found that this isn't Vivify's issue and that you can run the CI builds I would say we can merge this, right? For local development you can always use yarn dev anyways, and if you want to install a version that isn't released yet, you can just always use the CI build since I made it expose the build artifacts on every run now :) Does that sound good?

I'll just make one tiny change to the CI to get rid of a deprecation warning and then we can merge this if you don't have anything else🥳 Let me know!

[tuurep]

Yeah, sadly this breaks the AUR package (or rather I'll have to mark it outdated for now)

The packages on the AUR are merely "build scripts", i.e. recipes to build binaries for pacman.

So is it possible to get the CI build for a local branch without having a PR open for it?

[jannis-baum]

Does the AUR package always have to be built on the user's machine or is there also some way to distribute compiled binaries?

So is it possible to get the CI build for a local branch without having a PR open for it?

If you push the branch to the repo I think the CI will run regardless of whether there is a PR. I can check in a bit, but it case it's not like that at the moment we can easily make it do that. On local branches that are not pushed to the repo however the CI of course can't run

[tuurep]

AFAIK in that case it should be a new AUR package with the name vivify-bin, looks like this is convention: https://aur.archlinux.org/packages?K=-bin&SB=p&SO=d

I could try setting that up as well.

If you push the branch to the repo I think the CI will run regardless of whether there is a PR

Ok this definitely softens the blow!

Well, hesitantly, I think we could merge but there's this looming unsolved problem still 😄 I'll consider making an issue at the node SEA repo.

Since the SEA is at an alpha stage, perhaps things will settle with time as well.

[jannis-baum]

AFAIK in that case it should be a new AUR package with the name vivify-bin, looks like this is convention: https://aur.archlinux.org/packages?K=-bin&SB=p&SO=d

I could try setting that up as well.

Nice! Then the convention is to have two packages, one for building yourself and one that is a binary?

If you push the branch to the repo I think the CI will run regardless of whether there is a PR

Ok this definitely softens the blow!

I checked, right now it will only run on branches that have a PR to main open. Should I set it to run on all branches then?

Well, hesitantly, I think we could merge but there's this looming unsolved problem still 😄 I'll consider making an issue at the node SEA repo.

Since the SEA is at an alpha stage, perhaps things will settle with time as well.

Yes, I think opening an issue there is a good idea. Maybe it'll help make it more stable in the end, haha

---

Edit I would suggest we merge this then (potentially after setting up the CI to run on all branches if you prefer that). I think our user base is small enough to risk using experimental software😂 So far I haven't had any problems with the binary🤞

---

Edit 2 I set the CI to always run :) will merge now. We can wait until we figure out what to do about the AUR package before we make the next release

[tuurep]

Nice! Then the convention is to have two packages, one for building yourself and one that is a binary?

Yeah, as an example for Zotero in AUR:

• https://aur.archlinux.org/packages/zotero
• https://aur.archlinux.org/packages/zotero-bin

A third variant is a -git package, which is very useful - it builds based on the latest commit regardless of releases

• https://aur.archlinux.org/packages/zotero-git

But I've felt no need because of how frequently you tend to release 😄

[tuurep]

Btw I have a couple of unrelated feature ideas that I'd like to ask about 😄 but it's very offtopic. Should I ask away here, or in discussions or somewhere?

[jannis-baum]

Btw I have a couple of unrelated feature ideas that I'd like to ask about 😄 but it's very offtopic. Should I ask away here, or in discussions or somewhere?

Nice! Yes, discussion sounds good :) I think this PR has enough messages now haha

[tuurep]

A Discussions tab isn't open on this repo, but it made sense to create an issue and add the other idea to an existing one.

Edit: oh, it is now 😄 but it's okay, maybe better this way