Demonstrating Native Addons

[CMCDragonkai]

Description

In helping solve the snapshot isolation problem in MatrixAI/js-db#18, we needed to lift the hood and go into the C++ level of nodejs.

To do this, I need to have a demonstration of how native addons can be done in our demo lib here.

There are 2 ecosystems for building native addons:

• prebuild
• node-pre-gyp

Of the 2, the prebuild ecosystem is used by UTP and leveldb. So we will continue using that. Advantages from 2016 was commented here: prebuild/prebuild#159

The basic idea is that Node supports a "NAPI" system that enables node applications to call into C++. So it's a the FFI system of NodeJS. It's also a bidirectional FFI as C++ code can call back into the NodeJS JS functions.

The core library is node-gyp. In the prebuild ecosystem is wrapped with node-gyp-build, which you'll notice is the one that we already using in this repo. The main feature here is the ability to supply prebuilt binaries instead of expecting the end-user to always compile from source.

Further details here: https://nodejs.github.io/node-addon-examples/build-tools/prebuild (it also compares it to node-pre-gyp).

The node-gyp-build has to be a dependency, not devDependencies, because it is used during runtime to automatically find the built shared-object/dynamic library and to load it.

It looks like this:

import nodeGypBuild from 'node-gyp-build';
const bindings = nodeGypBuild('./path/to/dir/containing/gyp/file');
bindings.someNativeFunction()

Internally nodeGypBuild ends up calling the require() function inside NodeJS. Which supports the ability to load *.node binaries (which is the shared-object that is compiled using the NAPI C++ headers). See: https://github.com/prebuild/node-gyp-build/blob/2e982977240368f8baed3975a0f3b048999af40e/index.js#L6

The require is supplied by the NodeJS runtime. If you execute the JS with a different runtime, they may support the commonjs standard, and thus understand the require calls, but they may be compatible with native modules that are compiled with NAPI headers. This is relevant since, you also have to load the binary that matches your OS libraries and CPU architecture. It's all dynamic linking under the hood. This is also why you use node-gyp-build which automates some of this lookup procedure.

As a side-note about bundlers. Bundlers are often used part of the build process that targets web-platforms. Since the web platform does not understand require calls, bundlers will perform some sort of transclusion. This is also the case when ES6 import targets files on disk. Details on this process is here: https://github.com/evanw/esbuild/blob/master/docs/architecture.md#notes-about-linking. Bundlers will often call this "linking", and when targetting web-platforms, this is basically a form of static linking since JS running in browsers cannot load JS files from disk. This is also why in some cases, one should replace native addons with WASM instead, as bundlers can support static linking of WASM (which are cross-platform) into a web-bundle. But some native addons depend on OS features (like databases with persistence), and fundamentally cannot be converted into WASM binaries. In the future, our crypto code would make sense to turn into WASM binaries. But DB code is likely to always be native, as they have to be persistent. As the web develops can gains extra features, then eventually it may be possible that all native code can be done via WASM (but this may be a few years off).

Now the native module itself is just done with a C++ file like index.cpp. We should prefer using .cpp and .h as the most portable extensions.

Additionally, there must be binding.gyp file that looks like this:

{
  "targets": [{
    "target_name": "somename",
    "include_dirs": [
      "<!(node -e \"require('napi-macros')\")"
    ],
    "sources": [ "./index.cpp" ]
  }]
}

Basically another configuration file that configures node-gyp and how it should be compiling the C++ code. The target_name specifies the name of the addon file, so the output result will be somename.node. The sources are self-explanatory. The include_dirs entries have the ability to execute shell commands, in this case, it is using node -e to execute a script that will return some string that is a path to C++ headers that will be included during compilation.

The C++ code needs to use the NAPI headers, however there's a macro library that makes writing NAPI addons easier: https://github.com/hyperdivision/napi-macros. I've seen this used in the utp-native and classic-level.

The C++ code may look like this:

#include <node_api.h>
#include <napi-macros.h>

NAPI_METHOD(times_two) {
  NAPI_ARGV(1)
  NAPI_ARGV_INT32(number, 0)

  number *= 2;

  NAPI_RETURN_INT32(number)
}

NAPI_INIT() {
  NAPI_EXPORT_FUNCTION(times_two)
}

This ends up exporting a native module containing the times_two function that multiples a number by 2, and returns an int32 number.

It's also important that node-gyp-build is setup as a install script in the package.json:

  "scripts": {
    "install": "node-gyp-build"
  }

This means when you run npm install (which is used to install all the dependencies for a NPM package, or to install a specific NPM package), it will run the node-gyp-build durin the installation process.

This means that currently in our utils.nix node2nixDev expression still requires the npm install command. This used to exist, however I removed it during MatrixAI/TypeScript-Demo-Lib#37 thinking it had no effect. But it was confirmed by svanderburg/node2nix#293 (comment) that the npm install command is still run in order to execute build scripts. And node-gyp-build is now part of the installation process. We should include: https://github.com/svanderburg/node2nix/blob/8264147f506dd2964f7ae615dea65bd13c73c0d0/nix/node-env.nix#L380-L387 with all the necessary flags and parameters too. We may be able to make it work if we hook our build command prior to npm install. I imagine that this should be possible since the npm rebuild command is executed prior. So we need to investigate this.

In order to make this all work, our Nix environment is going to need all the tools for source compilation. Now according to https://github.com/nodejs/node-gyp#on-unix we will need python3, make and gcc. Our shell.nix naturally has make and gcc because we are using pkgs.mkShell which must extend from stdenv.mkDerivation. However python3 will be needed as well.

The node2nix has some understanding of native dependencies (this is why it also brings in python in its generated derivation svanderburg/node2nix#281), and I believe it doesn't actually build from source (except in some overridden dependencies).

Some npm dependencies are brought in via nixpkgs nodePackages because node2nix derivation isn't enough to build them (because they have complex native dependencies). Such as node-gyp-build itself or vercel's pkg. This is also why I had to provide nodePackages.node-gyp-build in our buildInputs overrides in utils.nix. It is important that any dependencies acquired via nixpkgs must be the same version we use in our package.json. And this is the case for:

    "node-gyp-build": "4.4.0"
    "pkg": "5.6.0",

Ideally we won't need to do this our own native packages if js-db ends up forking classic-level or leveldown. I think this trick is only relevant in our "build tools" and not our runtime dependencies.

The remaining problem is cross-compilation, as this only enables building from source if you are on NixOS and/or using Nix. Windows and MacOS will require their own setup. Since our development environment is all Nix focused, we don't have to worry about those, but for end-users who may want to rebuild from scratch, they will need to setup their development environent based on information in https://github.com/nodejs/node-gyp. A more pressing question is how we in our Nix development environment will be capable of cross-platform native addons for distribution.

This is where the prebuild ecosystem comes in and in particular https://github.com/prebuild/prebuildify-cross. This is used in leveldb to enable them to build for different platforms, and then save these cross-compiled objects. These objects are then hosted on GitHub releases, and automatically downloaded upon installation for downstream users. In the case they are not downloadable, they are then built from source. https://github.com/Level/classic-level/blob/f4cabe9e6532a876f6b6c2412a94e8c10dc5641a/package.json#L21-L26

However in our Nix based environment, I wonder if we can avoid using docker to do cross compilation, and instead use Nix to provide all the tooling to do cross-compilation. We'll see how this plays out eventually.

Some additional convenience commands now:

# install the current package and install all its dependencies, and build them ALL from source
npm install --build-from-source
# install a specific dependency and build it from source
npm install classic-level --build-from-source
# runs npm build on current package and all dependencies, and also recompiles all C++ addons
npm rebuild
# runs npm build on current package and all dependencies, and specifically recompiles sqlite3 package which has C++ addon
npm rebuild --build-from-source=sqlite3

Issues Fixed

• Related Migrating/Integrating WebCrypto (possibly WASM) instead of Node Forge Crypto Polykey#270
• Related React Native and Mobile OS (iOS and Android) Compatibility Polykey#155
• Related Using QUIC/HTTP3 to replace utp-native for the Data Transfer Layer in the networking domain Polykey#234
• Related Node2nix uses nodejs.src for --nodedir when it can just use the nodejs svanderburg/node2nix#295
• Related Add support for distributing prebuilds as platform-specific packages prebuild/prebuildify#63 - this PR introduces the ability for optionalDependencies to contain platform-specific native addon packages, which the parent package relies on, this can avoid having to put all platform-specific native addons into a single prebuild directory that is installed from npm.
• Related Add support for finding prebuilds in platform-specific packages prebuild/node-gyp-build#45 - this adds support for node-gyp-build to find the prebuilt binaries that come from optional dependencies
• https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/ - When we want to deal with ARM
• https://amir.rachum.com/blog/2016/09/17/shared-libraries/ - How runpath works.
• The mkShell should set NIX_NO_SELF_RPATH = true; by default NixOS/nixpkgs#173025
• https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary - How to build universal binaries
• Executables within app.asar.unpacked are not being signed correctly in OSX 10.14.5 electron-userland/electron-builder#3940 (comment) - Relevant information for electron entitlements and useful packages
• https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow - Notarization of mac binaries
• When reinstalling a package, chocolatey always deletes download cache chocolatey/choco#2698 - choco cache is not a download cache

Tasks

• 1. Integrate node-gyp-build
• 2. Create a native module exporting a demo functions like addOne for primitives and setProperty for reference-passing procedure and makeArray for heap allocation
• 3. Fix the nix expressions to support node-gyp-build and other build scripts, and see if we can eliminate our postInstall hook, by relying on package.json hooks instead
• 4. Integrate prebuildify to precompile binaries and host them on our git release... but this depends on whether typescript-demo-lib is used as a library or as an application, if used as an application, then the pkg builds is used, if used as a library, then one must install the native binary from the same github release, this means the native binary must be part of the same release page.
  • The pkg integration may just be a matter of setting the assets path in package.json to the local prebuilds directory.
  • See the scripts that other projects used WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
  • Ensure that compiled native addons in nix have their rpath removed, because nodejs addons shouldn't have an rpath set, and this enables them to be portable
• [ ] 5. Cross compilation, prebuildify-cross or something else that uses Nix - we must use CI/CD to do cross compilation (not sure about other architectures like ARM)
• 6. Update the @typescript-eslint packages to match js-db to avoid the warning message.
• 7. Add typescript typings to a native module
• [ ] 8. Update README.md to indicate the 2 branches of typescript-demo-lib, the main and the native branch, where the native branch indicates how to build native addons - this will be done in a separate repo: https://github.com/MatrixAI/TypeScript-Demo-Lib-Native based off https://gitlab.com/MatrixAI/Employees/matrix-team/-/issues/8#note_885403611
• 9. Migrate changes to https://github.com/MatrixAI/TypeScript-Demo-Lib-Native and request mac access to it the repository on gitlab. This branch will just be for development first. The changes here are too significant to keep within the same template repository.
• 10. The pkg bundle can receive optimisation on which prebuild architectures it bundles, right now it bundles all architectures, when the target architecture implies only a single architecture is required. This can slim the final output pkg so it's not storing random unnecessary things. This may mean that pkg requires dynamic --config to be generated.
• 11. See if nix-build ./release.nix -A application can be use prebuilds/ directory as well, as this can unify with pkg. That way all things can use prebuilds/ directory. But we would want to optimise it with task 10.
• [ ] 12. Ensure that npm test can automatically run general tests, and platform-specific tests if detected on the relevant platform - this can be done in polykey as a script
• 13. Automatic npm publish for prerelease and release based on staging and master branches, add these to the CI/CD jobs
• 14. Ensure that integration CI/CD jobs are passing by running the final executable with all the bundled prebuilt binaries

Future Tasks

• Cross compilation for win-arm64, linux-arm64 (linux will require the necessary nix-shell environment)
• Android and iOS builds for library
• Consider using Nix directly on MacOS instead of homebrew, need to figure out how xcode works with nix, nix is likely to be faster than homebrew - MacOS Runner Nix Integration for Platform-Specific Build #1
• Use gitlab shared arm64 runners to do integration tests when it becomes available
• If nodejs and vercel/pkg has universal binary become available, then use that instead of producing arm64 and x64 executable bundles
• Check if chocolatey fixes their caching problems so that we can finally make use of it: WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
• Get the arm64 and x64 binaries for macos signed using either ldid or codesign WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
• Further optimise the pkg bundling script so that it doesn't bundle useless .md files, right now it's even bundling the CHANGELOG.md files WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
• Consider test load balancing as an alternative to nominal test splitting, it might end up being faster and work with the higher overhead of having windows and mac runners
• Fix the windows integration job script that doesn't fail even when the underlying executable fails WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
• Attempt using MacOS pkg instead of zip archives so you can do stapling and therefore not require the client systems to have access to the internet before running the executable: WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
• Automate MacOS code signing for both x64 and arm64 application at integration:macos job - WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
• Figure out how to do platform-specific grouping of tests for npm test (it should automatically understand how to conditionally test these things by loading files appropriately in the right platform, or just a script that knows): https://stackoverflow.com/questions/50171932/run-jest-test-suites-in-groups

Final checklist

• Domain specific tests
• Full tests
• Updated inline-comment documentation
• Lint fixed
• Squash and rebased
• Sanity check the final build

[ghost]

👆 Click on the image for a new way to code review

• Make big changes easier — review code in small groups of related files

• Know where to start — see the whole change at a glance

• Take a code tour — explore the change with an interactive tour

• Make comments and review — all fully sync’ed with github

  Try it now!

Legend

[CMCDragonkai]

The npm publish problem is a bit more complicated. So the situation is that the dev still needs to do npm version prerelease/prepatch/premajor/preminor --preid alpha if they want to produce a prerelease.

That should create a git tag.

Then when we push this git tag, a job called build:prerelease will run that will perform the npm publish --tag staging. The --tag here is a bit a strange thing because it makes it seem this only runs in staging.

Technically this was meant to be like this, however, you can submit tag pointing to any commit, it doesn't have to be pointing to a staging branch.

Ideally, upon a commit coming into staging branch, and if that commit had a tag pre-release associated, then we would perform the build:prerelease job.

However there's no actual good rules for triggering something like this. What we could do is run a job that checks whether to do a prerelease by checking if the commit also has a tag. It seems like this should be possible, as long as the git repo also has the tag information when cloned in.

The other alternative is to run the job any time a prerelease tag occurs. This is basically how I started, but the prerelease job has needs on build:linux, build:windows, build:macos. And these along with the check:lint were all manual for tags. And that basically meant the job was blocked waiting for manual running of the final pipeline for prerelease. This of course doesn't make sense since it should be done automatically.

[CMCDragonkai]

One of the problems with relying on commits is that pipelines are interruptible. So right now, if a commit that is meant to mean some sort of release is pushed, that may be hidden in other commits that interrupt it.

Currently Gitlab CI doesn't have a way of having conditional interruption: https://gitlab.com/gitlab-org/gitlab/-/issues/194023

This means we have to use tags.

However the previous reason why tagging wasn't working before was because I had it limited to protected refs, and I had to temporarily disable protection on these refs in order to test out these different ways of doing things.

[CMCDragonkai]

The only downside to having tag pipelines is that it doubles up the pipeline when a commit is submitted too. That is since npm version command produces both a tag and commit, we performing the pipeline twice.

So have a tradeoff here. Either we allow a pipeline to run twice which is a bit of a waste, or we allow for the possibility of skipped prerelease commits. It isn't just when another commit is pushed, but one may perform npm version then do another commit on top before pushing altogether.

[CMCDragonkai]

I believe for the second condition we will need to only perform the task on the tag pipeline. As for the first problem, we would need to add something that checks if the commit is associated to a release commit, and if so, skip it.

[CMCDragonkai]

This was accidentally closed due to merge into staging. Need to reopen.