Support Node.js embedder objects and bindings in the snapshot #37476

joyeecheung · 2021-02-22T17:36:06Z

Here is a list of embedder object classes we have in core. In order to make userland snapshot possible (#35711) we should support as many as possible of them.

Some of them can't simply be deserialized - for example, the HandleWraps, because theoretically we can't resurrect a uv_handle_t from a snapshot, at best we can restore them by creating new ones, but that involves more work (which is system-specific) than "just deserialize the object". Some of them may be helper singletons that we can just throw away at serialization time and recreate upon deserialization, and some of them may not be supposed to be supported.

The following are probably less likely to need support since they involve some kind of asynchronous state or references to opened system handles.

Another list for external references registration:

The text was updated successfully, but these errors were encountered:

joyeecheung · 2021-02-22T17:41:49Z

I'll be looking into the ones in module_wrap.h and node_contextify.h so that we can include the user modules in the snapshot. Also we need a better story for fast API objects like FastHrtime though right now we don't use fast API much in core so it's low priority.

joyeecheung · 2021-02-22T17:42:43Z

cc @nodejs/startup

devsnek · 2021-02-22T17:44:31Z

ModuleWrap is blocked on v8:10855

XadillaX · 2021-02-23T04:18:52Z

cc @nodejs/startup

So we have to do some division of labor?

joyeecheung · 2021-02-24T15:45:23Z

So we have to do some division of labor?

@XadillaX Just a FYI and to avoid stepping on each other's toes :) I suppose the majority of the list should be crossed out as invalid so there isn't actually that much work to do. Maybe it would be good to have a call about the list to figure out what should be excluded, or maybe discussing in the thread is already enough.

joyeecheung · 2021-02-24T15:50:09Z

Maybe a good place to start is to document (that is, adding comments about) exactly what these classes are and what their fields are for.

Print information relevant to snapshot building at the environment exit when NODE_DEBUG_NATIVE is set to include mksnapshot. This helps us understand what need to be fixed to build a snapshot when a script is run to completion. PR-URL: #37967 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

targos · 2021-08-09T13:33:29Z

@joyeecheung the OP probably needs an update.

jasnell · 2021-08-09T13:39:36Z

Fwiw, Blob objects likely should not be serialized. They should not live beyond the process in which they were created.

PR-URL: #40239 Refs: #38905 Refs: #37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Minwoo Jung <nodecorelab@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Put the hrtime backing store in the process methods binding data so that it can be integrated into the snapshot builder. For now we simply discard the contents of the hrtime buffer during serialization and create new buffers upon deserialization because the contents are only useful in a synchronous call. This also moves the helper function for creating V8 FastAPI methods into `Environment::SetFastMethod()` for code reuse. The v8::CFunction need to be created before the Environment is created so that we have the CTypeInfo address available for external reference registration. PR-URL: #40649 Refs: #35711 Refs: #37476 Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #40239 Refs: #38905 Refs: #37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Minwoo Jung <nodecorelab@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Put the hrtime backing store in the process methods binding data so that it can be integrated into the snapshot builder. For now we simply discard the contents of the hrtime buffer during serialization and create new buffers upon deserialization because the contents are only useful in a synchronous call. This also moves the helper function for creating V8 FastAPI methods into `Environment::SetFastMethod()` for code reuse. The v8::CFunction need to be created before the Environment is created so that we have the CTypeInfo address available for external reference registration. PR-URL: #40649 Refs: #35711 Refs: #37476 Reviewed-By: James M Snell <jasnell@gmail.com>

This patch enables internal/options to be included in the snapshot, so that when lazy loading the run time options, the modules only need to make sure that the options are queried lazily and do not have to lazy load the internal/options module together. We can still guarantee that no run time options are serialized into the state-independent bootstrap snapshot with the assertion inside GetCLIOptions(). PR-URL: #42203 Refs: #37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Bradley Farias <bradley.meck@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

To lazy load the run time options, the following properties are updated from value properties to accessor properties whose getter would turn them back to a value properties upon the initial access. - crypto.constants.defaultCipherList - crypto.pseudoRandomBytes - crypto.prng - crypto.rng PR-URL: #42203 Refs: #37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Bradley Farias <bradley.meck@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>

Instead of iterating over the bindings, iterate over the base objects that are snapshottable. This allows us to snapshot base objects that are not bindings. In addition this refactors the InternalFieldInfo class to eliminate potential undefined behaviors, and renames it to InternalFieldInfoBase. The {de}serialize callbacks now expect a InternalFieldInfo struct nested in Snapshotable classes that can be used to carry serialization data around. This allows us to create structs inheriting from InternalFieldInfo for Snapshotable objects that need custom fields. PR-URL: #44192 Refs: #37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Move util::WeakReference to a separate header and implement {de}serialization for it to be snapshotable. PR-URL: #44193 Refs: #44014 Refs: #37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

PR-URL: #44193 Refs: #44014 Refs: #37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Instead of creating an object template for every ContextifyContext, we now create one object template that can be reused by all contexts. The native pointer can be obtained through an embdder pointer field in the creation context of the receiver in the interceptors, because the interceptors are only meant to be invoked on the global object of the contextified contexts. This makes the ContextifyContext template context-independent and therefore snapshotable. PR-URL: #44252 Refs: #44014 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Include a minimally initialized contextify context in the embedded snapshot. This paves the way for user-land vm context snapshots. PR-URL: #44252 Refs: #44014 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Access to the global object from within a vm context is intercepted so it's slow, therefore we should try to avoid unnecessary access to it during the initialization of vm contexts. - Remove the Atomics.wake deletion as V8 now does not install it anymore. - Move the Intl.v8BreakIterator deletion into the snapshot. - Do not query the Object prototype if --disable-proto is not set. This should speed up the creation of vm contexts by about ~12%. PR-URL: #44252 Refs: #44014 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Instead of iterating over the bindings, iterate over the base objects that are snapshottable. This allows us to snapshot base objects that are not bindings. In addition this refactors the InternalFieldInfo class to eliminate potential undefined behaviors, and renames it to InternalFieldInfoBase. The {de}serialize callbacks now expect a InternalFieldInfo struct nested in Snapshotable classes that can be used to carry serialization data around. This allows us to create structs inheriting from InternalFieldInfo for Snapshotable objects that need custom fields. PR-URL: nodejs#44192 Refs: nodejs#37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Move util::WeakReference to a separate header and implement {de}serialization for it to be snapshotable. PR-URL: nodejs#44193 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

PR-URL: nodejs#44193 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Instead of creating an object template for every ContextifyContext, we now create one object template that can be reused by all contexts. The native pointer can be obtained through an embdder pointer field in the creation context of the receiver in the interceptors, because the interceptors are only meant to be invoked on the global object of the contextified contexts. This makes the ContextifyContext template context-independent and therefore snapshotable. PR-URL: #44252 Refs: #44014 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Include a minimally initialized contextify context in the embedded snapshot. This paves the way for user-land vm context snapshots. PR-URL: #44252 Refs: #44014 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Access to the global object from within a vm context is intercepted so it's slow, therefore we should try to avoid unnecessary access to it during the initialization of vm contexts. - Remove the Atomics.wake deletion as V8 now does not install it anymore. - Move the Intl.v8BreakIterator deletion into the snapshot. - Do not query the Object prototype if --disable-proto is not set. This should speed up the creation of vm contexts by about ~12%. PR-URL: #44252 Refs: #44014 Refs: #37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Instead of iterating over the bindings, iterate over the base objects that are snapshottable. This allows us to snapshot base objects that are not bindings. In addition this refactors the InternalFieldInfo class to eliminate potential undefined behaviors, and renames it to InternalFieldInfoBase. The {de}serialize callbacks now expect a InternalFieldInfo struct nested in Snapshotable classes that can be used to carry serialization data around. This allows us to create structs inheriting from InternalFieldInfo for Snapshotable objects that need custom fields. PR-URL: nodejs#44192 Refs: nodejs#37476 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Move util::WeakReference to a separate header and implement {de}serialization for it to be snapshotable. PR-URL: nodejs#44193 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

PR-URL: nodejs#44193 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Instead of creating an object template for every ContextifyContext, we now create one object template that can be reused by all contexts. The native pointer can be obtained through an embdder pointer field in the creation context of the receiver in the interceptors, because the interceptors are only meant to be invoked on the global object of the contextified contexts. This makes the ContextifyContext template context-independent and therefore snapshotable. PR-URL: nodejs#44252 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Include a minimally initialized contextify context in the embedded snapshot. This paves the way for user-land vm context snapshots. PR-URL: nodejs#44252 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>

Access to the global object from within a vm context is intercepted so it's slow, therefore we should try to avoid unnecessary access to it during the initialization of vm contexts. - Remove the Atomics.wake deletion as V8 now does not install it anymore. - Move the Intl.v8BreakIterator deletion into the snapshot. - Do not query the Object prototype if --disable-proto is not set. This should speed up the creation of vm contexts by about ~12%. PR-URL: nodejs#44252 Refs: nodejs#44014 Refs: nodejs#37476 Reviewed-By: Chengzhong Wu <legendecas@gmail.com>