vm.compileFunction(source) is much slower than new vm.Script(source)

[SimenB]

• Version: v14.12.0
• Platform: Darwin Simens-MacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64
• Subsystem: vm

What steps will reproduce the bug?

See https://github.com/SimenB/vm-script-vs-compile-function and run node index.js.

It's just yarn init -2 to fetch the bundled yarn v2 source code (to have a large JS file), then loading that into new Script and compileFunction 100 times and seeing how long it takes.

Running this prints the following on my machine (MBP 2018):

Running with Script took 50 ms
Running with compileFunction took 4041 ms

How often does it reproduce? Is there a required condition?

Every time

What is the expected behavior?

It should be comparably fast to pass source code into compileFunction as it is to pass it into new Script.

What do you see instead?

About 80x worse performance. It gets worse the larger the loop is, so I'm assuming it's "simply" some cached data so new Script avoids the hit of parsing the same code multiple times, as it's fairly constant regardless of loop iterations

Additional information

This might be the same as #26229.

As mentioned in #26229 (comment) we see a huge performance improvement in Jest if we switch from compileFunction to the "old" new Script approach. As also mentioned there, this might be "fixed" (or at least possible to solve in userland) by the seemingly stalled #24069?

[SimenB]

I added a quick and dirty produceCachedData, and it went down to about 180 ms. Which is a great improvement, albeit still 5x slower than new Script. And harder to manage (I'd prefer the implicit caching of the new Script APIs)

[devsnek]

cc @nodejs/v8, this might not be something fixable on node's side.

[bnoordhuis]

This is about v8::ScriptCompiler::CompileFunctionInContext() not using the code cache, right?

I don't think that's V8's issue to fix. V8 can't know whether it's safe to use the cache because the argument names and context extensions affect how the function is parsed.

Not that it's easy for Node.js. options.contextExtensions is a free-form list of mutable objects. Hard to cache on that unless object identity is good enough - but it probably isn't.

What Node.js (or perhaps V8) could optimize for is the presumably common case of no context extensions. TBD how to cache without introducing memory leaks.

[SimenB]

In Jest's case we'll be passing a different parsingContext all the time (Jest uses a different Context per test file for sandboxing), so if the cached data is dependent on the Context we'd probably still get a painful performance regression vs using vm.Script. If it's not possible to get caching based on the source code string (and the arguments) alone, we'll probably have to drop usage of compileFunction and stay on vm.Script.

Using the args array to vary the cache is no issue for our use case, though - 99% of the time they'll be the same, and for the cases it's not I guess it'd just create some more? Same as we have today with the manual function(some, args) {USER_SOURCE_CODE_HERE} wrapper we add which changes its source code via the arguments we pass in the manual function wrapper.

(That said, improving performance for compileFunction in the case of not passing parsingContext seems like a good thing even if it wouldn't be enough for Jest's use case 🙂)

[bnoordhuis]

If it's not possible to get caching based on the source code string (and the arguments) alone

I don't think that can work so you're probably better off sticking with vm.Script, yes. Argument names aren't insurmountable but context extensions are.

When you pass in context extensions, what's evaluated looks like this in pseudo code:

// assume a = { y: 42 } and b = { z: 1337 }
with (a)
  with (b)
    function f(x) {
      return x * y * z
    }

I say "pseudo code" but in fact it's really close to the runtime representation.

[SimenB]

@bnoordhuis reading your comment more closely, I see you're talking about contextExtensions. We don't use that, we use parsingContext - does that have the same limitations you think?

[devsnek]

I think this issue should be upstreamed.

[rthreei]

Fwiw, Node 16 has both Script and compileFunction performing similarly:

rishi vm-script-vs-compile-function % nvm use 14
Now using node v14.18.2 (npm v6.14.15)

rishi vm-script-vs-compile-function % node index.js
Running with Script took 51 ms
Running with compileFunction took 4379 ms
Running with compileFunction and cached data took 264 ms

rishi vm-script-vs-compile-function % nvm use 16
Now using node v16.13.1 (npm v8.1.2)

rishi vm-script-vs-compile-function % node index.js
Running with Script took 4280 ms
Running with compileFunction took 4350 ms
Running with compileFunction and cached data took 321 ms

[devsnek]

yes... in more recent versions of v8 the compilation cache is effectively broken. see v8:10284

[SimenB]

Just as a follow up here, these are the numbers I'm seeing with the latest 20.8 (nothing new, just posting to say it's still an issue).

$ nvm run 16.10 index.js
Running node v16.10.0 (npm v7.24.0)
Running with Script took 19 ms
Running with compileFunction took 1799 ms
Running with compileFunction and cached data took 145 ms

$ nvm run 16.11 index.js
Running node v16.11.1 (npm v8.0.0)
Running with Script took 1790 ms
Running with compileFunction took 1798 ms
Running with compileFunction and cached data took 147 ms

$ nvm run 20 index.js
Running node v20.8.0 (npm v10.1.0)
Running with Script took 1804 ms
Running with compileFunction took 1826 ms
Running with compileFunction and cached data took 141 ms

[joyeecheung]

I think we can at least not have any host-defined options at all when the importModuleDynamically isn't used. That should make it possible for the cache to get hit again in that case at least.

The performance hit would comeback again if you intent to support import() though, because until https://bugs.chromium.org/p/v8/issues/detail?id=10284 gets fixed, the host defined options is going to be serialized as part of the script, and since you might still want different host-defined options for different script even if they have the same source, V8 needs to reject the cache. The fix would be to either move the host defined options to somewhere more sensible (as proposed by the V8 issue initially), or as later comments in that issue suggested, just don't serialize it and provide some kind of callback for the embedder to compute the host-defined options (personally I'd prefer that, I think that provides more flexibility).

[SimenB]

Ooh, any workaround, even with caveats, would be awesome! 😃

The performance hit would comeback again if you intent to support import()

We do support that, but I believe that only happens if vm Modules are active (which is behind a flag)? If it's not behind a node flag, I'm happy to put it behind a Jest flag or some such so people who don't need it won't get the perf hit.

[joyeecheung]

I opened #49950 to address the "no import() needed" case which is a fairly simple change and can backport to earlier release lines (v20.x at least, not 100% sure about v18.x yet). Locally the in-isolate cache is getting hit again (I used the repro from OP but switched the script being compiled to test/fixtures/snapshot/typescript.js which is also big).

❯ ./node_main bench.js
Running with Script took 5428 ms
Running with compileFunction took 5332 ms
Running with compileFunction and cached data took 1010 ms

❯ ./node_pr bench.js
Running with Script took 56 ms
Running with compileFunction took 5369 ms
Running with compileFunction and cached data took 1013 ms

[SimenB]

Amazing! ♥️

[SimenB]

Can confirm the PR above works - gives these results on my machine with the repo from the OP

Running with Script took 20 ms
Running with compileFunction took 1804 ms
Running with compileFunction and cached data took 136 ms

It does seem I have to make sure to not pass in importModuleDynamically tho - I'll make sure to add a flag for that in Jest 30.

[csvan]

This is huge @joyeecheung ,thank you so much ❤️

[rluvaton]

So I did git bisect for v8 and the changes that cause this are v8 commit d2fd132bcbd4d069aacbe3ce7c8bb9bda93a7d88

I'll see what I can do

[joyeecheung]

So #50137 has landed, which means now if users compile a vm.Script with importModuleDynamically but without --experimental-vm-modules, the cache can still be hit if the code does not actually invoke import(). There are still some work that I think can be done:

1. The current way of "one callback (closure) per compilation call" doesn't seem to be really necessary for the use cases that I've seen. We should be able to provide a way for users to supply "one callback (closure) for a set of compilation call", thus allowing the set of calls sharing that callback (which needs one particular host-defined option) to hit the cache, even when import() is called and --experimental-vm-modules is used to implement a non-dummy importModuleDynamically.
2. Deal with the host defined options in the cache so that even the "one callback (closure) per compilation call" case can hit the cache i.e. address vm.compileFunction(source) is much slower than new vm.Script(source) #35375 (comment) - however this may not be that rewarding compared to the cost of the refactoring it needs in V8, I think 1 is still a nicer solution.
3. Regarding the specific issue reported by the OP - vm.compileFunction(source) is slow compared to new vm.Script(source), I think this is caused by the lack of support for in-isolation compilation cache of CompileFunction, and that seems to be implementatble (CompileFunction is a later-added V8 API for Node.js to address the "wrappers showing up in the debugger" problem, that's probably why support for in-isolation compilation cache was neglected). I am looking into implement that to bring the performance of vm.compileFunction(source) on par with new vm.Script(source) in repeated calls (also this should be useful to avoid the re-compilation cost in ESM detection mentioned in esm: detect ESM syntax in ambiguous JavaScript #50096 (comment)).

[joyeecheung]

WIP in https://chromium-review.googlesource.com/c/v8/v8/+/4962094 to add in-isolate compilation cache support to vm.compileFunction(), locally I get

❯ out/Release/node bench.js
Running with Script took 56 ms
Running with compileFunction took 55 ms
Running with compileFunction and cached data took 7 ms

If I change producedData: true to produceCachedData: cachedData === undefined (otherwise it's ~600ms because it then keeps serializing code cache even after the cache is produced on the first run, and the serialization comes with a cost).

Note that it still gives up on caching when context extensions are used as that can be much more complex to support. But I think context extensions are probably not used as often (we don't use it internally, at least).

About parsing context - I don't think it matters in terms of caching. The caching is done on context-independent scripts (which is why the cache is per-isolate, not per-context - context here being a V8 execution context, not a context for scope resolution which is actually part of the script). Binding to the context happens much later.

[SimenB]

Ooh, exciting! Using compileFunction simplifies the implementation in Jest quite a bit!

[karlhorky]

I opened #49950 to address the "no import() needed" case which is a fairly simple change and can backport to earlier release lines (v20.x at least

Looks like @joyeecheung's #49950 and #50137 will be backported to Node.js v20 in the release v20.10.0:

• v20.10.0 release proposal #50682