JavaScript heap out of memory an OpenAPI 2.0 document

[DavidBiesack]

Describe the bug
I ran:

spectral lint openapi.json

which resulted in a stack trace. The input OpenAPI 2.0 file is attached as
openapi.json.txt
(The OpenAPI contains some unused elements because it was extracted from a more complete API definition.)

Stack trace attached as

spectral-traceback.txt

Expected behavior

complete normally.

Environment (remove any that are not applicable):

• Library version: spectral 5.4.0
• OS: mac osx 10.15.2
• Browser: n/a (CLI)

Additional context

May be related to the size of the OpenAPI document. If I trim down some more of the content, eventually it works without running out of heap. But the source OpenAPI has many operations I left off of this, and other data.

[P0lip]

Spectral 5.5.x will come with many perf improvements as well as brand new JSON Path expression engine (hidden behind a flag for the time being).
I've tried it out against your spec and it's able to lint it without any significant problems.
There does seem to be something fishy going on with $ref resolving that results in a number of schema validation error, but we are planning to change the $ref resolver we use, therefore that error should go away at some point.

We have 5.5.0@beta published already, in case you are interested in trying it out live.
The stable release should be out soon.

[DavidBiesack]

Spectral 5.5.x will come with many perf improvements as well as brand new JSON Path expression engine (hidden behind a flag for the time being).
I've tried it out against your spec and it's able to lint it without any significant problems.

Thanks @P0lip - I'd be happy to try out 5.5.x early versions or pre-release candidates and can throw them at the full OpenAPI documents we have (several are 3000-4000 lines) for more feedback on performance. I've also noticed some strange errors on properties defined with $ref and error messages that confuse me. Maybe the new lib will resolve those. Let me know how I can help.

[nulltoken]

I'd be happy to try out 5.5.x early versions or pre-release candidates and can throw them at the full OpenAPI documents we have (several are 3000-4000 lines) for more feedback

@DavidBiesack That would be awesome. You can already leverage the latest published beta to give it a try.

[nulltoken]

I'd be happy to try out 5.5.x early versions or pre-release candidates and can throw them at the full OpenAPI documents we have (several are 3000-4000 lines) for more feedback

@DavidBiesack That would be awesome. You can already leverage the latest published beta to give it a try.

...passing useNimma: true to the Spectral constructor to active the new rule optimizer

[DavidBiesack]

@nulltoken Unfortunately, using 5.5.0-beta3 (with useNimma: true option for JavaScript or or export USE_NIMMA=true for CLI) does not resolve the heap error on our complete OpenAPI document.
You can try out the full document at https://developer.apiture.com/docs/apis/accountApplications/v0.17.9/openapi.json :

 spectral lint https://developer.apiture.com/docs/apis/accountApplications/v0.17.9/openapi.json
OpenAPI 2.0 (Swagger) detected

<--- Last few GCs --->

[59247:0x102888000]    88378 ms: Mark-sweep 2028.6 (2059.2) -> 2022.8 (2060.5) MB, 4585.8 / 0.0 ms  (average mu = 0.154, current mu = 0.005) allocation failure scavenge might not succeed
[59247:0x102888000]    93579 ms: Mark-sweep 2029.8 (2060.5) -> 2024.0 (2061.7) MB, 5121.1 / 0.0 ms  (average mu = 0.091, current mu = 0.015) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x100930c99]
Security context: 0x2268cbdc08a1 <JSObject>
    1: /* anonymous */ [0x22685553f0f9] [/Users/david.biesack/dev/tools/spectral/dist/linter.js:~31] [pc=0x1a67aee0ad69](this=0x2268c8b2da81 <Object map = 0x2268584e2801>,0x2268ca080139 <Object map = 0x2268bc8943e1>,0x2268c3dfa871 <Object map = 0x2268bc8a16d1>,0x22689a1c2ae9 <Object map = 0x2268c9f2a5a1>,0x2268efd804a9 <undefined>)
    2: callback(aka callback)...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200615.091056.59247.0.001.json
Node.js report completed
 1: 0x10007e9b3 node::Abort() [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 2: 0x10007eb37 node::OnFatalError(char const*, char const*) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 3: 0x100176337 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 4: 0x1001762d3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 5: 0x1002fa485 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 6: 0x1002fbb54 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 7: 0x1002f8a27 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 8: 0x1002f6a0d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
 9: 0x100302124 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
10: 0x10030219f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
11: 0x1002ced97 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
12: 0x1005f83e5 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
13: 0x100930c99 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/david.biesack/.nvm/versions/node/v12.13.1/bin/node]
Abort trap: 6

node report file: report.20200615.091056.59247.0.001.json.txt

[P0lip]

@DavidBiesack
Do you have any custom ruleset? If so, could you please provide it assuming it doesn't contain any sensitive information?

[DavidBiesack]

@P0lip I use this ruleset:

extends: spectral:oas
rules:
  oas2-api-host: off
  oas2-unused-definition: warn
  openapi-tags-alphabetical: off
  oas2-schema: warn

[P0lip]

Thanks a lot for all the details, really appreciate it.
I was able to narrow down the issue.
It's json-to-ast in better-ajv-errors.
With better-ajv-errors disabled (I had to do it manually in the code and built the binary afterwards), I was able to lint the spec, with nimma enabled

Results

USE_NIMMA=true /home/p0lip/Projects/Stoplight/spectral/binaries/spectral lint https://developer.apiture.com/docs/apis/accountApplications/v0.17.9/openapi.json
OpenAPI 2.0 (Swagger) detected

<REDACTED>

✖ xx problems (xx errors, x warning, x infos, x hints)

I'm going to fork better-ajv-errors and get rid of json-to-ast entirely, as we don't need it.

[DavidBiesack]

Thanks @P0lip - good to know there is a clean fix. (Can you perhaps delete the gory dirty laundry from the api document :-) ?)

[P0lip]

Not yet, but hopefully one day #98 😉

[DavidBiesack]

@P0lip I just meant delete all the error messages you pasted into the above comment :-)

[P0lip]

redacted. I'll let you know once a new release with the relevant fix is out.

[P0lip]

Just wanted to let you know that a new beta release is out.
LMK in case the issue still persists.

[DavidBiesack]

Thanks @P0lip - I can confirm that Spectral runs without the heap error on our source OpenAPI.

Performance it still sluggish - it takes 54s to scan the source file. Using the useNimma: true option in the constructor reduced time to 10 or 12s on our large OpenAPI, <1s one many.