Node domains causing a memory leak

[garethbowen]

Domains in Node are deprecated because they're bad. In particular, we call Domain.dispose which has been removed in Node v10 so uncaught exceptions cause even more exceptions to be thrown.

Domains can be used to limit errors to one specific request, but currently we're exiting the process which clearly isn't limiting it at all. Replace the use of domains with an unhandled exception handler.

[ngaruko]

Any clue how to AT this @garethbowen ?

[garethbowen]

No AT needed. Sorry, I should have just closed it!

[garethbowen]

@dianabarsan found this memory leak in node v10+ which is solved by this fix.

[garethbowen]

Due to the above memory leak Diana and I decide to backport this for 3.3.0 release.

[dianabarsan]

Unfortunately, domain also leaks on Node 8.11.4 (which is the version that we use in production), albeit the leak appears to be not as severe as for Node 10.

Offline user calls to _changes, _bulk_docs, _all_docs & _bulk_get (~5k requests):

admin calls to _session (~50k requests):

Above tests were done on 3.2.1 api code running on Node 8.11.4 in production mode.

[dianabarsan]

Same as above but after removing domain from api router:

Offline user calls to _changes, _bulk_docs, _all_docs & _bulk_get (~5k requests) 35MB vs 612MB:

admin calls to _session (~50k requests) 28MB vs 899MB:

[garethbowen]

Thanks for the additional info @dianabarsan .

As discussed I'm proposing to leave this in 3.3.0 for now and monitor running instances and restart as needed. If it becomes too problematic we can backport to a new 3.2.2 release.

[derickl]

Probably worth back porting given MUSO's now frequent incidences of 502s.

[garethbowen]

@derickl Have those crashes been investigated? How confident are we that it's this issue causing them?