Virtual memory leak on Linux?

[marcdumais-work]

Hi,

OS: Ubuntu 16.04
node: v8.1.1

We use NSFW for our project . We have noticed an apparent virtual memory leak, at least on Linux. Digging some more, it seems that it's related to receiving NSFW events. Note that the resident memory does not seem to increase much.

I was able to reproduce the issue, using a slightly modified version of your readme example. I have changed the watched directory to /tmp/ and removed watcher1.stop(), so that the program doesn't exit after receiving the first event.

var nsfw = require('nsfw');

var watcher1;

return nsfw(
  '/tmp',
  function (events) {
    // handle events
  })
  .then(function (watcher) {
    // only "start()" once
    if (watcher1 !== watcher) {
      watcher1 = watcher;
      return watcher.start();
    }
  });

Then I added an "event generator" script, that appends to a file 100 times a second, under the watched directory. (lower append speed also triggers leak, but not as fast):

#!/bin/bash
counter=1
while [ $counter -le 2 ]
do
	echo "a" >> /tmp/file.txt
        sleep 0.01
done

When we start the watcher program, node reports that ~1 GB of virtual memory is used. When we then start generating events using the script above, the reported virtual memory increases at a rate of about 1GB a minute (up-to 256GB if we leave it run long enough)

---

Am I doing something wrong in the code above or does this look like a bug?

[MarkZ3]

If I do the same test on macOS, it seems to leak about 1MB a minute, which is much less noticeable.

[jwulf]

Do you have circular symlinks? My machine does this with nsfw when using it over a repo with circular symlinks on Linux.

[marcdumais-work]

@jwulf good thought, but that does not seem to be the case.

[ventr1x]

Is there any info on this?
We had a node server kill our host two times in two months because of random memory leaks coming from this plugin.
We currently don't even actively use it, but it's still included with default settings and watching one folder.

This is quite the issue, rendering this plugin not only entirely useless for production use but also a very high risk hazard.
I haven't taken much time reproducing it. But after restarting the server I see a constant raise in memory allocations (about 1mb every few minutes), which results in crashing the process and creating a big ass core dump file.
As we're using pm2 to restart the server, it does not only take all the memory, it also takes all the hdd space for the core dumps.

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [node /server/server.js]
 2: 0x12b82ac [node /server/server.js]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node /server/server.js]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node /server/server.js]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node /server/server.js]
 6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [node /server/server.js]
 7: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [node /server/server.js]
 8: v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector<unsigned char const>, v8::internal::PretenureFlag) [node /server/server.js]
 9: v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [node /server/server.js]
10: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [node /server/server.js]
11: NSFW::fireEventCallback(uv_async_s*) [/server/node_modules/nsfw/build/Release/nsfw.node]

[implausible]

Thanks for bringing this to my attention again. I will take a look at this to see if we can fix the leak.

[implausible]

I believe I've sourced the memory leak in linux / os x, and it's because uv_thread_create does not create detached threads. However, before I patch the memory leaks I've found a different leak on windows that I'd like to track down. Bad test 😄

[implausible]

It seems like it's very specific to pthread not being detached. I will patch this asap.

[marcdumais-work]

@implausible many thanks for the fix!