node 0.11.13 - core dump - Tests (Linux)

[migounette]

I am facing a core dump with node 0.11.13.

After a quick isolation I have this issue:
./BUILD/node-v0.11.13/out/Release/node --expose_gc /root/rpmbuild/BUILD/node-v0.11.13/test/simple/test-v8-gc.js

I can't reproduce the issue

On V0.11.11 because the test does not exist.
On Windows:
c:_GIT\node-v0.11.13\Release>node --expose_gc c:_GIT\node-v0.11.13\test\simple\test-v8-gc.js
c:_GIT\node-v0.11.13\Release>echo %ERRORLEVEL%
0

Since the migration of V8 I also have some NEW random memory corruptions with a native module (Nan v1.0) and node 0.11.13.

Not seen prior 0.11.11 but this is need to be confirmed that I didn't make a mistake with my native module. But the node tests seem to face the same issue !!!!

More details to come...

System: Rhel 6.5 x64

# gdb ./BUILD/node-v0.11.13/out/Release/node
./BUILD/node-v0.11.13/core.31104
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/root/rpmbuild/BUILD/node-v0.11.13/out/Release/node...done.
[New Thread 31104]
[New Thread 31108]
[New Thread 31105]
[New Thread 31109]
[New Thread 31107]
[New Thread 31106]
Missing separate debuginfo for
Try: yum --disablerepo='*' --enablerepo='*-debug*' install
/usr/lib/debug/.build-id/84/61a4a9bc26a3896c6a601643ae9bf1366a8607
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2 Core was generated by `out/Release/node --expose_gc /root/rpmbuild/BUILD/node-v0.11.13/test/simple/tes'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000009fd1a7 in
node::Environment::IsolateData::AfterGarbageCollection(v8::GCType,
v8::GCCallbackFlags) ()
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.132.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 
libstdc++-4.4.7-4.el6.x86_64
(gdb) bt
#0  0x00000000009fd1a7 in
node::Environment::IsolateData::AfterGarbageCollection(v8::GCType,
v8::GCCallbackFlags) ()
#1  0x00000000006f8b8d in
v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
v8::internal::GCTracer*, v8::GCCallbackFlags) ()
#2  0x00000000006f9037 in
v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) ()
#3  0x00000000006f948c in v8::internal::Heap::CollectAllGarbage(int,
char const*, v8::GCCallbackFlags) ()
#4  0x00000000005f58ed in
v8::Isolate::RequestGarbageCollectionForTesting(v8::Isolate::GarbageCollectionType)
()
#5  0x00000000006186ec in
v8::internal::FunctionCallbackArguments::Call(void
(*)(v8::FunctionCallbackInfo<v8::Value> const&)) ()
#6  0x000000000063618c in v8::internal::Builtin_HandleApiCall(int,
v8::internal::Object**, v8::internal::Isolate*) ()
#7  0x000012819d90634e in ?? ()
#8  0x000012819d9062a1 in ?? ()
#9  0x00007fff1d0cb2b0 in ?? ()
#10 0x00007fff1d0cb318 in ?? ()
#11 0x000012819d99d3ba in ?? ()
#12 0x000017741d8553a1 in ?? ()
#13 0x000017741d87cb21 in ?? ()
#14 0x000017741d8553a1 in ?? ()
#15 0x0000362c6b651c21 in ?? ()
#16 0x0000362c6b6428e9 in ?? ()
#17 0x0000362c6b640491 in ?? ()
#18 0x0000362c6b651ea9 in ?? ()
#19 0x0000362c6b651b01 in ?? ()
#20 0x0000362c6b651bd9 in ?? ()
#21 0x00007fff1d0cb380 in ?? ()
#22 0x000012819d9468a9 in ?? ()

[tjfontaine]

I've not been able to reproduce this yet, can you provide more details about your environment including the compiler toolchain versions?

Also do you think you could provide the core file and the node binary somewhere so I could inspect it?

[migounette]

It's a Rhel 6.5 system x64

I can provide you a core file, I will take a snapshot of the toolchain (versions)
python, gcc, .... and the command line to build.

In fact this crash occured on two differents systems. So, I can reproduce it.

I will upload it to a private ftp and I will send you credentials ASAP

Thanks

[indutny]

@migounette may I ask you to try reproducing it with debug build too? BUILDTYPE=Debug make -C out/ -j16 and the executable is ./out/Debug/node

[migounette]

I will provide you the core before end of monday... I am working on memory corruption which took me most of my time. All stuff on ftp for next monday (GMT+1 end of day)

[migounette]

Humm... I can't reproduce it in debug mode !!!!

Maybe linked to a timing issue ?

[root@ducati6 node-v0.11.13]# ./out/Release/node --expose_gc $PWD/test/simple/test-v8-gc.js
Segmentation fault (core dumped)
[root@ducati6 node-v0.11.13]# ./out/Debug/node --expose_gc $PWD/test/simple/test-v8-gc.js

I am currently uploading the core in release mode

[migounette]

Check your mail for the password:

ftp://nodejs@ftp.usa.hp.com/

[indutny]

I looked at the core, but it is a bit hard to figure out anything out of it. I'll continue the investigation, but if you have some spare time - it'd be cool if you'd try following thing:

• Change -O0 to -O3 https://github.com/joyent/node/blob/master/common.gypi#L42
• Rebuild node in Debug mode
• Try it again and upload a core file if it'll fail

If it still won't fail - could you please try adding -g at this line https://github.com/joyent/node/blob/master/common.gypi#L68 ?

[migounette]

Okay, I will do my best for tomorrow. I will update you.

[migounette]

Well... no chance with the core.

I tried various operation:

Compiling with 01 02 03 or only 0 in release and debug mode
The only way to get it it's Release and 03, with g flag enabled ! No chance

I am still trying every day a different falvor once I have a valid core with debug info, I will let you know.

[kkoopa]

Most likely a missing HandleScope or you are returning handles without pushing them to the surrounding scope. V8 in 0.11.13 is more aggressive with garbage collection and such, also the distinction between escapable and unescapable scopes was introduced around this vetsion.

[migounette]

Thanks for the hint, but this is an official node test around the GC. The GC is invoked by the test, --expose-gc and it seems to be a kind of race condition, not directly linked to an HandleScope problem.

I will try to look at it, when back to real life (summer time...) I have just finished a huge dev with node v0.11.13 and webRTC, I was exposed to all changes (Escpable and unescpable scope), but at this stage this issue seems to be a bit more complex and related to node and V8.

[trevnorris]

Right now I actually wouldn't worry too much about this. It's questionable whether the tracing module will make it into v0.12 (there are a lot of implementation details that still need to be worked out).

[migounette]

@trevnorris Not scared at all, node 0.11.13 works fine in our labs, we passed long endurance tests and stressfull tests.

No GC issues seen at this stage,and node is very stable for long run operations.

So far so good, Very soon,