Skip to content
This repository has been archived by the owner on Aug 8, 2023. It is now read-only.

[node] Crash when running stress tests #8812

Closed
tmpsantos opened this issue Apr 25, 2017 · 7 comments
Closed

[node] Crash when running stress tests #8812

tmpsantos opened this issue Apr 25, 2017 · 7 comments
Assignees
@tmpsantos
Labels
crash Node.js node-mapbox-gl-native

Comments

@tmpsantos
Copy link
Contributor

I let the memory.test.js running overnight, trying to render 1200000 times with 10 maps on the pool. It blew up with a SEGV.

(master) tmpsantos@macbook:~/Projects/mapbox-gl-native$ BUILDTYPE=Release make node -j8 && time node --expose-gc platform/node/test/memory.test.js 
platform/linux/ninja  -j4 -C build/linux-x86_64/Release mbgl-node
ninja: Entering directory `build/linux-x86_64/Release'
ninja: no work to do.
TAP version 13
# Memory
# Rendering (0/1200000)
# Rendering (120000/1200000)
# Rendering (240000/1200000)
# Rendering (360000/1200000)
# Rendering (480000/1200000)
# Rendering (600000/1200000)
# Rendering (720000/1200000)
# Rendering (840000/1200000)
# Rendering (960000/1200000)
# Rendering (1080000/1200000)
Segmentation fault (core dumped)

real	458m20.525s
user	1088m48.156s
sys	51m18.880s

The dmesg was showing:

[1858468.176996] traps: node[1284] general protection ip:7fd906370d43 sp:7fff789a8ee8 error:0 in mapbox_gl_native.node[7fd90623b000+3c6000]

With this backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fd906370d43 in std::_Function_handler<void (), node_mbgl::NodeRequest::Execute()::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    () from /home/tmpsantos/Projects/mapbox-gl-native/lib/mapbox_gl_native.node
[Current thread is 1 (Thread 0x7fd90ca17740 (LWP 1284))]
(gdb) bt
#0  0x00007fd906370d43 in std::_Function_handler<void (), node_mbgl::NodeRequest::Execute()::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    () from /home/tmpsantos/Projects/mapbox-gl-native/lib/mapbox_gl_native.node
#1  0x00007fd90c404703 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#2  0x00007fd90c4047e6 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#3  0x00007fd90c4136d8 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#4  0x00007fd90c4050ac in uv_run () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#5  0x000055a499562338 in node::StartNodeInstance (arg=<synthetic pointer>) at ../src/node.cc:4084
#6  node::Start (argc=2, argv=<optimized out>) at ../src/node.cc:4160
#7  0x00007fd90a2b33f1 in __libc_start_main (main=0x55a4990a4270 <main(int, char**)>, argc=3, argv=0x7fff789ac838, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff789ac828) at ../csu/libc-start.c:291
#8  0x000055a4990a45a9 in _start ()

I'm going to run it again printing metrics about the V8 heap growth and see if this could be related.

/cc @bsudekum
@tmpsantos tmpsantos self-assigned this Apr 25, 2017
@tmpsantos tmpsantos added Node.js node-mapbox-gl-native crash labels Apr 25, 2017
@tmpsantos
Copy link
Contributor Author

V8 heap was stable during the test, but it crashed again. This time with node v4.8.2:

$ nvm run v4.8.2 --expose-gc platform/node/test/memory.test.js
platform/linux/ninja  -j4 -C build/linux-x86_64/Release mbgl-node
ninja: Entering directory `build/linux-x86_64/Release'
ninja: no work to do.
Running node v4.8.2 (npm v2.15.11)
TAP version 13
# Memory
# Rendering (0/1200000) - V8 Heap: 4358136
# Rendering (12000/1200000) - V8 Heap: 4532944
# Rendering (24000/1200000) - V8 Heap: 4111976
# Rendering (36000/1200000) - V8 Heap: 4108776
# Rendering (48000/1200000) - V8 Heap: 4106656
# Rendering (60000/1200000) - V8 Heap: 4353896
# Rendering (72000/1200000) - V8 Heap: 4366216
# Rendering (84000/1200000) - V8 Heap: 4359312
# Rendering (96000/1200000) - V8 Heap: 4353456
# Rendering (108000/1200000) - V8 Heap: 4355672
# Rendering (120000/1200000) - V8 Heap: 4372728
# Rendering (132000/1200000) - V8 Heap: 4373664
# Rendering (144000/1200000) - V8 Heap: 4347776
# Rendering (156000/1200000) - V8 Heap: 4351376
# Rendering (168000/1200000) - V8 Heap: 4347384
# Rendering (180000/1200000) - V8 Heap: 4349072
# Rendering (192000/1200000) - V8 Heap: 4345336
# Rendering (204000/1200000) - V8 Heap: 4370832
# Rendering (216000/1200000) - V8 Heap: 4363720
# Rendering (228000/1200000) - V8 Heap: 4362032
# Rendering (240000/1200000) - V8 Heap: 4366448
# Rendering (252000/1200000) - V8 Heap: 4353384
# Rendering (264000/1200000) - V8 Heap: 4371280
# Rendering (276000/1200000) - V8 Heap: 4361456
# Rendering (288000/1200000) - V8 Heap: 4354480
# Rendering (300000/1200000) - V8 Heap: 4362744
# Rendering (312000/1200000) - V8 Heap: 4364832
# Rendering (324000/1200000) - V8 Heap: 4357160
# Rendering (336000/1200000) - V8 Heap: 4354840
# Rendering (348000/1200000) - V8 Heap: 4361776
# Rendering (360000/1200000) - V8 Heap: 4342296
# Rendering (372000/1200000) - V8 Heap: 4344648
# Rendering (384000/1200000) - V8 Heap: 4364080
# Rendering (396000/1200000) - V8 Heap: 4362152
# Rendering (408000/1200000) - V8 Heap: 4341952
# Rendering (420000/1200000) - V8 Heap: 4352040
# Rendering (432000/1200000) - V8 Heap: 4349480
# Rendering (444000/1200000) - V8 Heap: 4346120
# Rendering (456000/1200000) - V8 Heap: 4358688
# Rendering (468000/1200000) - V8 Heap: 4337496
# Rendering (480000/1200000) - V8 Heap: 4361024
# Rendering (492000/1200000) - V8 Heap: 4349712
# Rendering (504000/1200000) - V8 Heap: 4348368
# Rendering (516000/1200000) - V8 Heap: 4367504
# Rendering (528000/1200000) - V8 Heap: 4364640
# Rendering (540000/1200000) - V8 Heap: 4359440
# Rendering (552000/1200000) - V8 Heap: 4344632
# Rendering (564000/1200000) - V8 Heap: 4346616
# Rendering (576000/1200000) - V8 Heap: 4355376
# Rendering (588000/1200000) - V8 Heap: 4352752
# Rendering (600000/1200000) - V8 Heap: 4359456
# Rendering (612000/1200000) - V8 Heap: 4351032
# Rendering (624000/1200000) - V8 Heap: 4358440
# Rendering (636000/1200000) - V8 Heap: 4362480
# Rendering (648000/1200000) - V8 Heap: 4347520
# Rendering (660000/1200000) - V8 Heap: 4339768
# Rendering (672000/1200000) - V8 Heap: 4352568
# Rendering (684000/1200000) - V8 Heap: 4356104
# Rendering (696000/1200000) - V8 Heap: 4359952
# Rendering (708000/1200000) - V8 Heap: 4344544
# Rendering (720000/1200000) - V8 Heap: 4353376
# Rendering (732000/1200000) - V8 Heap: 4341504
Segmentation fault (core dumped)

Different backtrace:

#0  0x0000000000000000 in ?? ()
#1  0x0000000000fa822b in uv__async_event (loop=0x1998e00 <default_loop_struct>, w=<optimized out>, nevents=<optimized out>)
    at ../deps/uv/src/unix/async.c:98
#2  0x0000000000fa8303 in uv__async_io (loop=0x1998e00 <default_loop_struct>, w=0x1998fc8 <default_loop_struct+456>, events=<optimized out>)
    at ../deps/uv/src/unix/async.c:138
#3  0x0000000000fb88d0 in uv__io_poll (loop=0x1998e00 <default_loop_struct>, timeout=22) at ../deps/uv/src/unix/linux-core.c:380
#4  0x0000000000fa8de6 in uv_run (loop=0x1998e00 <default_loop_struct>, mode=UV_RUN_ONCE) at ../deps/uv/src/unix/core.c:354
#5  0x0000000000df3550 in node::Start(int, char**) ()
#6  0x00007f5a0c0403f1 in __libc_start_main (main=0x723240 <main>, argc=3, argv=0x7fff9185c878, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fff9185c868) at ../csu/libc-start.c:291
#7  0x00000000007234ad in _start ()

@mikemorris
Copy link
Contributor

@tmpsantos I'd take a close look at

mapbox-gl-native/platform/node/src/node_request.cpp

Line 125 in 5700c99

asyncExecute = std::make_unique<mbgl::util::AsyncTask>([this] { doExecute(); Unref(); });
, possible that something may get prematurely garbage collected (which could explain the slight inconsistency) around there?

@tmpsantos
Copy link
Contributor Author

Better backtrace now building Debug, also got the crash quicker:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
btCore was generated by `node platform/node/test/memory.test.js'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb11c7af328 in main_arena () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fb11d869740 (LWP 17483))]
(gdb) bt
#0  0x00007fb11c7af328 in main_arena () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fb1198fc2f6 in node_mbgl::NodeRequest::<lambda()>::operator()(void) const (__closure=0xc830ef8)
    at ../../../platform/node/src/node_request.cpp:125
#2  0x00007fb1198fc95c in std::_Function_handler<void(), node_mbgl::NodeRequest::Execute()::<lambda()> >::_M_invoke(const std::_Any_data &) (
    __functor=...) at /usr/include/c++/6/functional:1740
#3  0x00007fb1199274ea in std::function<void ()>::operator()() const (this=0xc830ef8) at /usr/include/c++/6/functional:2136
#4  0x00007fb119d2a366 in mbgl::util::AsyncTask::Impl::asyncCallback (handle=0xbcfa970) at ../../../platform/default/async_task.cpp:43
#5  0x0000000000fa822b in uv__async_event (loop=0x1998e00 <default_loop_struct>, w=<optimized out>, nevents=<optimized out>)
    at ../deps/uv/src/unix/async.c:98
#6  0x0000000000fa8303 in uv__async_io (loop=0x1998e00 <default_loop_struct>, w=0x1998fc8 <default_loop_struct+456>, events=<optimized out>)
    at ../deps/uv/src/unix/async.c:138
#7  0x0000000000fb88d0 in uv__io_poll (loop=0x1998e00 <default_loop_struct>, timeout=3) at ../deps/uv/src/unix/linux-core.c:380
#8  0x0000000000fa8de6 in uv_run (loop=0x1998e00 <default_loop_struct>, mode=UV_RUN_ONCE) at ../deps/uv/src/unix/core.c:354
#9  0x0000000000df3550 in node::Start(int, char**) ()
#10 0x00007fb11c40d3f1 in __libc_start_main (main=0x723240 <main>, argc=2, argv=0x7ffff78bf958, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffff78bf948) at ../csu/libc-start.c:291
#11 0x00000000007234ad in _start ()
(gdb)

@kkaefer
Copy link
Contributor

kkaefer commented Apr 27, 2017

I believe this crash happens when calling target->handle(). target is a pointer to NodeMap that is initialized when the NodeRequest object is created. However, we are not holding a strong reference to NodeMap anywhere, which means that the NodeMap object can get destructed before NodeRequest's lambda is invoked.

@kkaefer
Copy link
Contributor

kkaefer commented Apr 27, 2017

We're wrapping a pointer to NodeMap in a v8::External, but we could just the actual handle instead, and store it in a Persistent handle in the NodeRequest.

@tmpsantos
Copy link
Contributor Author

Closing in favor of #8966

@jfirebaugh
Copy link
Contributor

This isn't the same as #8966.

@jfirebaugh jfirebaugh reopened this Jul 19, 2017
@jfirebaugh jfirebaugh mentioned this issue Jul 19, 2017
Merged
@jfirebaugh jfirebaugh mentioned this issue Sep 11, 2017
Closed
@springmeyer springmeyer mentioned this issue Jun 29, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tmpsantos tmpsantos

Labels
crash Node.js node-mapbox-gl-native
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kkaefer @jfirebaugh @tmpsantos @mikemorris