seg fault / Invalid cid: 0 corrupt heap #44219
Comments
keertip
added
the
area-vm
|
//cc @rmacnak-google |
|
We would need some reproduction instructions for this, is is possible to provide some test code? |
a-siva
added
the
crash
|
Thanks for the quick response @a-siva ( @rmacnak-google ) - and nice to be in touch again We've spent a couple of days trying to build a small reproduction example rather than a full production project, unfortunately without much success. So the reproduction currently includes a large datafile and bundle of confidential source code. I think probably the easiest thing to do might be for us to give you SSH access to a development machine that's setup with the reproduction case. Would that work? Alternatively if there's any extra information we could capture, happy to do so. We've seen the crash on both our production and dev infra, both on Ubuntu 20.04. |
|
Hi @danrubel and @lukechurch, Lately there have been several fixes related to making the VM more reliable when it runs out of memory. AFAICT, these are newer than the 2.10.4 version reported in your stack traces, so you might first try to run your program on the most recent dev build and see if the crashes reproduce there. The default value of --old-gen-heap-size is set at ~32GB to reflect a constraint coming from the default value of Linux's vm.max_map_count and the VM's heap page alignment (64k mappings * 512kB heap pages = 32GB). Since you are increasing --old-gen-heap-size, you might try increasing vm.max_map_count as well. If neither of those fix the issue, I will probably need access to a reproduction that I can interact with while making debugging changes to the VM: unfortunately for GC crashes like this stack traces and log output are usually not sufficient to track down the bug. |
|
Thank you @rmacnak-google for the suggestions and background information. I'll try the dev VM and various settings with our large data set later this week. In addition I'm trying to build a test case that exercises the VM in a way that causes this error condition to occur. I'll keep you posted as to my progress. |
…nough to support fully allocating a max heap size given by --old_gen_heap_size. TEST=manually adjusting --old_gen_heap_size Bug: #44219 Change-Id: I7ef6d46aca85024028dbd64bddd84438d028f0cf Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/172361 Reviewed-by: Alexander Aprelev <aam@google.com> Reviewed-by: Martin Kustermann <kustermann@google.com> Commit-Queue: Ryan Macnak <rmacnak@google.com>
|
@rmacnak-google From our testing, it appears that Results from a large data run are still pending but wanted to get back to you with what I've learned before the weekend. |
|
@rmacnak-google We are no longer seeing the crashes above given your suggestion to set |
Dart SDK version: 2.10.4 (stable) (Wed Nov 11 13:35:58 2020 +0100) on "linux_x64"running on UbuntuI am running a large program that uses more than 32 GiB of memory, reads from one file, keeps lots of maps lists and simple data types in memory manifesting in a large memory footprint, and writes to another file.
dart --old-gen-heap-size=61440 run bin/driver.dartfollowed by program argumentsWith smaller data sets and smaller memory footprint all works well.
If I run with a larger data set but omit the
--old-gen-heap-sizeargument,then I get the expected
Exhausted heap space, trying to allocate ...error messageBut with a larger data set and the
--old-gen-heap-sizeargument,the program terminates with one of the following... see (1) and (2) below.
The text was updated successfully, but these errors were encountered: