Core dump is not created on OOM exceptions in Docker even though COMPlus_DbgEnableMiniDump is set to 1

[itai]

Steps to reproduce

Run the mcr.microsoft.com/dotnet/sdk:5.0.202-alpine3.13-amd64 docker image with a memory limit and privileged capabilites:

docker run --memory 10g --memory-swap 10g --privileged --cap-add=ALL --security-opt seccomp:unconfined -it mcr.microsoft.com/dotnet/sdk:5.0.202-alpine3.13-amd64

Inside the container, create a new app:

dotnet new console -o App -n App

Change App/Program.cs to the following:

using System;
using System.Collections.Generic;

namespace App
{
    class Program
    {
        static void Main(string[] args)
        {
            var l = new List<byte[]>();
            while (true)
            {
                l.Add(new byte[1024 * 1024 * 1024]);
            }
        }
    }
}

Publish the application:

dotnet publish -c Release App/App.csproj

Run the following:

export COMPlus_DbgEnableMiniDump=1
export COMPlus_DbgMiniDumpType=1
export COMPlus_DbgMiniDumpName=/core.dmp
export COMPlus_CreateDumpDiagnostics=1

Run the executable:

dotnet ./App/bin/Release/net5.0/App.dll

As expected, we get an "out of memory" exception. However, a core dump is not created. It appears that createdump starts running but doesn't finish. The following file contains the on-screen output: app_output.txt

Configuration

> docker --version
Docker version 20.10.5, build 55c4c88

Inside the container:

# dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.202
 Commit:    db7cc87d51

Runtime Environment:
 OS Name:     alpine
 OS Version:  3.13
 OS Platform: Linux
 RID:         linux-musl-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.202/

Host (useful for support):
  Version: 5.0.5
  Commit:  2f740adc14

.NET SDKs installed:
  5.0.202 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Steps to reproduce

Run the mcr.microsoft.com/dotnet/sdk:5.0.202-alpine3.13-amd64 docker image with a memory limit and privileged capabilites:

docker run --memory 10g --memory-swap 10g --privileged --cap-add=ALL --security-opt seccomp:unconfined -it mcr.microsoft.com/dotnet/sdk:5.0.202-alpine3.13-amd64

Inside the container, create a new app:

dotnet new console -o App -n App

Change App/Program.cs to the following:

using System;
using System.Collections.Generic;

namespace App
{
    class Program
    {
        static void Main(string[] args)
        {
            var l = new List<byte[]>();
            while (true)
            {
                l.Add(new byte[1024 * 1024 * 1024]);
            }
        }
    }
}

Publish the application:

dotnet publish -c Release App/App.csproj

Run the following:

export COMPlus_DbgEnableMiniDump=1
export COMPlus_DbgMiniDumpType=1
export COMPlus_DbgMiniDumpName=/core.dmp
export COMPlus_CreateDumpDiagnostics=1

Run the executable:

dotnet ./App/bin/Release/net5.0/App.dll

As expected, we get an "out of memory" exception. However, a core dump is not created. It appears that createdump starts running but doesn't finish. The following file contains the on-screen output: app_output.txt

Configuration

> docker --version
Docker version 20.10.5, build 55c4c88

Inside the container:

# dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.202
 Commit:    db7cc87d51

Runtime Environment:
 OS Name:     alpine
 OS Version:  3.13
 OS Platform: Linux
 RID:         linux-musl-x64
 Base Path:   /usr/share/dotnet/sdk/5.0.202/

Host (useful for support):
  Version: 5.0.5
  Commit:  2f740adc14

.NET SDKs installed:
  5.0.202 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 5.0.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Author:	itai
Assignees:	-
Labels:	area-Diagnostics-coreclr, untriaged
Milestone:	-

[davidfowl]

cc @mikem8361

[mikem8361]

From the app_output.txt it looks like createdump is getting launched but it gets killed in the middle of the dump generation because of Linux's OOM-killer. Could you try to disable it and run this again?

[mikem8361]

To disable the OOM-killer use the same command with the value 0:
# cat /proc/sys/vm/panic_on_oom. When you set the value to 0 that means the kernel will not panic when out of memory error occurred.
$ echo 0 > /proc/sys/vm/panic_on_oom. ...
echo 1 > /proc/sys/vm/panic_on_oom.

[ghost]

This issue has been automatically marked no recent activity because it has been marked as needs author feedback but has not had any activity for 14 days. It will be closed if no further activity occurs within 7 more days. Any new comment (by anyone, not necessarily the author) will remove no recent activity

[itai]

The default value of /proc/sys/vm/panic_on_oom appears to be 0. I tried setting it to both 0 and 1 and running dotnet ./App/bin/Release/net5.0/App.dll and I still did not get a core dump.

run with panic_on_oom set to 0.txt
run with panic_on_oom set to 1.txt

[mikem8361]

Yes, it does look like in both cases, createdump is aborted. It seems that createdump is still being aborted by the OOM killer. It also could be aborting itself when a memory allocation fails but that would be a crash/exception/signal. Not sure what to do about it in either case.

[mikem8361]

/cc: @hoyosjs

[mikem8361]

I don't know the details but I was told that it is hard to turn off the OOM killer. The above directions I found on the internet. I don't have any experience with it.

[mikem8361]

Moving this to 7.0.0 because there is nothing we can do in createdump to prevent it from being killed in low memory scenarios