Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker will swallow OOM exceptions #2757

Closed
paulbatum opened this issue Oct 7, 2024 · 0 comments · Fixed by #2789
Closed

Worker will swallow OOM exceptions #2757

paulbatum opened this issue Oct 7, 2024 · 0 comments · Fixed by #2789
Assignees
Labels
bug Something isn't working

Comments

@paulbatum
Copy link
Member

paulbatum commented Oct 7, 2024

Description

Invocationhandler.InvokeAsync will catch any exception and never rethrows. This creates confusing hangs for terminal exceptions such as OutOfMemoryException where the desirable behavior is to crash the process.

catch (Exception ex)
{
response.Result.Exception = _workerOptions.EnableUserCodeException ? ex.ToUserRpcException() : ex.ToRpcException();
response.Result.Status = StatusResult.Types.Status.Failure;
if (ex.InnerException is TaskCanceledException or OperationCanceledException)
{
response.Result.Status = StatusResult.Types.Status.Cancelled;
}
}

Steps to reproduce

The simplest repro is to explicitly throw out of memory with a function. The worker process will not crash.

        [Function(nameof(ThrowOutOfMemoryDirectly))]
        public IActionResult ThrowOutOfMemoryDirectly(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequest req)
        {
            throw new OutOfMemoryException();
        }

You can also reproduce with a more realistic setup of running the worker in a constrained memory environment and allocating above that limit, but this will introduce other variables, such as whether the kernel chooses to kill the process due to being out of memory.

@paulbatum paulbatum added the potential-bug Items opened using the bug report template, not yet triaged and confirmed as a bug label Oct 7, 2024
@satvu satvu added bug Something isn't working and removed Needs: Triage (Functions) potential-bug Items opened using the bug report template, not yet triaged and confirmed as a bug labels Oct 10, 2024
@fabiocav fabiocav self-assigned this Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants