Function running out of memory

[CarlosIS2]

Greetings,
I am running an aws lambda (dotnet6 runtime) that gathers information from several databases, serializes it, and exports it to an S3 bucket.

Despite changing memory configuration (up to 10240 MB), cloudwatch logs still report Out of Memory. The code in question contains a method that runs a loop of sql connections in using blocks, and I have realized that this error shows up after 1500 iterations approximately. Even if I skip code ran before this method, the error still happens at the same iteration.

Here is the relevant code:

public async Task FunctionHandler(ILambdaContext context)
{
    context.Logger.LogInformation($"Date: {DateTime.UtcNow:dd/MM/yyyy HH:mm}");
    try
    {            
        List<Empresa> empresas = _servicioEmpresa.ObtenerInformacionBasica();
        _servicioEmpresa.ObtenerInformacionEmpresas(empresas, context);                        

        await GuardarJSonEnS3(JsonSerializer.Serialize(empresas), "empresas", context);            
    catch (Exception ex)
    {
        context.Logger.LogError($"FunctionHandler - {ex}");
    }       
}

public List<Empresa> ObtenerInformacionBasica()
    {
        List<Empresa> empresas = new();
        using (SqlConnection connection = new SqlConnection(_cadenaConexionCentral))
        {
            using (SqlCommand command = new SqlCommand("aws_Analytics_Obtener_Empresas", connection))
            {
                command.CommandType = CommandType.StoredProcedure;
                connection.Open();
                using (SqlDataReader reader = command.ExecuteReader())
                {
                    while (reader.Read())
                    {
                        //fill object
                    }
                }
            }
        }
        return empresas;
    }

public void ObtenerInformacionEmpresas(List<Empresa> empresas, ILambdaContext context)
    {
        foreach (Empresa empresa in empresas)
        {
            context.Logger.LogInformation($"FunctionHandler::ObtenerInformacionEmpresas::Empresa - {empresa.Nombre}");
            try
            {
                using (SqlConnection connection = new SqlConnection(empresa.CadenaConexion))
                {
                    using (SqlCommand command = new SqlCommand("aws_Analytics_Obtener_Informacion_Empresa", connection))
                    {
                        command.CommandType = CommandType.StoredProcedure;
                        connection.Open();
                        using (SqlDataReader reader = command.ExecuteReader())
                        {
                            while (reader.Read())
                            {
                                //fill object
                            }
                        }
                    }
                }                   
            }               
            catch (Exception ex)
            {                   
                context.Logger.LogError($"FunctionHandler::ObtenerInformacionEmpresas - {ex}");
                throw;
            }
        }                        
    }

By knowing the list that comes as parameter, with that logger at line 5 I can see that the error happens at iteration number 1500 aprox (out of 1700 items).

Here are some logs from cloudwatch that illustrate the issue:

As the error suggest, it seems like a memory issue but the max memory usage does not surpass the lambda memory size. As stated before, I have tried max memory configuration and timeout (15 minutes).

Moreover, the lambda runs like a charm on local machine (Windows 10) through mock lambda test tool.

Any help would be much appreciated.

[normj]

@CarlosIS2 Can you try setting the max heap size environment variable for the function described in this blog post? I doubt in your case it will make difference but it would be good to rule that out. https://aws.amazon.com/blogs/developer/configuring-net-garbage-collection-for-amazon-ecs-and-aws-lambda/

The reason I doubt it will make a difference is Lambda report the memory usage is still much less then the max memory. That implies in that approx 1500th iteration something in the fill object or the reader.Read() is now suddenly attempting to allocate a very large chunk of memory that doesn't fit in the available memory space. In the test tool running on your dev machine I'm sure you have a lot more then 1024 of memory available allowing it to get past the large chunk of memory being asked to allocate.

[CarlosIS2]

Thanks for the quick response @normj
The problem with the "fill object" code block inside the reader is that even if i leave it empty, comment the code or change the iterations order it still runs out of memory at the same point. In my local environment the whole function (which has a few more methods after the one i posted) ends up taking not more than 350MB of memory.

[CarlosIS2]

Changing the max heap environment variable didnt solve the issue. Im starting to suspect that the lambda has some sort of sql connections limit, since the memory usage does not seem to be that high. Its important to note that all the sql connections aim to the same server, just different databases.

[ashishdhingra]

@CarlosIS2 Good morning. Just wanted to check if you were able to resolve issue. If yes, could you please post the root cause and fix?

[CarlosIS2]

@CarlosIS2 Good morning. Just wanted to check if you were able to resolve issue. If yes, could you please post the root cause and fix?
Greetings. Sorry for the late response, I gave up at that time and recently I retried to solve the issue with the new aws lambda update to dotnet 8. Unfortunately, my issue has not been resolved yet.

[CarlosIS2]

Greetings, I am running an aws lambda (dotnet6 runtime) that gathers information from several databases, serializes it, and exports it to an S3 bucket.

Despite changing memory configuration (up to 10240 MB), cloudwatch logs still report Out of Memory. The code in question contains a method that runs a loop of sql connections in using blocks, and I have realized that this error shows up after 1500 iterations approximately. Even if I skip code ran before this method, the error still happens at the same iteration.

Here is the relevant code:

public async Task FunctionHandler(ILambdaContext context)
{
    context.Logger.LogInformation($"Date: {DateTime.UtcNow:dd/MM/yyyy HH:mm}");
    try
    {            
        List<Empresa> empresas = _servicioEmpresa.ObtenerInformacionBasica();
        _servicioEmpresa.ObtenerInformacionEmpresas(empresas, context);                        

        await GuardarJSonEnS3(JsonSerializer.Serialize(empresas), "empresas", context);            
    catch (Exception ex)
    {
        context.Logger.LogError($"FunctionHandler - {ex}");
    }       
}

public List<Empresa> ObtenerInformacionBasica()
    {
        List<Empresa> empresas = new();
        using (SqlConnection connection = new SqlConnection(_cadenaConexionCentral))
        {
            using (SqlCommand command = new SqlCommand("aws_Analytics_Obtener_Empresas", connection))
            {
                command.CommandType = CommandType.StoredProcedure;
                connection.Open();
                using (SqlDataReader reader = command.ExecuteReader())
                {
                    while (reader.Read())
                    {
                        //fill object
                    }
                }
            }
        }
        return empresas;
    }

public void ObtenerInformacionEmpresas(List<Empresa> empresas, ILambdaContext context)
    {
        foreach (Empresa empresa in empresas)
        {
            context.Logger.LogInformation($"FunctionHandler::ObtenerInformacionEmpresas::Empresa - {empresa.Nombre}");
            try
            {
                using (SqlConnection connection = new SqlConnection(empresa.CadenaConexion))
                {
                    using (SqlCommand command = new SqlCommand("aws_Analytics_Obtener_Informacion_Empresa", connection))
                    {
                        command.CommandType = CommandType.StoredProcedure;
                        connection.Open();
                        using (SqlDataReader reader = command.ExecuteReader())
                        {
                            while (reader.Read())
                            {
                                //fill object
                            }
                        }
                    }
                }                   
            }               
            catch (Exception ex)
            {                   
                context.Logger.LogError($"FunctionHandler::ObtenerInformacionEmpresas - {ex}");
                throw;
            }
        }                        
    }

By knowing the list that comes as parameter, with that logger at line 5 I can see that the error happens at iteration number 1500 aprox (out of 1700 items).

Here are some logs from cloudwatch that illustrate the issue:

As the error suggest, it seems like a memory issue but the max memory usage does not surpass the lambda memory size. As stated before, I have tried max memory configuration and timeout (15 minutes).

Moreover, the lambda runs like a charm on local machine (Windows 10) through mock lambda test tool.

Any help would be much appreciated.

Thanks to everyone for the help,
Turns out that the issue was indeed related to sql connections. I had a wrong understanding of the "using" keyword in c#. In fact, the connection is not really closed despite being in a using block, rather released back to the connection pool. Since all the databases I was connecting to where within the same sql server, all I had to do to fix this was to set "Pooling=false" in the connection strings.

Apologies to everyone for the confussion, its not really aws lambda related error, although the memory log from cloudwatch didnt really help that much.

I leave this link about sql connection pooling on ado.net from microsoft docs for everyone insterested down below.

https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling