Connection to APM Server timed out from AWS Lambda using sqlalchemy/postgres RDS database

[fjoanr]

Describe the issue:

We have a lambda function within a VPC in AWS that is used to write and update entries in one of our RDS databases using sqlalchemy for postgres db.

Setting the capture_serverless() decorator in this lambda displays the following logs (lambda does not fail, thus not a critical bug)

INIT_START Runtime Version: python:3.8.v15	Runtime Version ARN: arn:aws:lambda:xxx::runtime:xxx

[WARNING]	Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7efa6cad90>: Failed to establish a new connection: [Errno 111] Connection refused')': /

[WARNING]	Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7efa5ab310>: Failed to establish a new connection: [Errno 111] Connection refused')': /

[WARNING]	Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7efa5abb20>: Failed to establish a new connection: [Errno 111] Connection refused')': /

> all logs related to the lambda execution appear - lambda finishes successfully

> same Retry logs appear again

[ERROR]	Failed to submit message: 'Connection to APM Server timed out (url: http://localhost:8200/intake/v2/events, timeout: 5 seconds)'

> END

I have tested and validated that the issue is not related to the VPC not having internet access, as all other lambdas in the same VPC work as expected.

I found a similar issue elsewhere, where they pointed that the localhost URL was the issue, and should be updated in lambda's ENV VARS, but we have them properly set up:

ELASTIC_APM_ENVIRONMENT 
ELASTIC_APM_HOSTNAME
ELASTIC_APM_LAMBDA_APM_SERVER 
ELASTIC_APM_LOG_ECS_REFORMATTING
ELASTIC_APM_LOG_LEVEL
ELASTIC_APM_SECRETS_MANAGER_API_KEY_ID 

We start the connection to RDS via a Singleton class, but I am unsure whether that would be any issue, since the connection error happens on module loading already.

I found this issue #1711 - which seems they have it working as expected, so I haven't considered adding additional elasticapm calls within our code, so I don't really know what to do in this case.

Any help appreciated!

Environment (please complete the following information)

• OS: AWS Lambda Python
• Python version: 3.8
• Framework and version: sqlalchemy==1.4.44, psycopg2-binary==2.9.5
• APM Server version: 6.14.0
• Agent version:

[beniwohli]

Hey @fjoanr

Did you set up your lambda function with our extension? The extension provides a forwarder that runs next to your function (hence localhost:8200) and forwards the data collected by the agent to your real APM Server.

[fjoanr]

OMG, somehow, somewhere, that lambda layer was not added to this specific function. Well, I'll see myself out 🤣

P.S. thanks for the comment, sometimes all it takes is a finger pointing at the right place. Cheers!