Lambda: cdk deploy suddenly getting ECONNRESET on asset publish

[matan-orland-lmnd]

Describe the bug

I have a stack that contains docker images. I've been deploying it locally for the past 4 months without a hitch (deploy also worked on my partner's computer), and then suddenly today I've started getting fail: read ECONNRESET. I'm using PythonLambda construct, as well as DockerImageFunction construct. All fail.

When running cdk deploy with --verbose I see these errors are as follows:

[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)

and

[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)

Other AWS CLI requests to the account work (such as to ECR).

This started today on both computers I and my partner have access to. Could not find a solution for the life of me. both AWS CLI and AWS CDK, and all libraries are on latest versions.

Deploy finishes with:

Deployment failed: Error: Failed to publish asset 109bd108d6c60e825885f808d1cf43f42060382afaceed0935d0028c2d821a18:REDACTED-us-east-1
    at Deployments.publishSingleAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:11655)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.publishAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:180233)
    at async /Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:164772

No issues on AWS status pages reported for S3.

Expected Behavior

Stack should deploy and assets should be published.

Current Behavior

Failing with:

Deployment failed: Error: Failed to publish asset 109bd108d6c60e825885f808d1cf43f42060382afaceed0935d0028c2d821a18:REDACTED-us-east-1
    at Deployments.publishSingleAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:11655)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.publishAsset (/Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:180233)
    at async /Users/matan.orland/.nvm/versions/node/v18.9.0/lib/node_modules/aws-cdk/lib/index.js:424:164772

after getting:

[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)

and

[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)

Reproduction Steps

Create a stack in us-east-1 which uses the PythonLambda or the DockerImageFunction constructs. Example usage:

    const lambda = new lambda.DockerImageFunction(this, 'some-name', {
      functionName: 'some-name',
      code: lambda.DockerImageCode.fromImageAsset('path/to/dockerfile'),
      memorySize: 2048,
      ephemeralStorageSize: cdk.Size.gibibytes(2),
      architecture: lambda.Architecture.X86_64,
      timeout: cdk.Duration.minutes(10),
      retryAttempts: 0,
    })

Possible Solution

I have read somewhere that this is because the underlying AWS SDK uses V4 signatures, and S3 expects V2. Not sure if that is the case here, or why it would change all of a sudden.

Additional Information/Context

I have already tried several versions of CDK, as well as to delete the CDKToolkit stack and reinstall it. Nothing worked.

CDK CLI Version

2.122.0 (build 7e77e02)

Framework Version

No response

Node.js Version

v18.9.0

OS

MacOS 14.2.1 (23C71) (Sonoma)

Language

TypeScript

Language Version

No response

Other information

Thank you <3

[matan-orland-lmnd]

Update - I tried deploying my stack to us-west-2 and it seems to be working fine. I guess something is wrong with S3 in us-east-1?

[pahud]

[23:47:53] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1"}) => read ECONNRESET (code=TimeoutError)

[23:47:32] Call failed: listObjectsV2({"Bucket":"cdk-hnb659fds-assets-REDACTED-us-east-1","Prefix":"1ce78b814fad9fc710c056afbe94fd1fe40b81d185a68ffa975c6079c9a9cd53.zip","MaxKeys":1}) => read ECONNRESET (code=TimeoutError)

Looks like it's a timeout error. Are you able to list all objects in the bucket using AWS CLI?

[matan-orland-lmnd]

@pahud thank you, yes I am able to list no problem.

aws s3 ls s3://cdk-hnb659fds-assets-REDACTED-us-east-1

works just fine.

Just to add some clarification from before - my partner's computer is not only on a different internet connection, but in a different city.