@aws-sdk/client-s3 (@aws-sdk/signature-v4-crt) 3.29.0 introduces aws-crt and deprecated dependencies #2750

yo1dog · 2021-09-03T17:19:59Z

Describe the bug

When updating to 3.29.0, @aws-sdk/client-s3 and @aws-sdk/s3-request-presigner now have @aws-sdk/signature-v4-crt as a dependency, which has aws-crt as a dependency, which is both very large (over 200MB) and has several deprecated dependencies. Notably: har-validator@5.1.5, uuid@3.4.0, and request@2.88.2. The total number of dependencies for @aws-sdk/client-s3 also more than triples from 85 to 273.

Your environment

SDK version number

@aws-sdk/client-s3@3.29.0
@aws-sdk/s3-request-presigner@3.29.0
@aws-sdk/signature-v4-crt@3.29.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Node v14.17.6

Steps to reproduce

% npm i @aws-sdk/client-s3@3.29.0
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

added 273 packages, and audited 274 packages in 9s

found 0 vulnerabilities

% du -sh node_modules
244M    node_modules

% npm why har-validator
har-validator@5.1.5
node_modules/har-validator
  har-validator@"~5.1.3" from request@2.88.2
  node_modules/request
    request@"^2.54.0" from cmake-js@6.1.0
    node_modules/cmake-js
      cmake-js@"6.1.0" from aws-crt@1.9.0
      node_modules/aws-crt
        aws-crt@"^1.9.0" from @aws-sdk/signature-v4-crt@3.29.0
        node_modules/@aws-sdk/signature-v4-crt
          @aws-sdk/signature-v4-crt@"3.29.0" from @aws-sdk/middleware-sdk-s3@3.29.0
          node_modules/@aws-sdk/middleware-sdk-s3
            @aws-sdk/middleware-sdk-s3@"3.29.0" from @aws-sdk/client-s3@3.29.0
            node_modules/@aws-sdk/client-s3
              @aws-sdk/client-s3@"^3.29.0" from the root project

% npm i @aws-sdk/client-s3@3.28.0

added 85 packages, and audited 86 packages in 4s

found 0 vulnerabilities

% du -sh node_modules
28M     node_modules

Observed behavior

Warnings about deprecated packages are printed when updating. Very large node_modules directory.

Expected behavior

No deprecated or overly large packages are installed when updating.

The text was updated successfully, but these errors were encountered:

jvlch · 2021-09-03T19:41:19Z

I am also having issues with the new release. My codebuild projects running a serverless package/deploy are unable to correctly trace and package lambdas.

Serverless: [serverless-jetpack] .serverless/examplelambdaname.zip dependency package tracing misses:
--
574 | - node_modules/aws-crt/dist/native/binding.js [55:18]: require(path)

relevant snippet from above file:

const bin_path = path.resolve(source_root, 'bin');
const search_paths = [
    path.join(bin_path, 'native', binary_name),
    path.join(bin_path, platformDir, binary_name),
];
let binding;
for (const path of search_paths) {
    if (fs_1.existsSync(path + '.node')) {
        binding = require(path);
        break;
    }
}

darrinholst · 2021-09-03T20:44:36Z

I'm having issues as well on alpine...

/app # node
Welcome to Node.js v14.17.6.
Type ".help" for more information.
> require('aws-crt')
Segmentation fault
/app #

and

    Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /builds/node_modules/aws-crt/dist/bin/linux-x64/aws-crt-nodejs.node)
      at Runtime._loadModule (node_modules/jest-runtime/build/index.js:991:29)
      at Object.<anonymous> (node_modules/aws-crt/lib/native/binding.js:40:19)

tschleu · 2021-09-04T07:32:59Z

We ran into the size issue of the dependencies:

"Unzipped size must be smaller than 201542732 bytes (Service: Lambda, Status Code: 400, Request ID: 569bd 46f-a948-4fe5-b218-d08ee5c607bd, Extended Request ID: null)"

Even with a minimal dependency like

  "dependencies": {
    "@aws-sdk/client-s3": "3.29.0"
  },

The node_modules folder is 20750416 in size which is already too large to be deployed.

lukasa1993 · 2021-09-04T12:12:05Z

this makes it unusable how this lib breaks at every update ?

tripodsan · 2021-09-04T18:25:57Z

we also run into problems when trying to bundle and deploy on lambda:

2021-09-04T18:17:33.745Z	34da863b-c195-438e-b323-9b03b4d13ed9	ERROR	error while invoking function Error: AWS CRT binary not present in any of the following locations:
	/var/task/node_modules/aws-crt/bin/native/aws-crt-nodejs
	/var/task/node_modules/aws-crt/bin/linux-x64/aws-crt-nodejs
    at Object../node_modules/aws-crt/dist/native/binding.js (/var/task/index.js:163137:11)
    at __webpack_require__ (/var/task/index.js:322288:42)
    at Object../node_modules/aws-crt/dist/native/crt.js (/var/task/index.js:163172:35)
    at __webpack_require__ (/var/task/index.js:322288:42)
    at Object../node_modules/aws-crt/dist/index.js (/var/task/index.js:162591:26)
    at __webpack_require__ (/var/task/index.js:322288:42)
    at Object../node_modules/@aws-sdk/signature-v4-crt/dist/cjs/CrtSignerV4.js (/var/task/index.js:125828:19)
    at __webpack_require__ (/var/task/index.js:322288:42)
    at Object../node_modules/@aws-sdk/signature-v4-crt/dist/cjs/index.js (/var/task/index.js:126098:27)
    at __webpack_require__ (/var/task/index.js:322288:42)

bluepeter · 2021-09-05T04:02:02Z

Ugh. Same here :(

mtalbert · 2021-09-05T15:43:45Z

I am having the same issue. The aws-crt dependency is forcing the filesize of an unpacked Lambda Layer to be larger than the 250MB hard limit. Does anyone know a version of @aws-sdk/client-s3 that does not carry this heavy dependency? I would prefer to use this than revert back to the aws-sdk dependency.

utsav-techwondoe · 2021-09-05T21:15:17Z

I kept on having this issue

    Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /builds/node_modules/aws-crt/dist/bin/linux-x64/aws-crt-nodejs.node)
      at Runtime._loadModule (node_modules/jest-runtime/build/index.js:991:29)
      at Object.<anonymous> (node_modules/aws-crt/lib/native/binding.js:40:19)

For temp. resolving it, I reverted my s3-client lib to 3.26.0

pnutmath · 2021-09-06T07:37:53Z

I have also issue with this version,

Server terminations signals getting after this update

rtrvrtg · 2021-09-06T07:44:45Z

To get my Serverless deployments working after this update, I had to blow away the following files before running sls deploy:

node_modules/aws-crt/crt/aws-lc/crypto/hpke/test-vectors.json
node_modules/aws-crt/crt/aws-lc/generated-src/*.cc
node_modules/aws-crt/crt/aws-lc/third_party/wycheproof_testvectors
node_modules/aws-crt/crt/aws-lc/ssl/test

And while it's not related to aws-crt, since it's still related to the AWS SDK:

node_modules/aws-sdk/dist/*-react-native.js

This is a pretty bad hack, and won't help everyone, but it might help make sure to keep your Lambda bundles free of a lot of the bulky source or test data that won't be used in your PROD environments anyway. It also won't do much to help people who use React Native or need to do testing stuff in their Lambda envs.

hiepxanh · 2021-09-06T07:52:32Z

I like your action ❤️ blow away , BTW you blow away our block today, thank you so much, it such a heavy stone on our way

danrivett · 2021-09-06T20:40:06Z

We just ran into this issue and only caught it so quickly because of the "Unzipped size must be smaller than 262144000 bytes" error when attempting to deploy.

I can also confirm pinning to 3.28.0 resolved the issue until it's addressed in a future release.

The change in 3.29.0 appears to be introduced in PR #2742 so copying @AllanZhengYP.

AllanZhengYP · 2021-09-07T02:18:51Z

@yo1dog @danrivett

Hi, thank you all very much for reporting the issue, I will work with the aws-crt package author to address the dependency and size issue. One alternative is making aws-crt package a peer dependency, and only required when users using multi-regional request.

I will post updates here as we progress. Now the affected version is 3.29.0, while we are working on it, you should be ok if reverted back to 3.28.0

danrivett · 2021-09-07T02:46:19Z

Thanks @AllanZhengYP for the update.

I don't know enough about the library to make a very informed suggestion, but if the vast majority of users of this library do not need multi-region support, perhaps making it an explicit optional peer dependency makes the most sense, even if it only bumps the package size by a few MB if included after fixing aws-crt.

As a few MB added here, and a few MB added by another package can add quickly add up, so the added complexity of a peer dependency may well be worth it.

Also, I'm not sure if tree-shaking can help and we've just not configured webpack properly for example, as I'm going beyond my skill level, but that might be another option. Perhaps aws-crt doesn't support tree-shaking and that is something that could also be addressed if so?

moltar · 2021-09-07T03:06:36Z

Yeah, the package size is definitely something to look at, especially with this library.

A pattern that is becoming common is to use AWS CDK, and use the construct that bundles the code for each Lambda. Which then bundles all of the deps, and one rogue dependency will bump the Lambda bundle sizes to several megabytes.

yo1dog · 2021-09-07T13:11:40Z

A bit off topic, but I agree with the notion of being vigilant of package sizes. @aws-sdk/client-s3@3.28.0 is already 11MB itself and 28MB including dependencies.

AllanZhengYP · 2021-09-07T15:32:16Z

I just deployed a basic Lambda function with only client-s3@3.29.0 client using SAM. The function size is 65MB. It's definitely too big still, but deployable(under Lambda 250 MB limit).

danrivett · 2021-09-07T16:10:54Z

@AllanZhengYP is that the compressed or uncompressed size? Our lambda package had a compressed size of approximately 80MB when using 3.29.0 but was over the 250MB uncompressed limit when unzipped.

When using 3.28.0 the compressed size reduced to 27MB (which is still large, but due to other unrelated dependencies).

AllanZhengYP · 2021-09-07T23:46:40Z

The revert has been released in 3.30.0. The dependency should be removed.

yo1dog · 2021-09-08T00:50:05Z

Appreciate you guys jumping on this so quickly.

abualsamid · 2021-09-09T20:41:14Z

this has haunted me for days. Simply including the getSignedUrl from @aws-sdk/s3-request-presigner breaks the lambda function with the useless error message: "AWS CRT binary not present in any of the following locations..."
All google searches for the CRT point to iot which i am not using. This not only wasted a ton of time, there is no way to figure out why it is breaking.

abualsamid · 2021-09-11T19:40:15Z

upgrading to 3.30.0 solved the issue for me.

…

On Sat, Sep 11, 2021 at 2:35 PM Sean ***@***.***> wrote: I'm running Node in docker with node:14-alpine and the aws-crt binary wants to use ld-linux-x86-64.so.2 but can't find it. This is in order for me to use the V3 S3 client only. I have tried adding this in multiple ways (using info from multiple posts) but without luck and still get a crash when aws-crt tries to access it on a call to any S3 method/command. I'm having to use another S3 library for now. Any info on a solve for this? Is there a different docker image with this file included? I'm very far from a docker expert. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2750 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANUL6QKPVVZRXLTMRRQUU3UBOOOJANCNFSM5DMFCJOQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

henhal · 2021-09-23T12:46:58Z

Is this really solved? Is signature-v-crt a mandatory peer dependency from client-s3 now?

When using webpack to package my code I get this:

Module not found: Error: Can't resolve '@aws-sdk/signature-v4-crt' in '/foo/node_modules/@aws-sdk/middleware-sdk-s3/dist/es'
 @ /foo/node_modules/@aws-sdk/middleware-sdk-s3/dist/es/S3SignatureV4.js 58:45-80
 @ /foo/node_modules/@aws-sdk/middleware-sdk-s3/dist/es/index.js
 @ /foo/node_modules/@aws-sdk/client-s3/dist/es/S3Client.js
 @ /foo/node_modules/@aws-sdk/client-s3/dist/es/index.js

So it seems this module is strictly required, although package.json says

"dependencies": {
    "@aws-sdk/protocol-http": "3.32.0",
    "@aws-sdk/signature-v4": "3.33.0",
    "@aws-sdk/types": "3.32.0",
    "@aws-sdk/util-arn-parser": "3.32.0",
    "tslib": "^2.3.0"
  },
  "devDependencies": {
    "@aws-sdk/signature-v4-crt": "3.33.0",
    "@types/jest": "^26.0.4",
    "jest": "^26.1.0",
    "typescript": "~4.3.5"
  },
  "peerDependencies": {
    "@aws-sdk/signature-v4-crt": "^3.31.0"
  },

talkingnews · 2021-10-03T14:37:03Z

I don't know why this and the related issues have been closed - the bug with the massive package size and deprecated dependencies is back in 3.34.0, so for now I have to stick with 3.28.0

$ npm install @aws-sdk/client-s3
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

added 283 packages, and audited 284 packages in 8s

9 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ du -hs node_modules/
61M     node_modules/

But then...

$ npm i @aws-sdk/client-s3@3.28.0

removed 196 packages, changed 69 packages, and audited 88 packages in 15s

2 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ du -hs node_modules/
28M     node_modules/

danrivett · 2021-10-03T19:30:38Z

We are still pinned to 3.28.0 as have had higher priority work than upgrading and rereviewing, so I can't speak to this, but am also interested in these issues raised recently, as sooner rather than later we will upgrade.

moltar · 2021-10-04T02:54:43Z

Interestingly, I am not seeing the bundling issue on the v3.34.0.

Lambda packages are of normal size (for us).

During the last breakage, these sizes have ballooned to an extreme size, outgrowing the max limit for the lambda package size.

rvitaliy · 2021-10-08T10:27:20Z

same problem with lates version(3.36.0) i rollback to 3.28.0

warning "@aws-sdk/client-s3 > @aws-sdk/middleware-sdk-s3@3.36.0" has unmet peer dependency "@aws-sdk/signature-v4-crt@^3.31.0".

superjose · 2021-10-08T21:54:59Z

I can confirm this as well. I recently upgraded my SDK from ver 2 to ver 3 and glad I found this thread!

kyeotic · 2021-10-15T15:53:06Z

@AllanZhengYP please re-open this, the issue has returned in 3.36.0 for several libraries. I encountered it with dynamodb

davidcheal · 2021-10-19T21:25:43Z

Getting similar issues trying to use
client-s3@3.37.0 => "Can't resolve '@aws-sdk/signature-v4-crt'"
client-secrets-manager@3.37.0 => "Can't resolve 'aws-crt'"

bluepeter · 2021-10-23T18:40:10Z

Is anyone at AWS actively trying to solve this? cc @AllanZhengYP

github-actions · 2021-11-07T00:01:55Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

yo1dog added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 3, 2021

LoneRifle mentioned this issue Sep 6, 2021

Revert "build(deps): bump @aws-sdk/client-s3 from 3.28.0 to 3.29.0 in /server" opengovsg/askgovsg#222

Merged

berenddeboer mentioned this issue Sep 6, 2021

AWS CRT binary not present in any of the following locations #2753

Closed

StuAA78 mentioned this issue Sep 6, 2021

Bump @aws-sdk/client-s3 from 3.28.0 to 3.29.0 DEFRA/sroc-charging-module-api#535

Closed

DavidWells mentioned this issue Sep 6, 2021

Pnpm install issues awslabs/aws-crt-nodejs#243

Closed

AllanZhengYP removed the needs-triage This issue or PR still needs to be triaged. label Sep 7, 2021

AllanZhengYP self-assigned this Sep 7, 2021

AllanZhengYP mentioned this issue Sep 7, 2021

@aws-sdk/client-s3 makes node_modules too large #2747

Closed

AllanZhengYP mentioned this issue Sep 7, 2021

fix(client-s3): revert MRAP customizations #2759

Merged

AllanZhengYP closed this as completed in #2759 Sep 7, 2021

dphang mentioned this issue Sep 8, 2021

feat: Add domain locale detection for rewrites and non default locales (#1640) serverless-nextjs/serverless-next.js#1641

Merged

AllanZhengYP mentioned this issue Sep 15, 2021

To solve the npm package size issue awslabs/aws-crt-nodejs#252

Merged

henhal mentioned this issue Sep 23, 2021

Missing dependency @aws-sdk/signature-v4-crt after updating to client-s3 3.33.0 #2806

Closed

AllanZhengYP mentioned this issue Sep 24, 2021

S3 Multi-Region Access Point(MRAP) is not available unless with additional dependency #2822

Open

zraineri mentioned this issue Oct 30, 2021

Missing Peer Dependency | @aws-sdk/client-s3 missing @aws-sdk/signature-v4-crt #2960

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

@aws-sdk/client-s3 (@aws-sdk/signature-v4-crt) 3.29.0 introduces aws-crt and deprecated dependencies #2750

@aws-sdk/client-s3 (@aws-sdk/signature-v4-crt) 3.29.0 introduces aws-crt and deprecated dependencies #2750

yo1dog commented Sep 3, 2021 •

edited

Loading

jvlch commented Sep 3, 2021

darrinholst commented Sep 3, 2021

tschleu commented Sep 4, 2021

lukasa1993 commented Sep 4, 2021

tripodsan commented Sep 4, 2021

bluepeter commented Sep 5, 2021 •

edited

Loading

mtalbert commented Sep 5, 2021 •

edited

Loading

utsav-techwondoe commented Sep 5, 2021

pnutmath commented Sep 6, 2021

rtrvrtg commented Sep 6, 2021 •

edited

Loading

hiepxanh commented Sep 6, 2021

danrivett commented Sep 6, 2021 •

edited

Loading

AllanZhengYP commented Sep 7, 2021 •

edited

Loading

danrivett commented Sep 7, 2021

moltar commented Sep 7, 2021

yo1dog commented Sep 7, 2021

AllanZhengYP commented Sep 7, 2021

danrivett commented Sep 7, 2021

AllanZhengYP commented Sep 7, 2021

yo1dog commented Sep 8, 2021

abualsamid commented Sep 9, 2021

abualsamid commented Sep 11, 2021 via email

henhal commented Sep 23, 2021

talkingnews commented Oct 3, 2021 •

edited

Loading

danrivett commented Oct 3, 2021

moltar commented Oct 4, 2021

rvitaliy commented Oct 8, 2021 •

edited

Loading

superjose commented Oct 8, 2021

kyeotic commented Oct 15, 2021

davidcheal commented Oct 19, 2021

bluepeter commented Oct 23, 2021

github-actions bot commented Nov 7, 2021

@aws-sdk/client-s3 (@aws-sdk/signature-v4-crt) 3.29.0 introduces aws-crt and deprecated dependencies #2750

@aws-sdk/client-s3 (@aws-sdk/signature-v4-crt) 3.29.0 introduces aws-crt and deprecated dependencies #2750

Comments

yo1dog commented Sep 3, 2021 • edited Loading

Describe the bug

Your environment

SDK version number

Is the issue in the browser/Node.js/ReactNative?

Details of the browser/Node.js/ReactNative version

Steps to reproduce

Observed behavior

Expected behavior

jvlch commented Sep 3, 2021

darrinholst commented Sep 3, 2021

tschleu commented Sep 4, 2021

lukasa1993 commented Sep 4, 2021

tripodsan commented Sep 4, 2021

bluepeter commented Sep 5, 2021 • edited Loading

mtalbert commented Sep 5, 2021 • edited Loading

utsav-techwondoe commented Sep 5, 2021

pnutmath commented Sep 6, 2021

rtrvrtg commented Sep 6, 2021 • edited Loading

hiepxanh commented Sep 6, 2021

danrivett commented Sep 6, 2021 • edited Loading

AllanZhengYP commented Sep 7, 2021 • edited Loading

danrivett commented Sep 7, 2021

moltar commented Sep 7, 2021

yo1dog commented Sep 7, 2021

AllanZhengYP commented Sep 7, 2021

danrivett commented Sep 7, 2021

AllanZhengYP commented Sep 7, 2021

yo1dog commented Sep 8, 2021

abualsamid commented Sep 9, 2021

abualsamid commented Sep 11, 2021 via email

henhal commented Sep 23, 2021

talkingnews commented Oct 3, 2021 • edited Loading

danrivett commented Oct 3, 2021

moltar commented Oct 4, 2021

rvitaliy commented Oct 8, 2021 • edited Loading

superjose commented Oct 8, 2021

kyeotic commented Oct 15, 2021

davidcheal commented Oct 19, 2021

bluepeter commented Oct 23, 2021

github-actions bot commented Nov 7, 2021

yo1dog commented Sep 3, 2021 •

edited

Loading

bluepeter commented Sep 5, 2021 •

edited

Loading

mtalbert commented Sep 5, 2021 •

edited

Loading

rtrvrtg commented Sep 6, 2021 •

edited

Loading

danrivett commented Sep 6, 2021 •

edited

Loading

AllanZhengYP commented Sep 7, 2021 •

edited

Loading

talkingnews commented Oct 3, 2021 •

edited

Loading

rvitaliy commented Oct 8, 2021 •

edited

Loading