Support using the SDK with browserify

[konklone]

Right now, the AWS SDK for JavaScript is usable by browserify, but with constraints:

• You have to download the minified version from the CDN, not use npm. If you use the version from npm, you get an error:

From this place:

• When freezing aws-sdk into your project, you can only use the minified version, not the full version. I'm not sure what the processing is for the minified version, though I can see it uses many less require() calls.
• When freezing the minified version of aws-sdk into your project, you have to add the following to the bottom of the .js file:

module.exports = AWS;

The fact that I can get it working via browserify like this suggests it shouldn't be a huge lift for the project to support browserify via npm.

The benefit of using it via browserify, compared to using it in a more traditional way, is that it makes it possible to combine the AWS SDK with other Node-packaged modules in npm (including those designed to work with this library in Node-land).

In my case, I'm connecting filereader-stream' to s3-upload-stream in the browser to pipe data from disk directly into S3 using the Multipart Upload API. Stream syntax is incredibly helpful at gluing complicated workflows together in a simple way, and I wouldn't be able to pull this off without using browserify.

[lsegal]

@konklone unfortunately it would be difficult to enable browserifying the npm package of the SDK. Because of the way we dynamically load service models (via dynamic require), the SDK is actually not compatible with pure browserify-- we actually build the hosted version with a script that manually adds in service models after browserify runs (see here). The error you are seeing when running browserify on the package would actually be the least of your problems-- once you get past that one you will run into many more related to services.

We're open to suggestions on how to make it more browserify-friendly if you have any, but browserify doesn't handle dynamic requires, which we take advantage of in Node.js. At the end of the day you'll need to use a specially built version of the SDK, especially if you want to take advantage of services that are not part of the default hosted build / npm package.

Note that it should be fairly easy to use browserify's multitude of options to work around this without changing the SDK. The following command worked for me:

$ browserify -r ./aws.js:aws-sdk index.js

Where index.js is:

var AWS = require('aws-sdk');
var s3 = new AWS.S3();

And aws.js is additionally placed in your project to load the SDK with the following single line:

module.exports = window.AWS;

You still won't get the SDK to use Stream objects though, as we don't use or support them in the browser. I haven't tested whether the SDK whether this affects the usage of s3-upload-stream, but it might.

[konklone]

Note that it should be fairly easy to use browserify's multitude of options to work around this without changing the SDK.

OK, that works for me too. Then, could this issue be resolved by adding a section to the docs about using browserify with AWS?

Also, would it be possible to add a line to the resulting build that checks for the existence of module.exports and window (which I think is only going to happen in a browserify environment) and automatically exports AWS the way that you've described?

You still won't get the SDK to use Stream objects though, as we don't use or support them in the browser. I haven't tested whether the SDK whether this affects the usage of s3-upload-stream, but it might.

That's all right - s3-upload-stream presents a stream input interface, with which it builds little 5MB+ buffers that it hands off to the AWS SDK to perform parts of multipart uploads. So it basically acts as the Stream interface that this SDK doesn't support in-browser. The important part of s3-upload-stream is that it can have things .pipe()-d into it.

[lsegal]

Also, would it be possible to add a line to the resulting build that checks for the existence of module.exports and window (which I think is only going to happen in a browserify environment) and automatically exports AWS the way that you've described?

Unfortunately this is not that simple to do. You would still need to stub out the aws-sdk package to point to the prebuilt file. I've done some quick tests against adding module.exports = AWS to the built aws-sdk.js file and browserifying with the following command:

$ browserify -r ./path/to/aws-sdk.js:aws-sdk index.js

But I'm currently getting resolution errors from browserify:

Error: Cannot find module './sha' from './path/to'

If you have any insights on getting this to work, I'm all ears. Right now I think the best solution is to provide the separate module.exports = AWS hook so as to indirectly link to AWS without browserifying anything in. That also gives you the benefit of being able to load the SDK from the CDN where it would be cached without having to host your entire own version browserified into your code.

On a sidenote, I agree that we can improve our docs to describe this use case more clearly.

[lsegal]

@konklone I have a WIP branch (npm-browserifiable) that allows you to call browserify index.js on a file that requires the aws-sdk without any extra arguments (no hacks). It also allows you to pass in a list of services via environment variable (similar to our browser building script) when calling browserify to get a customized build!

Take a look at the branch and let me know if this works for you. You can test it out by using:

tmp$ cat index.js
var AWS = require('aws-sdk');
s3 = new AWS.S3();
tmp$ npm install git://github.com/aws/aws-sdk-js#npm-browserifiable
tmp$ browserify index.js > compiled.js

[mablack]

@lsegal I was getting a Fatal error: Cannot call method 'sort' of undefined with the following stacktrace:

TypeError: Cannot call method 'sort' of undefined
    at APP_DIR/node_modules/aws-sdk/lib/api_loader.js:39:71
    at Array.forEach (native)
    at buildServiceMap (APP_DIR/node_modules/aws-sdk/lib/api_loader.js:36:27)
    at Object.getServices [as services] (APP_DIR/node_modules/aws-sdk/lib/api_loader.js:46:3)
    at Object.<anonymous> (APP_DIR/node_modules/aws-sdk/lib/services.js:7:5)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)

It looks like it was caused by some missing files - the aws-sdk/apis/****.normal.json files were not installed via npm install, so the regex on line 22 of aws-sdk/lib/api_loader.js fails to match any file names. Changing the regex to match the aws-sdk/apis/****.min.json files seems to fix the issue:

var match = file.match(/^(.+?)-(\d+-\d+-\d+)\.min\.json$/);

Thanks for working on this!!

[lsegal]

@mablack this was actually a great catch! Thanks! Avoided a potentially botched release :) That was related to the way we recently changed filenames in #379. We try to glob on the .normal. files but those files are excluded from npm installs, so this would have broken the package!

Just pushed a fix to master and merged it over into the browserifiable branch. Try now?

[mablack]

@lsegal all good now!! :)

[konklone]

Sorry I didn't have a chance to test this, but thank you @lsegal for making this happen in spite of the challenges! I'll be un-freezing my aws-sdk dependency right away. 😸

[lsegal]

Just FYI, this is now in master and will be available in our next release. Thanks for the feature suggestion and help testing, @konklone and @mablack!

[lsegal]

Release is out. You should now be able to use this from a regular npm install.

[mablack]

Great, just switched to the latest release - thanks again for working on this @lsegal!

[Fedia]

@lsegal
Broken on Windows.
In dist-tools/transform.js the path separator is supposed to be /. The condition below is never satisfied on Windows.

if (file.match(/\/lib\/browser\.js$/)) { // dirty fix: /[\/\\]lib[\/\\]browser\.js$/
    stream.push(license);
    var src = collector(process.env.AWS_SERVICES);

[doapp-ryanp]

@lsegal I'm working on using browserify in nodejs, with the output being targeted at a lambda function. FWIW I'm doing this as part of my work on the new JAWS project - essentially making it easier to use Lambda/api gateway etc.

Anyways, we'd like to use browserify to keep the footprint small (to combat the well documented cold start problems). We also want to bundle aws-sdk for node so we can target specific versions and not rely on what is exposed in the lambda nodejs runtime.

We are concerned with the dynamic require brought up in this issue, however it looks like you have it solved in >= v2.0.20

I'm struggling with when its necessary to use the AWS_SERVICES env var and when it is not. My question is, when do you need to set this env var? From the example I see in building the sdk it looks like it is not required. What am I missing here?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.