Load Project and Site-Packages from S3

[mcrowson]

Currently, the entire application and site packages are uploaded to Lambda. This issue proposes uploading the project contents and deps to S3 instead, and having the zappa handler load the project files.

Proposed Changes:

• Zappa CLI: Zip up minimal handler application with Zappa and dependencies for lambda
• Zappa CLI: Send project zip to S3
• Handler: On handler initialization, load the project zip in S3 into /tmp/{project_name}
• Handler: Add the /tmp/{project_name} to sys path.

The first time the application runs it will likely be slower pulling from S3, but if the application uses the keep_warm appropriately, then the subsequent loads shouldn't have the same slowdown due to lambda keeping /tmp around between function calls.

The benefit of this change would be allowing much larger applications to run on Lambda.

Thoughts? Am I missing something?
Credits to @mathom for suggesting P3 via S3 this way.

[mathom]

You probably want to check the file hash somehow (s3 etag) in case you push an update to s3 and need to pull down over an already unzipped project in a warm handler. Not sure if that's really an issue depending on how long your stuff lives but it might be a while with the keep warm setting.

It should be pretty obvious when the application is actually having to download and unzip in the timings, so it will be easy to keep an eye on.

[mcrowson]

Thanks @machbio for wanting to help (as per Slack convo). Here is my fork for getting going. https://github.com/mcrowson/Zappa/tree/project-to-s3

There are a few things I'm working through now.

• We will need to keep the project in S3. The current zappa paradigm cleans up after itself in S3, so we might need a new settings for how many of these to retain and where.
• As this approach is only needed for projects over 50 MB, we might want this feature to only take place if the size of the packaged zip is > 50 MB. This way smaller projects can avoid the additional startup time on cold-starts.

[mathom]

A cool side effect of keeping your project in a bucket is you could move all the lambda management stuff over to cloud formation, too. Not sure if that's worth pursuing at the moment but it's def an option to simplify some of the update stuff.

[machbio]

Sorry about doubting '/tmp' space guarantee - its pretty clear that '/tmp' space is 512MB as given in the Limit - http://docs.aws.amazon.com/lambda/latest/dg/limits.html#limits-list

Ephemeral disk capacity ("/tmp" space) 512 MB

and also on the FAQ page for Lambda - https://aws.amazon.com/lambda/faqs/

Q: What if I need scratch space on disk for my AWS Lambda function?
Each Lambda function receives 500MB of non-persistent disk space in its own /tmp directory.