Slim handler #548

mcrowson · 2016-12-18T04:12:42Z

Description

Adds an option to zappa_settings.json for "slim_handler" which defaults to false. When set to true does the following:

Creates 2 Zip files for S3. The first is a small handler-only zip (apx 5M) that contains the Zappa handler.py file and zappa's dependencies. The second is the application with all of its dependencies.
Copies the versioned application zip to proj_current_project.zip
Adds a ZIP_PATH=proj_current_project.zip line to the zappa_settings.py module that is loaded by the handler.
When the handler sees the ZIP_PATH= line, it downloads that zip from S3 into /tmp, unzips it, and adds it to path.

Known Issues:

~~The project zip needs to remain in S3 so that the handler can download it as needed. There is not a way to remove old project zip files. They persist in S3.~~ Resolved

GitHub Issues

#510

coveralls · 2016-12-18T04:19:51Z

Coverage decreased (-0.8%) to 79.965% when pulling cf4d959 on mcrowson:slim_handler into 60cd824 on Miserlou:master.

bjinwright · 2016-12-18T14:25:25Z

How long would the files stay in cache?

mcrowson · 2016-12-18T14:31:35Z

I haven't done exhaustive testing on that. My assumption is that as long as the function is warm and that is what I saw on my limited testing.

…

On Dec 18, 2016, at 9:25 AM, Brian Jinwright ***@***.***> wrote: How long would the files stay in cache? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

… for verionsed project zip files to be removed from S3 as per zappa_settings., metching current functionality.

coveralls · 2016-12-18T18:50:14Z

Coverage decreased (-1.1%) to 79.625% when pulling 5ad7b99 on mcrowson:slim_handler into 60cd824 on Miserlou:master.

mcrowson · 2016-12-18T18:55:47Z

@bjinwright I've had keep warm events and been making requests while working on stuff. It has kept the project in /tmp for going on 30 minutes now and hasn't had to re-download from S3. I'll keep checking as it goes on.

mcrowson · 2016-12-19T17:38:04Z

@Miserlou let me know if there is anything else needed for this.

kapilt · 2017-01-04T11:45:38Z

is the goal about keeping the size small ? or supporting large projects over 250mb ?

for keeping the size small which i think is the more universal win, conceptually, this feels PR like the wrong solution and just adds complexity and cold start latency, the right solution imo, would focus on keeping the zappa handler size minimal and getting a better default exclude. the fixed overhead on zappa handler should only be 1mb uncompressed (werkzeug and requestlogger) ~ +310k on the zip

mcrowson · 2017-01-04T13:55:54Z

The lambda max is 50mb, not 250. The goal of this PR was to support larger projects above 50mb. A project with SciPy alone as a dependency is approaching 35mb. This only adds to cold start for the initial launch. If you have the keep warm event it responds just as quickly on subsequent requests as the temp folder is cached and shared across lambada calls.

…

Sent from my iPhone

On Jan 4, 2017, at 5:45 AM, Kapil Thangavelu ***@***.***> wrote: is the goal about keeping the size small ? or supporting large projects over 250mb ? for keeping the size small which i think is the more universal win, conceptually, this feels PR like the wrong solution and just adds complexity and cold start latency, the right solution imo, would focus on keeping the zappa handler size minimal and getting a better default exclude. the fixed overhead on zappa handler should only be 1mb uncompressed (werkzeug and requestlogger) ~ +310k on the zip — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

smorin · 2017-01-04T16:28:29Z

Here are the actual limits because it's both 50 and 250 depending on what your exactly talking about. There should be checks for both limits.

I think the ability to get around these limits is another aspect all together as reference in this bug.

There should be a check for what's being copied to tmp because of the /tmp space limits

Ephemeral disk capacity ("/tmp" space)	512 MB

http://docs.aws.amazon.com/lambda/latest/dg/limits.html

AWS Lambda Deployment Limits

Item	Default Limit
Lambda function deployment package size (.zip/.jar file)	50 MB
Total size of all the deployment packages that can be uploaded per region	75 GB
Size of code/dependencies that you can zip into a deployment package (uncompressed zip/jar size)	250 MB
Total size of environment variables set	4 KB

kapilt · 2017-01-09T19:59:21Z

so my thought is time spent reducing zappa's overhead from 10mb of the 50mb compressed zip max size to under 1mb, will give more headroom for all apps, and cut the bloat from zappa. the work associated to #556 was to enable that, the work already committed shaved about 7mb off the zip.

for projects that really want bring in the kitchen sink/large bins though this approach is the only that will really work, though it would be nice to see a unit test.

smorin · 2017-01-09T20:06:29Z

That sounds very good to me from that perspective

…

On Mon, Jan 9, 2017 at 11:59 AM, Kapil Thangavelu ***@***.***> wrote: so my thought is time spent reducing zappa's overhead from 10mb of the 50mb compressed zip max size to under 1mb, will give more headroom for all apps, and cut the bloat from zappa. the work associated to #556 <#556> was to enable that, the work already committed shaved about 7mb off the zip. for projects that really want bring in the kitchen sink/large bins though this approach is the only that will really work, though it would be nice to see a unit test. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#548 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHnL5GRrrJdZqSeXslItf_Nkrd5BXG2ks5rQpGbgaJpZM4LQC2_> .

-- Steve Morin | Hacker, Entrepreneur, Startup Advisor twitter.com/SteveMorin | stevemorin.com *Live the dream start a startup. Make the world ... a better place.*

narfman0 · 2017-01-12T19:19:53Z

Thanks for this! Long term it might be handy to collect packages, measure size, and do this automatically if >50MB. 👍

mcrowson · 2017-01-12T19:42:23Z

I agree. I think for now it seems like something to opt into for a few reasons:

Applications may already be taxing their /tmp and a default change like this could have unknown side effects
This code isn't as battle tested as the rest of Zappa. All of my tests indicate it works well, but there are always edge cases that we can find later.
Currently there is a warning in cli.py if the zip is too large that informs the user about the slim_handler setting.

I looked into adding more tests, but I'm just not sure where. Any suggestions?

# Conflicts: # README.md # tests/tests.py # zappa/cli.py

coveralls · 2017-01-26T17:50:21Z

Coverage decreased (-1.2%) to 81.153% when pulling f672b82 on mcrowson:slim_handler into 953223b on Miserlou:master.

Matthew Crowson added 3 commits December 17, 2016 22:26

well, now i just have to work on tests

fbb5387

let's give it a go

78c888a

some comment updates

cf4d959

using a current project zip to persist for slim handlers. This allows…

5ad7b99

… for verionsed project zip files to be removed from S3 as per zappa_settings., metching current functionality.

Miserlou mentioned this pull request Dec 20, 2016

small lambdas by default (zappa on diet) #556

Open

Merge branch 'upstream_master' into slim_handler

f672b82

# Conflicts: # README.md # tests/tests.py # zappa/cli.py

Miserlou merged commit 4ae771e into Miserlou:master Jan 26, 2017

mcrowson deleted the slim_handler branch May 14, 2017 19:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slim handler #548

Slim handler #548

mcrowson commented Dec 18, 2016 •

edited

Loading

coveralls commented Dec 18, 2016 •

edited

Loading

bjinwright commented Dec 18, 2016

mcrowson commented Dec 18, 2016 via email

coveralls commented Dec 18, 2016 •

edited

Loading

mcrowson commented Dec 18, 2016

mcrowson commented Dec 19, 2016

kapilt commented Jan 4, 2017

mcrowson commented Jan 4, 2017 via email

smorin commented Jan 4, 2017

kapilt commented Jan 9, 2017

smorin commented Jan 9, 2017 via email

narfman0 commented Jan 12, 2017

mcrowson commented Jan 12, 2017 •

edited

Loading

coveralls commented Jan 26, 2017 •

edited

Loading

Slim handler #548

Slim handler #548

Conversation

mcrowson commented Dec 18, 2016 • edited Loading

Description

GitHub Issues

coveralls commented Dec 18, 2016 • edited Loading

bjinwright commented Dec 18, 2016

mcrowson commented Dec 18, 2016 via email

coveralls commented Dec 18, 2016 • edited Loading

mcrowson commented Dec 18, 2016

mcrowson commented Dec 19, 2016

kapilt commented Jan 4, 2017

mcrowson commented Jan 4, 2017 via email

smorin commented Jan 4, 2017

kapilt commented Jan 9, 2017

smorin commented Jan 9, 2017 via email

narfman0 commented Jan 12, 2017

mcrowson commented Jan 12, 2017 • edited Loading

coveralls commented Jan 26, 2017 • edited Loading

mcrowson commented Dec 18, 2016 •

edited

Loading

coveralls commented Dec 18, 2016 •

edited

Loading

coveralls commented Dec 18, 2016 •

edited

Loading

mcrowson commented Jan 12, 2017 •

edited

Loading

coveralls commented Jan 26, 2017 •

edited

Loading