Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update/deploy a lambda using a Docker container image #967

Merged
merged 3 commits into from
Jun 19, 2021

Conversation

ian-whitestone
Copy link
Contributor

Migrated from Miserlou/Zappa#2192

Background

As discussed in #2188, AWS recently announced container image support for AWS Lambda. This means you can now package and deploy lambda functions as container images, instead of using zip files. The container image based approach will solve a lot of headaches caused by the zip file approach, particularly with file sizes (container images can be up to 10GB) and the dependency issues we all know & love.

What does this PR do?

  1. Make the zappa update and zappa deploy commands accept a new docker_image_uri parameter. docker_image_uri must point to an image in Elastic Container Registry (ECR) that complies with these guidelines. Instructions on how to configure this docker image will be provided in an accompanying blog post.
  2. Add a new save-python-settings-file CLI command, which will be used to generate & save the zappa_settings.py file that must be included in your docker image

Reproducible example

See this repo which contains an example Dockerfile that can be used for testing.

Discussion points for reviewers

  • Test cases I added
  • Accompanying blog post we can point users to for a more in-depth explanation/overview
  • Any other key zappa workflows that should be updated and/or tested in conjunction with this? I've tested the zappa update, zappa deploy, zappa rollback` commands for both creating from a zip file (traditional method) and from a docker image (new method)
  • If you would like this PR to be a bit smaller, I can pull out some of the stuff (new save-python-settings-file command, wait until lambda is ready function, etc.) into separate PRs

Future Work & Next Steps

  • Add functionality to zappa to create new docker images. We've lost some of the zappa magic since you can no longer do a "1-line create & deploy" with zappa deploy, you need to first create your own docker and then deploy.
  • Implement zappa rollback, see this comment for details

Copy link

@techdragon techdragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I cant merge this, it does look quite good to me. I did just have the one small stylistic thought while reading through.

zappa/core.py Outdated Show resolved Hide resolved
Copy link

@Quidge Quidge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering when Zappa would support those new AWS Docker features!

Some non-functional changes can be removed -- I think your editor may have accidentally modified some docstrings (unless you meant to do that).

@techdragon
Copy link

I was wondering when Zappa would support those new AWS Docker features!

Some non-functional changes can be removed -- I think your editor may have accidentally modified some docstrings (unless you meant to do that).

Nice catch on the docstring changes @Quidge. The blank lines in the docstrings should be there. PEP 257 docstring formatting specifies using blank lines to break up multi-line docstrings the way they are in the code currently. https://www.python.org/dev/peps/pep-0257/#id17

Can you touch this up as well @ian-whitestone?

@ian-whitestone
Copy link
Contributor Author

Can you touch this up as well @ian-whitestone?

@techdragon @Quidge I'm just waiting on a final review from @jneves. I will fix those up together with any other changes requested by @jneves

@onimitch
Copy link

onimitch commented May 6, 2021

As someone about to get started with Zapper I just want to say a big thank you for doing this. Support for Docker was a bit of a sticking point so I'm really thankful you've spent the time to add it and PR it @ian-whitestone

@ian-whitestone
Copy link
Contributor Author

@jneves do you think we'll be able to move this forwards?

@onimitch
Copy link

By the way, in the example on your blog, in the Dockerfile you copy in the zappa handler, but I think you can avoid this by referencing the handler like this?

CMD [ "zappa.handler.lambda_handler" ]

@onimitch
Copy link

One gotcha (sort of) I found with the docker approach is if you use environment_variables in the Zappa file, they won't update unless you rebuild the docker image. To get around this, I used aws_environment_variables which means they don't need to be included in the docker image. I don't think there's really anything that can be done about this but might be worth mentioning in the docs when talking about environment variables.

@ian-whitestone
Copy link
Contributor Author

Thanks @onimitch , i'll test out both of those things / update the post when I come back to this.

Right now @jneves is the only maintainer of this project and I haven't heard from him in a few months, so TBH i'm not sure if this will get merged unfortunately.

@onimitch
Copy link

CMD [ "zappa.handler.lambda_handler" ]

Update on this.
It breaks the warm up callback because that's set to look for handler.keep_warm_callback

So it seems like for now, copying in the handler is the best route.

@anush0247
Copy link

@ian-whitestone good work, we are looking forward for this feater

@ian-whitestone ian-whitestone force-pushed the add-docker-support branch 3 times, most recently from 81128e4 to 1329900 Compare June 13, 2021 16:41
Copy link
Contributor Author

@ian-whitestone ian-whitestone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI @mcrowson I ran black zappa tests and it reformatted a bunch of test files...so this PR will include some changes un-related to the docker functionality

@mcrowson
Copy link
Contributor

mcrowson commented Jun 13, 2021

Had a conversation on Slack with Ian, putting summary here for posterity.

First we're working through an issue with Travis CI where they are not running our tests, so hopefully that will be addressed soon and we can see all of the tests passing.

The second is I have a hard time mixing arguments via the CLI and arguments via our config file. I made #988 to capture an enhancement to allow for any of them. I asked Ian to implement this feature configuration through the settings file instead of a CLI argument. CI/CD systems could use something like sed to update the config file if the tag changes and is not statically set.

Edit: After more analysis/discussion we think that this dockerfile passing is similar to the zip argument. Therefore this approach is fine for now. I'm good to approve once this passes CI.

@jneves any other thoughts on this one?

@mcrowson mcrowson closed this Jun 13, 2021
@mcrowson mcrowson reopened this Jun 13, 2021
@mcrowson
Copy link
Contributor

closed/reopened to try and trigger Travis. Nothing to see here folks.

@onimitch
Copy link

FWIW I've been using Ian's fork for a good few weeks now and not ran into any issues so far deploying using Docker. Anecdotal I know, but just thought I'd put it out there :D Thanks again for adding this to Zappa @ian-whitestone

@mcrowson mcrowson merged commit 1cdbd7f into zappa:master Jun 19, 2021
@ArtikUA
Copy link

ArtikUA commented Jun 21, 2021

It works like a charm even with public.ecr.aws/lambda/python:3.8
Here is my Dockerfile, it's even simpler and requirements.txt will be cached separately so no need to rebuild them every time when only code is changed:

FROM public.ecr.aws/lambda/python:3.8

COPY ./requirements.txt ./requirements.txt
RUN pip3 install -r requirements.txt

ADD ./ ./

RUN ZAPPA_HANDLER_PATH=$( \
    python -c "from zappa import handler; print (handler.__file__)" \
    ) \
    && echo $ZAPPA_HANDLER_PATH \
    && cp $ZAPPA_HANDLER_PATH ./

CMD [ "handler.lambda_handler" ]

@piotrdrazikowski
Copy link
Contributor

In case someone wants to migrate their zip-based project to a docker image, remember to remove "slim_handler": true from the zappa_settings.json file, otherwise zappa will still download old code from s3 bucket. Just learned it the hard way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants