feat(lambda-python): add support for custom build image

[setu4993]

Adding support for custom build images for packaging Python dependencies. This resolves #10298 since Code Artifact login step with AWS CLI or credentials can be passed in with Docker build args, handled within the custom Docker build image.

This also makes #15306 redundant, so I closed it.

Fixes #10298, #12949 and #16234.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gitpod-io]

[setu4993]

@nija-at : This PR is ready for review.

[setu4993]

The CI test is failing on a test related to appsync for an API key error, which is weird since that's not something that this PR is touching. Not sure what's going on there...

[nija-at]

Thanks so much for submitting this pull request. I am marking this pull request as p2, which means that we are unable to work on it immediately, but it's definitely still on our radar.

We use +1s to help prioritize our work, and are happy to revaluate this pull request based on community feedback.

[setu4993]

Thanks, @nija-at. A few questions:

1. What do you think is the approximate timeframe for getting this in?
2. I posted on one of the issues a few weeks ago, but it has been mostly inactive. This is an important fix for us as we expand our use of aws-cdk, so if we can get this in sooner, that'd be very much appreciated.
3. This is already supported with the NodejsFunction and right now, the PythonFunction not supporting custom bundling keeps it at a lower par in comparison.

[setu4993]

@nija-at : Would appreciate any thoughts / comments about #15324 (comment) and this PR generally. Especially since without this PR, PythonFunction remains one step behind NodeJsFunction in supporting custom build images, which makes it less usable.

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
• Commit ID: 9eb06a2
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[SamStephens]

@setu4993 whilst I think this is great, I don't think it actually solves the Code Artifact issue #10298, because as discussed in #16234 (comment), Code Artifact credentials are emphemeral, they are only valid for 12 hours.

My understanding of Docker images is that they're relatively durable. I would expect that for a Function definition with dockerBuildImageOptions provided, a Docker image would be built based on the base SAM build image with the custom options applied, and then that Docker image would be used until either the base SAM build image changes, or the dockerBuildImageOptions change. Which doesn't work for something that is only valid for 12 hours.

[setu4993]

@SamStephens : Thanks for your feedback. The reason I think this solves #10298 is that Code Artifact credentials can be passed into the Docker image at the build step. That allows installing private Python packages from Code Artifact, and then packaging the function to be executed. The credentials expiring later doesn't matter since the packages are already cached into the function once it has been built.

So, the idea would be that a custom Dockerfile.build contains steps / args required to either fetch Code Artifact credentials or parse them from build arguments (likely preferable).

We have had success doing something similar for Fargate-based Docker images.

[setu4993]

@nija-at : Any additional feedback on timeline here? This also closes #16234 and #12949, so there's atleast 3 issues open that could be resolved and use the changes in here.

[corymhall]

@setu4993 I think that I would prefer to adopt the same approach that is used by both NodejsFunction and GoFunction and just allow the user to supply their own DockerImage. i.e.

new lambda.PythonFunction(this, 'my-handler', {
  bundling: {
    dockerImage: DockerImage.fromBuild('/path/to/Dockerfile'),
  },
});

[setu4993]

Thanks for the feedback, @corymhall! I should have some space to work on this and update it over the weekend.

[setu4993]

@corymhall : Thinking through this a bit more, I'm open to switching to that structure, but I think it'd require quite a bit of refactoring of the current dependency installation system since this isn't inheriting from cdk.BundlingOptions right now... I'm open to doing that (possibly in a separate PR first) and then inheriting those changes to support bundling here. Thoughts?

[corymhall]

@setu4993 If you are willing to work on that refactoring then I think it is a good idea! Before making this library stable we would want to bring it in line with the other two anyway.

[setu4993]

@corymhall : What do you think about #18082? The allure of getting lambda-Python to stable (finally!) was enough motivation :).

[setu4993]

Closing in favor of #18082.