Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(aws_lambda_python): Poetry exported requirements not readable by pip #14201

Closed
l0b0 opened this issue Apr 16, 2021 · 8 comments
Closed

(aws_lambda_python): Poetry exported requirements not readable by pip #14201

l0b0 opened this issue Apr 16, 2021 · 8 comments
Labels

Comments

@l0b0
Copy link
Contributor

l0b0 commented Apr 16, 2021

Pip refuses to install the requirements generated by poetry export.

Reproduction Steps

  1. Create a simple pyproject.toml and poetry.lock in a directory with existing Lambda job code (poetry init && poetry add jsonschema="*" pynamodb="*")
  2. Instantiate a PythonFunction for the directory above
  3. cdk synth

What did you expect to happen?

The build should install the packages mentioned in poetry.lock.

What actually happened?

Step 1/9 : ARG IMAGE=amazon/aws-sam-cli-build-image-python3.7
Step 2/9 : FROM $IMAGE
 ---> 8aae4ac71397
Step 3/9 : RUN yum -q list installed rsync &>/dev/null || yum install -y rsync
 ---> Using cache
 ---> 97cffc20d856
Step 4/9 : RUN pip install --upgrade pip
 ---> Running in 71bfdf9b78cb
Collecting pip
  Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-21.0.1
Removing intermediate container 71bfdf9b78cb
 ---> 4509190dda1a
Step 5/9 : RUN pip install pipenv poetry
 ---> Running in fd0a1868ac36
Collecting pipenv
  Downloading pipenv-2020.11.15-py2.py3-none-any.whl (3.9 MB)
Collecting poetry
  Downloading poetry-1.1.6-py2.py3-none-any.whl (172 kB)
Requirement already satisfied: setuptools>=36.2.1 in /var/lang/lib/python3.8/site-packages (from pipenv) (49.2.1)
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.4-py2.py3-none-any.whl (6.6 kB)
Collecting virtualenv
  Downloading virtualenv-20.4.3-py2.py3-none-any.whl (7.2 MB)
Requirement already satisfied: pip>=18.0 in /var/lang/lib/python3.8/site-packages (from pipenv) (21.0.1)
Collecting certifi
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting html5lib<2.0,>=1.0
  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting packaging<21.0,>=20.4
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting shellingham<2.0,>=1.1
  Downloading shellingham-1.4.0-py2.py3-none-any.whl (9.4 kB)
Collecting cleo<0.9.0,>=0.8.1
  Downloading cleo-0.8.1-py2.py3-none-any.whl (21 kB)
Collecting crashtest<0.4.0,>=0.3.0
  Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
Collecting tomlkit<1.0.0,>=0.7.0
  Downloading tomlkit-0.7.0-py2.py3-none-any.whl (32 kB)
Collecting poetry-core<1.1.0,>=1.0.3
  Downloading poetry_core-1.0.3-py2.py3-none-any.whl (424 kB)
Collecting keyring<22.0.0,>=21.2.0
  Downloading keyring-21.8.0-py3-none-any.whl (32 kB)
Collecting pkginfo<2.0,>=1.4
  Downloading pkginfo-1.7.0-py2.py3-none-any.whl (25 kB)
Collecting clikit<0.7.0,>=0.6.2
  Downloading clikit-0.6.2-py2.py3-none-any.whl (91 kB)
Collecting requests<3.0,>=2.18
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting requests-toolbelt<0.10.0,>=0.9.1
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting pexpect<5.0.0,>=4.7.0
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting cachy<0.4.0,>=0.3.0
  Downloading cachy-0.3.0-py2.py3-none-any.whl (20 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.4
  Downloading CacheControl-0.12.6-py2.py3-none-any.whl (19 kB)
Collecting msgpack>=0.5.2
  Downloading msgpack-1.0.2-cp38-cp38-manylinux1_x86_64.whl (302 kB)
Collecting lockfile>=0.9
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pylev<2.0,>=1.3
  Downloading pylev-1.3.0-py2.py3-none-any.whl (4.9 kB)
Collecting pastel<0.3.0,>=0.2.0
  Downloading pastel-0.2.1-py2.py3-none-any.whl (6.0 kB)
Collecting six>=1.9
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting cryptography>=2.0
  Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting filelock<4,>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Installing collected packages: pycparser, urllib3, idna, chardet, cffi, certifi, requests, pylev, pastel, msgpack, jeepney, cryptography, crashtest, webencodings, six, SecretStorage, pyparsing, ptyprocess, lockfile, filelock, distlib, clikit, cachecontrol, appdirs, virtualenv-clone, virtualenv, tomlkit, shellingham, requests-toolbelt, poetry-core, pkginfo, pexpect, packaging, keyring, html5lib, cleo, cachy, poetry, pipenv
Successfully installed SecretStorage-3.3.1 appdirs-1.4.4 cachecontrol-0.12.6 cachy-0.3.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 cleo-0.8.1 clikit-0.6.2 crashtest-0.3.1 cryptography-3.4.7 distlib-0.3.1 filelock-3.0.12 html5lib-1.1 idna-2.10 jeepney-0.6.0 keyring-21.8.0 lockfile-0.12.2 msgpack-1.0.2 packaging-20.9 pastel-0.2.1 pexpect-4.8.0 pipenv-2020.11.15 pkginfo-1.7.0 poetry-1.1.6 poetry-core-1.0.3 ptyprocess-0.7.0 pycparser-2.20 pylev-1.3.0 pyparsing-2.4.7 requests-2.25.1 requests-toolbelt-0.9.1 shellingham-1.4.0 six-1.15.0 tomlkit-0.7.0 urllib3-1.26.4 virtualenv-20.4.3 virtualenv-clone-0.5.4 webencodings-0.5.1
Removing intermediate container fd0a1868ac36
 ---> 19e59c530325
Step 6/9 : WORKDIR /var/dependencies
 ---> Running in dbe9f58c3bde
Removing intermediate container dbe9f58c3bde
 ---> 595da7e52ae0
Step 7/9 : COPY Pipfile* pyproject* poetry* requirements.tx[t] ./
 ---> 6023d400a134
Step 8/9 : RUN [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt;     [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt;     [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;
 ---> Running in 1f799baa02e9
Collecting attrs==20.3.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting botocore==1.20.53
  Downloading botocore-1.20.53-py2.py3-none-any.whl (7.4 MB)
Collecting jmespath==0.10.0
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting jsonschema==3.2.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting pynamodb==5.0.3
  Downloading pynamodb-5.0.3-py3-none-any.whl (56 kB)
Collecting pyrsistent==0.17.3
  Downloading pyrsistent-0.17.3.tar.gz (106 kB)
Collecting python-dateutil==2.8.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.26.4
  Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting setuptools
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://files.pythonhosted.org/packages/ae/42/2876a3a136f8bfa9bd703518441c8db78ff1eeaddf174baa85c083c1fd15/setuptools-56.0.0-py3-none-any.whl#sha256=7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5 (from jsonschema==3.2.0->-r requirements.txt (line 10))
The command '/bin/sh -c [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt;     [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt;     [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;' returned a non-zero code: 1
jsii.errors.JavaScriptError: 
  Error: docker exited with status 1
      at dockerExec (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/core/lib/bundling.js:207:15)
      at Function.fromAsset (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/core/lib/bundling.js:66:9)
      at Object.bundle (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/aws-lambda-python/lib/bundling.js:36:43)
      at new PythonFunction (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/aws-lambda-python/lib/function.js:39:30)
      at /tmp/tmpq1on2qlt/lib/program.js:8154:58
      at Kernel._wrapSandboxCode (/tmp/tmpq1on2qlt/lib/program.js:8582:24)
      at Kernel._create (/tmp/tmpq1on2qlt/lib/program.js:8154:34)
      at Kernel.create (/tmp/tmpq1on2qlt/lib/program.js:7895:29)
      at KernelHost.processRequest (/tmp/tmpq1on2qlt/lib/program.js:9479:36)
      at KernelHost.run (/tmp/tmpq1on2qlt/lib/program.js:9442:22)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 75, in <module>
    main()
  File "app.py", line 51, in main
    APIStack(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/victor/dev/geospatial-data-lake/infrastructure/api_stack.py", line 58, in __init__
    datasets_endpoint_lambda = LambdaEndpoint(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/victor/dev/geospatial-data-lake/infrastructure/constructs/lambda_endpoint.py", line 17, in __init__
    super().__init__(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/aws_cdk/aws_lambda_python/__init__.py", line 252, in __init__
    jsii.create(PythonFunction, self, [scope, id, props])
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 275, in create
    response = self.provider.create(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 344, in create
    return self._process.send(request, CreateResponse)
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 326, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: docker exited with status 1

Environment

  • CDK CLI Version : 1.98.0 (build 79f4512)
  • Framework Version: aws-cli/1.19.52 Python/3.8.6 Linux/5.11.13-arch1-1 botocore/1.20.53
  • Node.js Version: v14.15.1
  • OS : Arch Linux
  • Language (Version): Python (3.8.6)

Other

Looks like it's this Poetry issue, which means one of these workarounds have to be used.


This is 🐛 Bug Report

@l0b0 l0b0 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Apr 16, 2021
@eladb
Copy link
Contributor

eladb commented May 2, 2021

@adamelmore can you please take a look?

@eladb
Copy link
Contributor

eladb commented May 25, 2021

@l0b0 it seems like this is the culprit:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://files.pythonhosted.org/packages/ae/42/2876a3a136f8bfa9bd703518441c8db78ff1eeaddf174baa85c083c1fd15/setuptools-56.0.0-py3-none-any.whl#sha256=7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5 (from jsonschema==3.2.0->-r requirements.txt (line 10))
The command '/bin/sh -c [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt;     [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt;     [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;' returned a non-zero code: 1

It's a best practice to pin versions in order to avoid unexpected updates.

@l0b0
Copy link
Contributor Author

l0b0 commented May 25, 2021

setuptools isn't included in the export from Poetry, because it's excluded from poetry.lock and the Poetry section of pyproject.toml. It looks like pip installs it anyway. So it looks like the way to fix this would be for CDK to either add some version of setuptools with hashes to the requirements.txt (risky) or to force pip to not install setuptools (--no-deps?).

@davidkelley
Copy link

Hi, any update on this issue? We've just encountered this too

@eladb
Copy link
Contributor

eladb commented Jun 30, 2021

@l0b0 would you be interested to contribute a fix?

@eladb eladb added p1 and removed needs-triage This issue or PR still needs to be triaged. labels Jun 30, 2021
@eladb eladb removed their assignment Jun 30, 2021
@abend-arg
Copy link

abend-arg commented Jun 19, 2022

Just to give some light into the matter, this problem is not technically an issue with the CDK but with pip. The issue was reported, solved and reverted, so we are still experiencing this issue: pypa/pip#9243 and pypa/pip#9644

My workaround was to remove the poetry.lock file and manually export to a requirements.txt file using poetry export -o requirements.txt --without-hashes. Take into account that it includes the line --index-url in the file and in my case it was completely omitting the PIP_INDEX_URL environment variable turning into a new problem since I was using a private repo. Removing that --index-url line solved the issue.

The only thing that could be done while pip fixes its problem, is to have the possibility of telling to the CDK that during the export we want to include the flag --without-hashes.

That could be a workaround to solve this bug as well: #19232

mergify bot pushed a commit that referenced this issue Oct 7, 2022
…2351)

Export poetry dependencies without hashes to prevent bundling failures when a dependency provides a hash. Without this flag, users relying on the Poetry python dependency manager need to manually export their own `requirements.txt` file, as described in #14201

Fixes #19232

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
arewa pushed a commit to arewa/aws-cdk that referenced this issue Oct 8, 2022
…s#22351)

Export poetry dependencies without hashes to prevent bundling failures when a dependency provides a hash. Without this flag, users relying on the Poetry python dependency manager need to manually export their own `requirements.txt` file, as described in aws#14201

Fixes aws#19232

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
homakk pushed a commit to homakk/aws-cdk that referenced this issue Dec 1, 2022
…s#22351)

Export poetry dependencies without hashes to prevent bundling failures when a dependency provides a hash. Without this flag, users relying on the Poetry python dependency manager need to manually export their own `requirements.txt` file, as described in aws#14201

Fixes aws#19232

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@madeline-k
Copy link
Contributor

Fixed by #22351

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants