Using VCS dependencies throws KeyError when writing Pipfile.lock #3404
Comments
|
I'm also seeing this issue and trying to get more info to help with debugging here. The Lines 692 to 693 in 3b3ebb4 The condition before even reads ... or k not in lockfile[lockfile_section], i.e. it seems that it's somewhat expected that k, which is the name of a dependency here, might not be in lockfile[lockfile_section] yet, but then the next line tries to read it anyway? Even though we just checked that it's not in there? 🤔
If I'm interpreting this code correctly, |
|
I think you're right @teeberg. Here's the commit where that was intro'd: 17a9c3f#diff-4b9d1c4c6c693a89e0cd2e6d9ca27631R678 It seems like that predicate is wrong. I edited my locally installed |
|
Per the commit comment linked above: - if k in getattr(project, vcs_section, {}) or k not in lockfile[lockfile_section]:
+ if k in getattr(project, vcs_section, {}) and k in lockfile[lockfile_section]:Does resolve the problem with |
|
That fixed it for me too! :) |
|
In the case of your change, the second predicate after the for k, v in vcs_lockfile.items():
if k in getattr(project, vcs_section, {}):
if k in lockfile[lockfile_section] lockfile[lockfile_section][k].update(v)
else lockfile[lockfile_section][k] = vSomeone might be able to code-golf or Pythonify that for me. It would also be helpful to get @techalchemy's take on the above... |
|
I wrote @techalchemy via the email address that he lists on GitHub. Hopefully he can weigh in. |
al-the-x
changed the title
techalchemy
added
Type: Bug 🐛
|
Thanks for checking in on my line note @techalchemy. Happy to submit a PR for those changes, but they seem pretty small if your prefer to tackle them. |
|
Most likely I’ll catch them in my current net of changes. I have two rounds of fixes to get working before I start crossing out the small wins and open PRs. I took a long break so my dream of handling it in a weekend wasn’t to be |
|
No worries. Best of luck. |
|
I have the same error, the only difference is that I simply install a private package with I patched utils.py as was suggested in 17a9c3f#r32012178 |
|
Ping @techalchemy? |
|
All changes have been on hold pending #3298 which will be merged today assuming tests are passing The release of pip 19.0 and implementation of PEP 518 and 517 have caused significant issues and required substantial rework of pipenv's resolver, specifically in https://github.com/sarugaku/requirementslib which is now functionally acting as a dependency resolver for any VCS or editable dependencies. As a result of those changes, I've introduced additional logic to feed the dependency graph of any VCS/File/local requirements back into the resolver before we begin the resolution process so that we can actually perform an accurate resolution. What this should mean for this specific issue is that it will be resolved by the merge of #3298 because it completely eliminates this section of the code. |
|
Hey fellows and @techalchemy, thank you for solving this issue! Do we have a temporary solution how to avoid it meanwhile? Dependencies in several of our systems are blocked because of it. Thanks, |
|
@amureki Adding the dependency on which the |
2020.5.28 (2020-05-28) ====================== Features & Improvements ----------------------- - `pipenv install` and `pipenv sync` will no longer attempt to install satisfied dependencies during installation. pypa#3057, pypa#3506 - Added support for resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148 - Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc. - Improved error handling and formatting. - Introduced improved cross platform stream wrappers for better `stdout` and `stderr` consistency. pypa#3298 - For consistency with other commands and the `--dev` option description, `pipenv lock --requirements --dev` now emits both default and development dependencies. The new `--dev-only` option requests the previous behaviour (e.g. to generate a `dev-requirements.txt` file). pypa#3316 - Pipenv will now successfully recursively lock VCS sub-dependencies. pypa#3328 - Added support for `--verbose` output to `pipenv run`. pypa#3348 - Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts. pypa#3368 - Added a new environment variable, `PIPENV_RESOLVE_VCS`, to toggle dependency resolution off for non-editable VCS, file, and URL based dependencies. pypa#3577 - Added the ability for Windows users to enable emojis by setting `PIPENV_HIDE_EMOJIS=0`. pypa#3595 - Allow overriding `PIPENV_INSTALL_TIMEOUT` environment variable (in seconds). pypa#3652 - Allow overriding `PIP_EXISTS_ACTION` evironment variable (value is passed to pip install). Possible values here: <https://pip.pypa.io/en/stable/reference/pip/#exists-action-option> Useful when you need to `PIP\_EXISTS\_ACTION=i` (ignore existing packages) - great for CI environments, where you need really fast setup. pypa#3738 - Pipenv will no longer forcibly override `PIP_NO_DEPS` on all vcs and file dependencies as resolution happens on these in a pre-lock step. pypa#3763 - Improved verbose logging output during `pipenv lock` will now stream output to the console while maintaining a spinner. pypa#3810 - Added support for automatic python installs via `asdf` and associated `PIPENV_DONT_USE_ASDF` environment variable. pypa#4018 - Pyenv/asdf can now be used whether or not they are available on PATH. Setting `PYENV_ROOT`/`ASDF_DIR` in a `.env` file allows Pipenv to install an interpreter without any shell customizations, so long as pyenv/asdf is installed. pypa#4245 - Added `--key` command line parameter for including personal PyUp.io API tokens when running `pipenv check`. pypa#4257 Behavior Changes ---------------- - Make conservative checks of known exceptions when subprocess returns output, so user won\'t see the whole traceback - just the error. pypa#2553 - Do not touch Pipfile early and rely on it so that one can do `pipenv sync` without a Pipfile. pypa#3386 - Re-enable `--help` option for `pipenv run` command. pypa#3844 - Make sure `pipenv lock -r --pypi-mirror {MIRROR_URL}` will respect the pypi-mirror in requirements output. pypa#4199 Bug Fixes --------- - Raise `PipenvUsageError` when \[\[source\]\] does not contain url field. pypa#2373 - Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message. pypa#2722 - Fixed an issue which caused errors due to reliance on the system utilities `which` and `where` which may not always exist on some systems. - Fixed a bug which caused periodic failures in python discovery when executables named `python` were not present on the target `$PATH`. pypa#2783 - Dependency resolution now writes hashes for local and remote files to the lockfile. pypa#3053 - Fixed a bug which prevented `pipenv graph` from correctly showing all dependencies when running from within `pipenv shell`. pypa#3071 - Fixed resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148 - Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances. - Fixed a bug with package discovery when running `pipenv clean`. pypa#3298 - Quote command arguments with carets (`^`) on Windows to work around unintended shell escapes. pypa#3307 - Handle alternate names for UTF-8 encoding. pypa#3313 - Abort pipenv before adding the non-exist package to Pipfile. pypa#3318 - Don\'t normalize the package name user passes in. pypa#3324 - Fix a bug where custom virtualenv can not be activated with pipenv shell pypa#3339 - Fix a bug that `--site-packages` flag is not recognized. pypa#3351 - Fix a bug where `pipenv --clear` is not working pypa#3353 - Fix unhashable type error during `$ pipenv install --selective-upgrade` pypa#3384 - Dependencies with direct `PEP508` compliant VCS URLs specified in their `install_requires` will now be successfully locked during the resolution process. pypa#3396 - Fixed a keyerror which could occur when locking VCS dependencies in some cases. pypa#3404 - Fixed a bug that `ValidationError` is thrown when some fields are missing in source section. pypa#3427 - Updated the index names in lock file when source name in Pipfile is changed. pypa#3449 - Fixed an issue which caused `pipenv install --help` to show duplicate entries for `--pre`. pypa#3479 - Fix bug causing `[SSL: CERTIFICATE_VERIFY_FAILED]` when Pipfile `[[source]]` has `verify_ssl=false` and url with custom port. pypa#3502 - Fix `sync --sequential` ignoring `pip install` errors and logs. pypa#3537 - Fix the issue that lock file can\'t be created when `PIPENV_PIPFILE` is not under working directory. pypa#3584 - Pipenv will no longer inadvertently set `editable=True` on all vcs dependencies. pypa#3647 - The `--keep-outdated` argument to `pipenv install` and `pipenv lock` will now drop specifier constraints when encountering editable dependencies. - In addition, `--keep-outdated` will retain specifiers that would otherwise be dropped from any entries that have not been updated. pypa#3656 - Fixed a bug which sometimes caused pipenv to fail to respect the `--site-packages` flag when passed with `pipenv install`. pypa#3718 - Normalize the package names to lowercase when comparing used and in-Pipfile packages. pypa#3745 - `pipenv update --outdated` will now correctly handle comparisons between pre/post-releases and normal releases. pypa#3766 - Fixed a `KeyError` which could occur when pinning outdated VCS dependencies via `pipenv lock --keep-outdated`. pypa#3768 - Resolved an issue which caused resolution to fail when encountering poorly formatted `python_version` markers in `setup.py` and `setup.cfg` files. pypa#3786 - Fix a bug that installation errors are displayed as a list. pypa#3794 - Update `pythonfinder` to fix a problem that `python.exe` will be mistakenly chosen for virtualenv creation under WSL. pypa#3807 - Fixed several bugs which could prevent editable VCS dependencies from being installed into target environments, even when reporting successful installation. pypa#3809 - `pipenv check --system` should find the correct Python interpreter when `python` does not exist on the system. pypa#3819 - Resolve the symlinks when the path is absolute. pypa#3842 - Pass `--pre` and `--clear` options to `pipenv update --outdated`. pypa#3879 - Fixed a bug which prevented resolution of direct URL dependencies which have PEP508 style direct url VCS sub-dependencies with subdirectories. pypa#3976 - Honor `PIPENV_SPINNER` environment variable pypa#4045 - Fixed an issue with `pipenv check` failing due to an invalid API key from `pyup.io`. pypa#4188 - Fixed a bug which caused versions from VCS dependencies to be included in `Pipfile.lock` inadvertently. pypa#4217 - Fixed a bug which caused pipenv to search non-existent virtual environments for `pip` when installing using `--system`. pypa#4220 - `Requires-Python` values specifying constraint versions of python starting from `1.x` will now be parsed successfully. pypa#4226 - Fix a bug of `pipenv update --outdated` that can\'t print output correctly. pypa#4229 - Fixed a bug which caused pipenv to prefer source distributions over wheels from `PyPI` during the dependency resolution phase. Fixed an issue which prevented proper build isolation using `pep517` based builders during dependency resolution. pypa#4231 - Don\'t fallback to system Python when no matching Python version is found. pypa#4232 Vendored Libraries ------------------ - Updated `pip_shims` to support `--outdated` with new pip versions. pypa#3766 - Update vendored dependencies and invocations - Update vendored and patched dependencies - Update patches on `piptools`, `pip`, `pip-shims`, `tomlkit` - Fix invocations of dependencies - Fix custom `InstallCommand` instantiation - Update `PackageFinder` usage - Fix `Bool` stringify attempts from `tomlkit` - Updated vendored dependencies: - **attrs**: `18.2.0 => `19.1.0` - **certifi**: `2018.10.15 => `2019.3.9` - **cached\_property**: `1.4.3 => `1.5.1` - **cerberus**: `1.2.0 => `1.3.1` - **click**: `7.0.0 => `7.1.1` - **click-completion**: `0.5.0 => `0.5.1` - **colorama**: `0.3.9 => `0.4.3` - **contextlib2**: `(new) => `0.6.0.post1` - **distlib**: `0.2.8 => `0.2.9` - **funcsigs**: `(new) => `1.0.2` - **importlib\_metadata** `1.3.0 => `1.5.1` - **importlib-resources**: `(new) => `1.4.0` - **idna**: `2.7 => `2.9` - **jinja2**: `2.10.0 => `2.11.1` - **markupsafe**: `1.0 => `1.1.1` - **more-itertools**: `(new) => `5.0.0` - **orderedmultidict**: `(new) => `1.0` - **packaging**: `18.0 => `19.0` - **parse**: `1.9.0 => `1.15.0` - **pathlib2**: `2.3.2 => `2.3.3` - **pep517**: `(new) => `0.5.0` - **pexpect**: `4.6.0 => `4.8.0` - **pip-shims**: `0.2.0 => `0.5.1` - **pipdeptree**: `0.13.0 => `0.13.2` - **pyparsing**: `2.2.2 => `2.4.6` - **python-dotenv**: `0.9.1 => `0.10.2` - **pythonfinder**: `1.1.10 => `1.2.2` - **pytoml**: `(new) => `0.1.20` - **requests**: `2.20.1 => `2.23.0` - **requirementslib**: `1.3.3 => `1.5.4` - **scandir**: `1.9.0 => `1.10.0` - **shellingham**: `1.2.7 => `1.3.2` - **six**: `1.11.0 => `1.14.0` - **tomlkit**: `0.5.2 => `0.5.11` - **urllib3**: `1.24 => `1.25.8` - **vistir**: `0.3.0 => `0.5.0` - **yaspin**: `0.14.0 => `0.14.3` - **zipp**: `0.6.0` - Removed vendored dependency **cursor**. pypa#4169 - Add and update vendored dependencies to accommodate `safety` vendoring: - **safety** `(none)` => `1.8.7` - **dparse** `(none)` => `0.5.0` - **pyyaml** `(none)` => `5.3.1` - **urllib3** `1.25.8` => `1.25.9` - **certifi** `2019.11.28` => `2020.4.5.1` - **pyparsing** `2.4.6` => `2.4.7` - **resolvelib** `0.2.2` => `0.3.0` - **importlib-metadata** `1.5.1` => `1.6.0` - **pip-shims** `0.5.1` => `0.5.2` - **requirementslib** `1.5.5` => `1.5.6` pypa#4188 - Updated vendored `pip` => `20.0.2` and `pip-tools` => `5.0.0`. pypa#4215 - Updated vendored dependencies to latest versions for security and bug fixes: - **requirementslib** `1.5.8` => `1.5.9` - **vistir** `0.5.0` => `0.5.1` - **jinja2** `2.11.1` => `2.11.2` - **click** `7.1.1` => `7.1.2` - **dateutil** `(none)` => `2.8.1` - **backports.functools\_lru\_cache** `1.5.0` => `1.6.1` - **enum34** `1.1.6` => `1.1.10` - **toml** `0.10.0` => `0.10.1` - **importlib\_resources** `1.4.0` => `1.5.0` pypa#4226 - Changed attrs import path in vendored dependencies to always import from `pipenv.vendor`. pypa#4267 Improved Documentation ---------------------- - Added documenation about variable expansion in `Pipfile` entries. pypa#2317 - Consolidate all contributing docs in the rst file pypa#3120 - Update the out-dated manual page. pypa#3246 - Move CLI docs to its own page. pypa#3346 - Replace (non-existant) video on docs index.rst with equivalent gif. pypa#3499 - Clarify wording in Basic Usage example on using double quotes to escape shell redirection pypa#3522 - Ensure docs show navigation on small-screen devices pypa#3527 - Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written. pypa#3629 - Updated the documentation with the new `pytest` entrypoint. pypa#3759 - Fix link to GIF in README.md demonstrating Pipenv\'s usage, and add descriptive alt text. pypa#3911 - Added a line describing potential issues in fancy extension. pypa#3912 - Documental description of how Pipfile works and association with Pipenv. pypa#3913 - Clarify the proper value of `python_version` and `python_full_version`. pypa#3914 - Write description for `--deploy` extension and few extensions differences. pypa#3915 - More documentation for `.env` files pypa#4100 - Updated documentation to point to working links. pypa#4137 - Replace docs.pipenv.org with pipenv.pypa.io pypa#4167 - Added functionality to check spelling in documentation and cleaned up existing typographical issues. pypa#4209
When converting an existing project from
requirements.txtto usepipenv, I performed apipenv install, which created the followingPipfile:Then
pipenvattempts to generate thePipenv.lockfile and generates the following error:I have run
pipenv lock --clearmultiple times and have gotten differentKeyErrorfailures forchardetas well, but mostly foridna. The stack trace is the same, regardless of the key that fails the lookup. When I usedpipenv install --skip-lock, all the dependencies installed except for one that needed native headers fromlibmemcached. The error message thatpipreported from that failure was very different and Google-able, so I was able to resolve and get it installed. But I still have noPipfile.lockandpipenv lock --clearany number of times just generates the same error as above. Help?$ pipenv --support
Pipenv version:
'2018.11.26'Pipenv location:
'/usr/local/Cellar/pipenv/2018.11.26/libexec/lib/python3.7/site-packages/pipenv'Python location:
'/usr/local/Cellar/pipenv/2018.11.26/libexec/bin/python3.7'Python installations found:
3.7.1:/usr/local/bin/python33.7.1:/usr/local/bin/python3.7m2.7.15:/Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov/bin/python2.7.15:/usr/local/bin/python2.7.15:/usr/local/bin/pythonw2.7.10:/usr/bin/python2.7.10:/usr/bin/pythonw2.7.10:/usr/bin/python2.7PEP 508 Information:
System environment variables:
PATHGREP_COLORLESS_TERMCAP_mbTERM_PROGRAMLESS_TERMCAP_mdNVM_CD_FLAGSLESS_TERMCAP_mePIP_PYTHON_PATHSHELLTERMTMPDIRApple_PubSub_Socket_RenderTERM_PROGRAM_VERSIONDIRENV_DIRTERM_SESSION_IDLESS_TERMCAP_ueNVM_DIRUSERLS_COLORSGREP_COLORSRPROMPTSSH_AUTH_SOCK__CF_USER_TEXT_ENCODINGVIRTUAL_ENVVIRTUAL_ENV_DISABLE_PROMPTPAGERWORKON_HOMETMUXDIRENV_WATCHESLESS_TERMCAP_usVIRTUALENVWRAPPER_PYTHONPIPENV_ACTIVE_git_log_oneline_format_PWDEDITORLANGTMUX_PANEXPC_FLAGSPYTHONDONTWRITEBYTECODEXPC_SERVICE_NAMEHOMESHLVLLESSLOGNAMEPIP_DISABLE_PIP_VERSION_CHECKVISUALLESS_TERMCAP_soTMUX_PLUGIN_MANAGER_PATHNVM_BINBROWSER_git_log_medium_formatDIRENV_DIFFSECURITYSESSIONIDLESS_TERMCAP_sePIP_SHIMS_BASE_MODULEPYTHONFINDER_IGNORE_UNSUPPORTEDPipenv–specific environment variables:
PIPENV_ACTIVE:1Debug–specific environment variables:
PATH:/usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/usr/local/opt/nvm/versions/node/v10.4.1/bin:/Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTov/bin:/Users/al_the_x/bin:/usr/local/Cellar/pipenv/2018.11.26/libexec/tools:/Users/al_the_x/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbinSHELL:/usr/local/bin/zshEDITOR:/usr/local/bin/nvimLANG:en_US.UTF-8PWD:/Users/al_the_x/Projects/grove.coVIRTUAL_ENV:/Users/al_the_x/.local/share/virtualenvs/grove.co-dcWwlTovContents of
Pipfile('/Users/al_the_x/Projects/grove.co/Pipfile'):The text was updated successfully, but these errors were encountered: