Tests failing to run with BackendUnavailable since pip 19 #1644

jaraco · 2019-01-25T21:27:34Z

It seems that since the release of pip 19, Setuptools' own tests fail to run thus:

setuptools fix_889_and_non-ascii_in_setup.cfg_take_2 $ tox -e py27                                                                                                                       
py27 create: /Users/jaraco/code/main/setuptools/.tox/py27
py27 installdeps: -rtests/requirements.txt
py27 develop-inst: /Users/jaraco/code/main/setuptools
ERROR: invocation failed (exit code 2), logfile: /Users/jaraco/code/main/setuptools/.tox/py27/log/py27-2.log
ERROR: actionid: py27
msg: developpkg
cmdargs: '/Users/jaraco/code/main/setuptools/.tox/py27/bin/pip install --exists-action w -e /Users/jaraco/code/main/setuptools'

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
BackendUnavailable

py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,apipkg==1.5,atomicwrites==1.2.1,attrs==18.2.0,configparser==3.7.1,contextlib2==0.5.5,coverage==4.5.2,enum34==1.1.6,execnet==1.5.0,flake8==3.6.0,funcsigs==1.0.2,futures==3.2.0,importlib-metadata==0.8,mccabe==0.6.1,mock==2.0.0,more-itertools==5.0.0,path.py==11.5.0,pathlib2==2.3.3,pbr==5.1.1,pluggy==0.8.1,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytest==3.10.1,pytest-cov==2.6.1,pytest-fixture-config==1.4.0,pytest-flake8==1.0.3,pytest-shutil==1.4.0,pytest-virtualenv==1.4.0,scandir==1.9.0,six==1.12.0,termcolor==1.1.0,virtualenv==16.3.0,zipp==0.3.3
________________________________________________________________________________________ summary ________________________________________________________________________________________
ERROR:   py27: InvocationError for command /Users/jaraco/code/main/setuptools/.tox/py27/bin/pip install --exists-action w -e /Users/jaraco/code/main/setuptools (see /Users/jaraco/code/main/setuptools/.tox/py27/log/py27-2.log) (exited with code 2)

I suspect this is related to #1642 if not just another manifestation of the same issue.

The text was updated successfully, but these errors were encountered:

jaraco · 2019-01-25T21:48:28Z

In order to unblock this project, I was going to configure tox to downgrade pip after creating the environment, but tox only allows one install command, and this project has already used up that configuration option to work around another issue, and it's currently not possible to specify a pip version.

jaraco · 2019-01-26T20:18:14Z

After getting a test failure from tox, I'm able to replicate the failure outside of tox by invoking the same command tox does (or similar):

setuptools master $ .tox/py27/bin/pip install -e .                                                                                                                                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
BackendUnavailable

The problem, of course, is that the virtualenv doesn't have setuptools installed. Setuptools historically has relied on itself to build/install itself... but pep517 is breaking from the convention/expectation that '' is on sys.path.

jaraco · 2019-01-26T20:21:41Z

the virtualenv doesn't have setuptools installed

I'm wrong about that. The virtualenv does have setuptools installed... it just doesn't show up with pip freeze (probably due to special handling).

jaraco · 2019-01-26T20:27:51Z

If I update pep517._in_process to inject '' to sys.path in main(), the command no longer fails. That might also explain why python -m pep517.build . succeeds also - because in that mode of invocation, '' will be on sys.path.

jaraco · 2019-01-26T20:53:40Z

This is particularly difficult to debug because of the subprocess invocation. If I put a breakpoint in the _in_process script to troubleshoot, it exits immediately because there's no pipes for stdin/stdout.

jaraco · 2019-01-26T21:14:13Z

With lots of edits in the pip codebase, I managed to get a PDB prompt inside the _in_process, and here's what I see:

setuptools master $ .tox/py27/bin/pip install -e .                                                                                                                                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies ... done
> /Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py(186)main()
-> if len(sys.argv) < 3:
(Pdb) import sys
(Pdb) sys.path
['/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/site', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python27.zip', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-darwin', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-mac', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-tk', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-old', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-dynload', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/overlay/lib/python2.7/site-packages', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages']
(Pdb) import os
(Pdb) os.getcwd()
'/Users/jaraco/code/main/setuptools'
(Pdb) import setuptools
*** ImportError: No module named setuptools

Why is setuptools not available? I suspect it has something to do with the 'overlay' and the fact that the site-packages of the virtualenv isn't in sys.path.

jaraco · 2019-01-26T21:20:15Z

~ $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages                                                                  
ls: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages: No such file or directory
~ $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/overlay/lib/python2.7/site-packages                                                                 
wheel                  wheel-0.32.3.dist-info

So it seems something is creating a clean environment of dependencies in which the build can happen, containing only the dependencies declared in build-system.requires. That makes sense. As I was tracing the code, I didn't see where that happens.

benoit-pierre · 2019-01-26T21:23:11Z

I've used pdb-clone for debugging such cases.

The build environment code is in pip._internal.build_env.

So setuptools is a particular case, since it provides its own backend, which mean that for supporting such a use-case, the package source directory must be added to sys path, and this must be done in pip.

benoit-pierre · 2019-01-26T21:24:56Z

I can only reproduce the issue with Python 3 when tox-venv is not installed, which is also weird.

jaraco · 2019-01-26T21:27:28Z

I can only reproduce the issue with Python 3 when tox-venv is not installed, which is also weird.

That's probably a pip bug in the build isolation code on a venv.

benoit-pierre · 2019-01-26T21:33:19Z

No, it's because the versions of pip being used are different...

.

jaraco · 2019-01-27T16:22:15Z

Following the acceptance of #1634, setuptools' own test suite now fails.

daa · 2019-01-28T08:34:17Z

As you have mentioned BackendUnavailable is raised because of ImportError: No module named setuptools and this happens because setuptools is not available in build environment. Speaking about details let's consider how pip builds wheel with pep517:

Prepare build environment
Install build requirements into that path
From prepared build environment invoke build backend hook to get requirements to build wheel - get_requires_for_build_wheel
Invoke build_wheel() hook from build environment

To find why setuptools is not available in the build environment inspite of it being in current working directory let's see how isolated build environment is prepared:

Create temporary directory with structure resembling python directory layout
Create sitecustomize.py (pip/_internal/build_env.py at line 79) which removes original paths from sys.path, adds system site directory and adds build environment site directory.

So step 2 is making setuptools from current directory unavailable for code run inside of build environment.

pganssle · 2019-01-28T21:51:30Z

Presumably even though @daa's solution works, it means that we can't actually bootstrap our PEP 517, right? We will always be relying on the previously built versions of setuptools.

That will probably work for now, but this seems like it must be a bigger issue, and I think it's one we can't fix in this project, right? The issue is that we supply our own PEP 517 backend, and there's no way to declare that you depend on a PEP 517 backend that exists only in your local directory.

I see two ways forward with this:

Fix this problem with pip, adjusting PEP 517 to allow this kind of situation if necessary.
Create a new "PEP 517 bootstrapper" backend, that simply delegates all backend tasks to a backend that exists in your local directory, then start using that.

Of course with option 2 the PEP 517 bootstrapper still needs some way to build itself, and all PEP 517 build systems have this problem, so if I understand the problem correctly, the most robust solution is to pursue option 1.

Is there already a ticket tracking this in pip?

ncoghlan · 2019-02-02T13:51:46Z

I think the discussionin Discourse is still going somewhere, but the direction it's now going is the direction you describe here: projects that need bootstrapping should add themselves to their own build-system.requires, and then the onus is on frontends to detect that and halt the infinite regress.

pganssle · 2019-02-04T11:12:14Z

Thinking about this, even if pip --no-binary :all: switches to resolving self-referential source distributions with wheels, if they don't handle long-range cycles, we're still going to break --no-binary :all: if we depend on wheel as part of the PEP 517 build.

That basically means that we cannot have any dependencies in build-system.requires, and we'll need to vendor wheel, so I think we have three options:

Specify 'setuptools' and 'wheel' in our build-system.requires and let pip fix --no-binary :all: or not, as desired.
Opt out of PEP 517 forever, meaning that we will never be installable by a PEP 517-only frontend except as a wheel.
Vendor wheel in our source tree and wait for PEP 517 to be extended to allow for self-bootstrapping builds.

I don't think number 3 is going to happen, so we're really choosing between options 1 and 2. I'm in favor of option 1, but we may want to give pip some fair warning that we'll be doing it, since it will likely cause at least some breakages.

ncoghlan · 2019-02-04T11:57:25Z

Reading pypa/pip#6222 (and based on the other threads), I think the pip folks already agree that the current interaction between --no-binary :all: and PEP 517 needs work, so Option 1 seems like a reasonable choice at the setuptools level to me.

Make it possible to use a more recent version of pip for tests.

improve workaround for #1644

Work around buggy pip detection code for "pip.exe install/update ... pip ...". commit 0d831c9 Author: Benoit Pierre <benoit.pierre@gmail.com> Date: Sat Aug 10 03:57:58 2019 +0200 improve

Work around buggy pip detection code for "pip.exe install/update pip ...".

tweak workaround for #1644

pganssle · 2019-11-16T21:52:32Z

So after the absurd amount of discussion on this, the final decision (around 9 months ago) was to modify PEP 517 to add backend-path for bootstrapping, but no front-ends are actually spec compliant as far as I can tell. pip doesn't seem to support it at all.

This makes it very difficult to install our own library from git. I suggest that we just go ahead with this configuration for now:

[build-system]
requires = ["wheel", "setuptools>=40.0"]
build-backend = "setuptools.build_meta"

We can continue to exclude pyproject.toml from the source distribution so that pip install setuptools --no-binary :all: will continue to work for now. Once pip implements this we can probably switch over to using backend-path.

webknjaz · 2021-02-04T14:54:44Z

So after the absurd amount of discussion on this, the final decision (around 9 months ago) was to modify PEP 517 to add backend-path for bootstrapping, but no front-ends are actually spec compliant as far as I can tell. pip doesn't seem to support it at all.

backend-path is now used on the main branch requiring pip >=20 or something like python-build.

jaraco · 2021-02-26T19:10:24Z

I see two ways forward with this:

1. Fix this problem with `pip`, adjusting PEP 517 to allow this kind of situation if necessary.

2. Create a new "PEP 517 bootstrapper" backend, that simply delegates all backend tasks to a backend that exists in your local directory, then start using that.

After some consideration, I do think this project needs something like a bootstrapper backend, mainly to address the situation of a clean source checkout. In that situation, there is no metadata present. Setuptools currently works around that situation by requiring a separate copy of setuptools to build, which is okay, but also clumsy, because then when the sdist is built, it also declares a requirement on setuptools, even when it's not needed.

jaraco · 2021-02-27T03:58:26Z

In the above referenced pull request, I figured out that Setuptools can supply its bootstrapping metadata in a separate metadata file. There's still no way to ever make Setuptools build properly on pip 19, so it will just have to break.

jaraco mentioned this issue Jan 25, 2019

BackendUnavailable traceback with 19.0 pypa/pip#6164

Closed

jaraco added bug blocker labels Jan 25, 2019

This was referenced Jan 25, 2019

improve encoding handling for setup.cfg #1180

Merged

Inject '' into sys.path in PEP 517 implementation #1643

Closed

jaraco added a commit that referenced this issue Jan 26, 2019

Extract pip wrapper launcher. Ref #1644.

3e23fa7

jaraco added a commit that referenced this issue Jan 26, 2019

Pin pip to <19 until a more permanent solution can be devised. Ref #1644

6671b3f

.

jaraco mentioned this issue Jan 26, 2019

Bugfix/downgrade pip #1645

Closed

jaraco added a commit that referenced this issue Jan 26, 2019

Pin pip to <19 until a more permanent solution can be devised. Ref #1644

abf07f6

.

benoit-pierre mentioned this issue Jan 26, 2019

tests: temporary pin pip to fix the Python 2.7 job #1647

Merged

jaraco removed the blocker label Jan 27, 2019

pganssle mentioned this issue Jan 27, 2019

Start testing installation against pip on master #1649

Closed

jaraco mentioned this issue Jan 27, 2019

Include pyproject.toml in sdist #1634

Merged

2 tasks

jaraco changed the title ~~Tests on Python 2.7 failing to run with BackendUnavailable since pip 19~~ Tests failing to run with BackendUnavailable since pip 19 Jan 27, 2019

jaraco mentioned this issue Jan 27, 2019

Include pyproject.toml in sdist #1650

Merged

2 tasks

pganssle mentioned this issue Jan 27, 2019

Add build_meta_legacy backend #1652

Merged

2 tasks

pganssle mentioned this issue Feb 5, 2019

Explicitly declare our build backend #1674

Closed

1 task

pganssle mentioned this issue Mar 4, 2019

Throw an error when backend-path is specified #1714

Open

ziirish mentioned this issue Mar 12, 2019

BackendUnavailable while building the documentation readthedocs/readthedocs.org#5436

Closed

danyeaw mentioned this issue Apr 14, 2019

org.gaphor.Gaphor Builds are Failing on FlatHub BuildBot flathub/flathub#957

Closed

benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Aug 12, 2019

improve workaround for pypa#1644

92c9a7b

Make it possible to use a more recent version of pip for tests.

benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Aug 13, 2019

improve workaround for pypa#1644

8793c2d

Make it possible to use a more recent version of pip for tests.

benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Aug 22, 2019

improve workaround for pypa#1644

90bed1a

Make it possible to use a more recent version of pip for tests.

PertuyF mentioned this issue Sep 2, 2019

Add a conda-forge recipe for bash_kernel conda-forge/staged-recipes#9348

Merged

1 task

benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Oct 7, 2019

improve workaround for pypa#1644

0d831c9

Make it possible to use a more recent version of pip for tests.

benoit-pierre added a commit that referenced this issue Nov 13, 2019

Merge pull request #1864 from benoit-pierre/improve_workaround_for_1644

7956986

improve workaround for #1644

benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Nov 15, 2019

tweak workaround for pypa#1644

77fa236

Work around buggy pip detection code for "pip.exe install/update pip ...".

mergify bot added a commit that referenced this issue Nov 15, 2019

Merge pull request #1907 from benoit-pierre/tweak_workaround_for_1644

5ef7e75

tweak workaround for #1644

AlanCoding mentioned this issue Nov 20, 2019

Upgrade to a pre-19 pip version ansible/awx#5363

Closed

benoit-pierre mentioned this issue Nov 27, 2019

pyproject.toml: add "setuptools>=40.8.0" #1923

Closed

jaraco added a commit that referenced this issue Dec 1, 2019

Restore build-backend and remove switch to avoid pep517. Ref #1644.

7a709b6

jaraco mentioned this issue Dec 1, 2019

Restore build-backend and remove switch to avoid pep517. #1927

Merged

peterjc mentioned this issue Dec 23, 2019

TravisCI failures with release of NumPy 1.18.0 biopython/biopython#2426

Open

jaraco mentioned this issue Dec 29, 2019

Patched sdist in PEP 517 to include pyproject.toml #1949

Closed

2 tasks

dHannasch mentioned this issue Oct 13, 2020

Fix PyPy BackendUnavailable crash on Travis. ionelmc/cookiecutter-pylibrary#204

Closed

jaraco mentioned this issue Feb 26, 2021

Clarity on installation from source #2550

Open

jaraco mentioned this issue Feb 27, 2021

Rely on backend build path and bootstrap metadata to easily build from source #2582

Merged

jaraco closed this as completed in #2582 Feb 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests failing to run with BackendUnavailable since pip 19 #1644

Tests failing to run with BackendUnavailable since pip 19 #1644

jaraco commented Jan 25, 2019

jaraco commented Jan 25, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

benoit-pierre commented Jan 26, 2019

benoit-pierre commented Jan 26, 2019

jaraco commented Jan 26, 2019 •

edited

Loading

benoit-pierre commented Jan 26, 2019

jaraco commented Jan 27, 2019

daa commented Jan 28, 2019 •

edited

Loading

pganssle commented Jan 28, 2019

ncoghlan commented Feb 2, 2019

pganssle commented Feb 4, 2019

ncoghlan commented Feb 4, 2019

pganssle commented Nov 16, 2019

webknjaz commented Feb 4, 2021

jaraco commented Feb 26, 2021

jaraco commented Feb 27, 2021

Tests failing to run with BackendUnavailable since pip 19 #1644

Tests failing to run with BackendUnavailable since pip 19 #1644

Comments

jaraco commented Jan 25, 2019

jaraco commented Jan 25, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

jaraco commented Jan 26, 2019

benoit-pierre commented Jan 26, 2019

benoit-pierre commented Jan 26, 2019

jaraco commented Jan 26, 2019 • edited Loading

benoit-pierre commented Jan 26, 2019

jaraco commented Jan 27, 2019

daa commented Jan 28, 2019 • edited Loading

pganssle commented Jan 28, 2019

ncoghlan commented Feb 2, 2019

pganssle commented Feb 4, 2019

ncoghlan commented Feb 4, 2019

pganssle commented Nov 16, 2019

webknjaz commented Feb 4, 2021

jaraco commented Feb 26, 2021

jaraco commented Feb 27, 2021

jaraco commented Jan 26, 2019 •

edited

Loading

daa commented Jan 28, 2019 •

edited

Loading