-
-
Notifications
You must be signed in to change notification settings - Fork 522
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for PEP-517 #573
Comments
Hi @Alexander255, thanks for giving a detailed overview how this could work in tox. I have not yet looked into that topic at all yet and don't even know how many projects are using this. But if pip implements it, it might be a good idea to add it to tox. I don't see this happening for a long time though unless a champion steps up and takes over the implementation. |
Currently, I estimate the number to be around 0 because it's only available in the GIT version of PIP. 🙂
Agreed.
I knew you'd say that. 😁 Any pointers where it would make sense to start refactoring to get this into |
I admit that I am a bit on a mission to prioritize on improving process and squashing bugs rather than adding (big) new features in core :) About your question: I start leaning towards implementing new/experimental functionality like this as plugins. Check the hookspecs - if not everything needed is implemented as a hook I'd rather suggest adding the necessary hooks to core - you can also check the drafts done at the sprint last year about generalizing package builds, as this could have significant overlap: https://github.com/tox-dev/tox/tree/master/doc/drafts About that toml library: tox is very conservative in pulling in new dependencies and they have to be compatible with all interpreters tox supports. So this would be another reason for implementing this as a plugin. If the toml library only supports Python3 or Python2.7 and above it would be a restriction of the plugin as atm tox still supports 2.6 (but we will drop that shortly after pip does it, which is also happening with the release of 10.0 (pypa/pip#3955)). |
@Alexander255 I just looked into towncrier for changelog management and your estimate is at least off by one :) https://github.com/hawkowl/towncrier/blob/master/pyproject.toml |
@obestwalter: Only insofar as that they respect the tool namespace collision avoidance rules of PEP-518:
So one could probably make up an interesting legal case about whether that already entailes "using" PEP-518 or not 😉. |
BTW, I have just tested the Python |
@Alexander255 its the toml parser towncrier depends upon https://github.com/hawkowl/towncrier/blob/master/setup.py#L30 |
I have now proposed #577 for review and while it doesn't implement PEP-518, it does give us the required building block for doing so: The ability of building sdists in a |
Did you check first how this could be implemented as a plugin? |
TOML libraries: I had a brief look at the code of each a few months back, and at the time I felt more confident in pytoml than in toml. Consequently, pytoml is the one which is vendored into pip, and which will become a dependency of flit once PEP 517 is finalised. The developer of pytoml did not express great enthusiasm for ongoing maintenance, IIRC. But if it's used in pip, there's an incentive for other people to take over the maintenance if it becomes necessary. |
Hi @takluyver, thanks for pointing that out. It will be best to have a close eye on the developments in pip and follow their lead then whatever they will use for parsing toml. |
I honestly don't care which one ( Regarding a PEP-518 plugin: Currently tox definitely doesn't expose or support the required funcationality, but if the options of #577 land; it should be possible to use the Thinking further ahead (PEP-517), there would also need to be a new configuration option to launch an abitrary command to generate the sdist instead of the currently hard-coded |
o.k. thanks for checking it out. As your PR is quite big it will take a while until somebody will have the time to review it properly, so I hereby thank you already in advance for your patience :) |
Hi @Alexander255, about that principle thing: in principle I think tox should support workflows that are considered "good practice" - which is hard to define already, but if it's an accepted PEP this is pretty much a no brainer here. So yes: we should support it and ideally not too long after pip starts supporting it. Whether your PR is the right direction to take is then to be judged by a thorough review. If you provide documentation and tests I would say PR #577 will definitely be the way to go and only a question of how to whip it into shape. So if you want to keep working on this and bring this PR up to speed, I assure you my assistance as good as I can. |
I have a project where we're trying to get rid of So what's the current state of tox support for a |
I don't think tox supports I'm definitely happy to see more people trying flit, but there's still a way to go before it works smoothly with the rest of the ecosystem. |
@warsaw no support at the moment, however I plan to add it at some point 👍 |
Drawing from the experience implementing this in toxn (and with pip 10 coming out with PEP-518 support too), I'll add support for this in the next month, drawing up and during the PyCon sprint. |
Hey @gaborbernat nice - have fun at PyCon! |
Things I learned at todays packaging open space from PyCon. There actually is little we need to do ourselves. PEP-517 and PEP-518 define responsibilities for pip and setuptools/twine respectively. Now we do It is pips responsibility to build a package (and for this to create a virtual environment, install build dependencies, and then call the builder - setuptools/flit). This requires people interested to solve pypa/pip#4589, pypa/pip#4589 and then build on top of that to add such feature support. Virtualenv needs renewed love (read rewrite), pypa/virtualenv#697 is a rewrite of it, this also needs interested people to take it on. |
There's also a pep517 module providing an API for tools to go from a source tree to an sdist/wheel using the PEP 517 interface. I wrote that in the hope that it would make it easier for pip to add support for that interface. |
@takluyver what does that package actually contain out of:
Thanks! |
If you use the high-level API, it tries to deal with all of those steps. The lower level API is just the last two steps, if you want more control over the build environment. See the README for how to use the two different levels. |
Hmm, I wonder what's the hold up then of just exposing it as |
imho tox should collaborate with pip there to move the entire process of creating such build artifacts into pip and to simplify tox |
That's the goal here ^_^ |
created pypa/pip#5407 Once that's implemented the description of this issue changes to: Use |
### Update [tox](https://pypi.org/project/tox) from **3.2.1** to **3.4.0**. <details> <summary>Changelog</summary> ### 3.4.0 ``` ------------------ Bugfixes ^^^^^^^^ - add ``--exists-action w`` to default pip flags to handle better VCS dependencies (`pip documentation on this <https://pip.pypa.io/en/latest/reference/pip/exists-action-option>`_) - by :user:`gaborbernat` (`503 <https://github.com/tox-dev/tox/issues/503>`_) - instead of assuming the Python version from the base python name ask the interpreter to reveal the version for the ``ignore_basepython_conflict`` flag - by :user:`gaborbernat` (`908 <https://github.com/tox-dev/tox/issues/908>`_) - PEP-517 packaging fails with sdist already exists, fixed via ensuring the dist folder is empty before invoking the backend and `pypa/setuptools 1481 <https://github.com/pypa/setuptools/pull/1481>`_ - by :user:`gaborbernat` (`1003 <https://github.com/tox-dev/tox/issues/1003>`_) Features ^^^^^^^^ - add ``commands_pre`` and ``commands_post`` that run before and after running the ``commands`` (setup runs always, commands only if setup suceeds, teardown always - all run until the first failing command) - by :user:`gaborbernat` (`167 <https://github.com/tox-dev/tox/issues/167>`_) - ``pyproject.toml`` config support initially by just inline the tox.ini under ``tool.tox.legacy_tox_ini`` key; config source priority order is ``pyproject.toml``, ``tox.ini`` and then ``setup.cfg`` - by :user:`gaborbernat` (`814 <https://github.com/tox-dev/tox/issues/814>`_) - use the os environment variable ``TOX_SKIP_ENV`` to filter out tox environment names from the run list (set by ``envlist``) - by :user:`gaborbernat` (`824 <https://github.com/tox-dev/tox/issues/824>`_) - always set ``PIP_USER=0`` (do not install into the user site package, but inside the virtual environment created) and ``PIP_NO_DEPS=0`` (installing without dependencies can cause broken package installations) inside tox - by :user:`gaborbernat` (`838 <https://github.com/tox-dev/tox/issues/838>`_) - tox will inject some environment variables that to indicate a command is running within tox: ``TOX_WORK_DIR`` env var is set to the tox work directory, ``TOX_ENV_NAME`` is set to the current running tox environment name, ``TOX_ENV_DIR`` is set to the current tox environments working dir - by :user:`gaborbernat` (`847 <https://github.com/tox-dev/tox/issues/847>`_) - While running tox invokes various commands (such as building the package, pip installing dependencies and so on), these were printed in case they failed as Python arrays. Changed the representation to a shell command, allowing the users to quickly replicate/debug the failure on their own - by :user:`gaborbernat` (`851 <https://github.com/tox-dev/tox/issues/851>`_) - skip missing interpreters value from the config file can now be overridden via the ``--skip-missing-interpreters`` cli flag - by :user:`gaborbernat` (`903 <https://github.com/tox-dev/tox/issues/903>`_) - keep additional environments config order when listing them - by :user:`gaborbernat` (`921 <https://github.com/tox-dev/tox/issues/921>`_) - allow injecting config value inside the ini file dependent of the fact that we're connected to an interactive shell or not - by :user:`gaborbernat` (`947 <https://github.com/tox-dev/tox/issues/947>`_) - do not build sdist if skip install is specified for the envs to be run - by :user:`gaborbernat` (`974 <https://github.com/tox-dev/tox/issues/974>`_) - when verbosity level increases above two start passing through verbosity flags to pip - by :user:`gaborbernat` (`982 <https://github.com/tox-dev/tox/issues/982>`_) - when discovering the interpreter to use check if the tox host Python matches and use that if so - by :user:`gaborbernat` (`994 <https://github.com/tox-dev/tox/issues/994>`_) - ``-vv`` will print out why a virtual environment is re-created whenever this operation is triggered - by :user:`gaborbernat` (`1004 <https://github.com/tox-dev/tox/issues/1004>`_) Documentation ^^^^^^^^^^^^^ - clarify that ``python`` and ``pip`` refer to the virtual environments executable - by :user:`gaborbernat` (`305 <https://github.com/tox-dev/tox/issues/305>`_) - add Sphinx and mkdocs example of generating documentation via tox - by :user:`gaborbernat` (`374 <https://github.com/tox-dev/tox/issues/374>`_) - specify that ``setup.cfg`` tox configuration needs to be inside the ``tox:tox`` namespace - by :user:`gaborbernat` (`545 <https://github.com/tox-dev/tox/issues/545>`_) ``` ### 3.3.0 ``` ------------------ Bugfixes ^^^^^^^^ - fix ``TOX_LIMITED_SHEBANG`` when running under python3 - by :user:`asottile` (`931 <https://github.com/tox-dev/tox/issues/931>`_) Features ^^^^^^^^ - `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ source distribution support (create a ``.package`` virtual environment to perform build operations inside) by :user:`gaborbernat` (`573 <https://github.com/tox-dev/tox/issues/573>`_) - `flit <https://flit.readthedocs.io>`_ support via implementing ``PEP-517`` by :user:`gaborbernat` (`820 <https://github.com/tox-dev/tox/issues/820>`_) - packaging now is exposed as a hook via ``tox_package(session, venv)`` - by :user:`gaborbernat` (`951 <https://github.com/tox-dev/tox/issues/951>`_) Miscellaneous ^^^^^^^^^^^^^ - Updated the VSTS build YAML to use the latest jobs and pools syntax - by :user:`davidstaheli` (`955 <https://github.com/tox-dev/tox/issues/955>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/tox - Changelog: https://pyup.io/changelogs/tox/ - Docs: https://tox.readthedocs.org/ </details> ### Update [Sphinx](https://pypi.org/project/Sphinx) from **1.8.0** to **1.8.1**. <details> <summary>Changelog</summary> ### 1.8.1 ``` ===================================== Incompatible changes -------------------- * LaTeX ``\pagestyle`` commands have been moved to the LaTeX template. No changes in PDF, except possibly if ``\sphinxtableofcontents``, which contained them, had been customized in :file:`conf.py`. (refs: 5455) Bugs fixed ---------- * 5418: Incorrect default path for sphinx-build -d/doctrees files * 5421: autodoc emits deprecation warning for :confval:`autodoc_default_flags` * 5422: lambda object causes PicklingError on storing environment * 5417: Sphinx fails to build with syntax error in Python 2.7.5 * 4911: add latexpdf to make.bat for non make-mode * 5436: Autodoc does not work with enum subclasses with properties/methods * 5437: autodoc: crashed on modules importing eggs * 5433: latex: ImportError: cannot import name 'DEFAULT_SETTINGS' * 5431: autodoc: ``autofunction`` emits a warning for callable objects * 5457: Fix TypeError in error message when override is prohibited * 5453: PDF builds of 'howto' documents have no page numbers * 5463: mathbase: math_role and MathDirective was disappeared in 1.8.0 * 5454: latex: Index has disappeared from PDF for Japanese documents * 5432: py domain: ``:type:`` field can't process ``:term:`` references * 5426: py domain: TypeError has been raised for class attribute ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/sphinx - Changelog: https://pyup.io/changelogs/sphinx/ - Homepage: http://sphinx-doc.org/ </details> ### Update [twine](https://pypi.org/project/twine) from **1.11.0** to **1.12.0**. <details> <summary>Changelog</summary> ### 1.12.0 ``` * :feature:`395 major` Add ``twine check`` command to check long description * :feature:`392 major` Drop support for Python 3.3 * :feature:`363` Empower ``--skip-existing`` for Artifactory repositories * 🐛`367` Avoid MD5 when Python is compiled in FIPS mode ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/twine - Changelog: https://pyup.io/changelogs/twine/ - Docs: https://twine.readthedocs.io/ </details>
This PR updates [tox](https://pypi.org/project/tox) from **3.1.2** to **3.4.0**. <details> <summary>Changelog</summary> ### 3.4.0 ``` ------------------ Bugfixes ^^^^^^^^ - add ``--exists-action w`` to default pip flags to handle better VCS dependencies (`pip documentation on this <https://pip.pypa.io/en/latest/reference/pip/exists-action-option>`_) - by :user:`gaborbernat` (`503 <https://github.com/tox-dev/tox/issues/503>`_) - instead of assuming the Python version from the base python name ask the interpreter to reveal the version for the ``ignore_basepython_conflict`` flag - by :user:`gaborbernat` (`908 <https://github.com/tox-dev/tox/issues/908>`_) - PEP-517 packaging fails with sdist already exists, fixed via ensuring the dist folder is empty before invoking the backend and `pypa/setuptools 1481 <https://github.com/pypa/setuptools/pull/1481>`_ - by :user:`gaborbernat` (`1003 <https://github.com/tox-dev/tox/issues/1003>`_) Features ^^^^^^^^ - add ``commands_pre`` and ``commands_post`` that run before and after running the ``commands`` (setup runs always, commands only if setup suceeds, teardown always - all run until the first failing command) - by :user:`gaborbernat` (`167 <https://github.com/tox-dev/tox/issues/167>`_) - ``pyproject.toml`` config support initially by just inline the tox.ini under ``tool.tox.legacy_tox_ini`` key; config source priority order is ``pyproject.toml``, ``tox.ini`` and then ``setup.cfg`` - by :user:`gaborbernat` (`814 <https://github.com/tox-dev/tox/issues/814>`_) - use the os environment variable ``TOX_SKIP_ENV`` to filter out tox environment names from the run list (set by ``envlist``) - by :user:`gaborbernat` (`824 <https://github.com/tox-dev/tox/issues/824>`_) - always set ``PIP_USER=0`` (do not install into the user site package, but inside the virtual environment created) and ``PIP_NO_DEPS=0`` (installing without dependencies can cause broken package installations) inside tox - by :user:`gaborbernat` (`838 <https://github.com/tox-dev/tox/issues/838>`_) - tox will inject some environment variables that to indicate a command is running within tox: ``TOX_WORK_DIR`` env var is set to the tox work directory, ``TOX_ENV_NAME`` is set to the current running tox environment name, ``TOX_ENV_DIR`` is set to the current tox environments working dir - by :user:`gaborbernat` (`847 <https://github.com/tox-dev/tox/issues/847>`_) - While running tox invokes various commands (such as building the package, pip installing dependencies and so on), these were printed in case they failed as Python arrays. Changed the representation to a shell command, allowing the users to quickly replicate/debug the failure on their own - by :user:`gaborbernat` (`851 <https://github.com/tox-dev/tox/issues/851>`_) - skip missing interpreters value from the config file can now be overridden via the ``--skip-missing-interpreters`` cli flag - by :user:`gaborbernat` (`903 <https://github.com/tox-dev/tox/issues/903>`_) - keep additional environments config order when listing them - by :user:`gaborbernat` (`921 <https://github.com/tox-dev/tox/issues/921>`_) - allow injecting config value inside the ini file dependent of the fact that we're connected to an interactive shell or not - by :user:`gaborbernat` (`947 <https://github.com/tox-dev/tox/issues/947>`_) - do not build sdist if skip install is specified for the envs to be run - by :user:`gaborbernat` (`974 <https://github.com/tox-dev/tox/issues/974>`_) - when verbosity level increases above two start passing through verbosity flags to pip - by :user:`gaborbernat` (`982 <https://github.com/tox-dev/tox/issues/982>`_) - when discovering the interpreter to use check if the tox host Python matches and use that if so - by :user:`gaborbernat` (`994 <https://github.com/tox-dev/tox/issues/994>`_) - ``-vv`` will print out why a virtual environment is re-created whenever this operation is triggered - by :user:`gaborbernat` (`1004 <https://github.com/tox-dev/tox/issues/1004>`_) Documentation ^^^^^^^^^^^^^ - clarify that ``python`` and ``pip`` refer to the virtual environments executable - by :user:`gaborbernat` (`305 <https://github.com/tox-dev/tox/issues/305>`_) - add Sphinx and mkdocs example of generating documentation via tox - by :user:`gaborbernat` (`374 <https://github.com/tox-dev/tox/issues/374>`_) - specify that ``setup.cfg`` tox configuration needs to be inside the ``tox:tox`` namespace - by :user:`gaborbernat` (`545 <https://github.com/tox-dev/tox/issues/545>`_) ``` ### 3.3.0 ``` ------------------ Bugfixes ^^^^^^^^ - fix ``TOX_LIMITED_SHEBANG`` when running under python3 - by :user:`asottile` (`931 <https://github.com/tox-dev/tox/issues/931>`_) Features ^^^^^^^^ - `PEP-517 <https://www.python.org/dev/peps/pep-0517/>`_ source distribution support (create a ``.package`` virtual environment to perform build operations inside) by :user:`gaborbernat` (`573 <https://github.com/tox-dev/tox/issues/573>`_) - `flit <https://flit.readthedocs.io>`_ support via implementing ``PEP-517`` by :user:`gaborbernat` (`820 <https://github.com/tox-dev/tox/issues/820>`_) - packaging now is exposed as a hook via ``tox_package(session, venv)`` - by :user:`gaborbernat` (`951 <https://github.com/tox-dev/tox/issues/951>`_) Miscellaneous ^^^^^^^^^^^^^ - Updated the VSTS build YAML to use the latest jobs and pools syntax - by :user:`davidstaheli` (`955 <https://github.com/tox-dev/tox/issues/955>`_) ``` ### 3.2.1 ``` ------------------ Bugfixes ^^^^^^^^ - ``--parallel--safe-build`` no longer cleans up its folders (``distdir``, ``distshare``, ``log``). - by :user:`gaborbernat` (`849 <https://github.com/tox-dev/tox/issues/849>`_) ``` ### 3.2.0 ``` ------------------ Features ^^^^^^^^ - Switch pip invocations to use the module ``-m pip`` instead of direct invocation. This could help avoid some of the shebang limitations. - by :user:`gaborbernat` (`935 <https://github.com/tox-dev/tox/issues/935>`_) - Ability to specify package requirements for the tox run via the ``tox.ini`` (``tox`` section under key ``requires`` - PEP-508 style): can be used to specify both plugin requirements or build dependencies. - by :user:`gaborbernat` (`783 <https://github.com/tox-dev/tox/issues/783>`_) - Allow to run multiple tox instances in parallel by providing the ``--parallel--safe-build`` flag. - by :user:`gaborbernat` (`849 <https://github.com/tox-dev/tox/issues/849>`_) ``` ### 3.1.3 ``` ------------------ Bugfixes ^^^^^^^^ - A caching issue that caused the ``develop-inst-nodeps`` action, which reinstalls the package under test, to always run has been resolved. The ``develop-inst-noop`` action, which, as the name suggests, is a no-op, will now run unless there are changes to ``setup.py`` or ``setup.cfg`` files that have not been reflected - by stephenfin (`909 <https://github.com/tox-dev/tox/issues/909>`_) Features ^^^^^^^^ - Python version testenvs are now automatically detected instead of comparing against a hard-coded list of supported versions. This enables ``py38`` and eventually ``py39`` / ``py40`` / etc. to work without requiring an upgrade to ``tox``. As such, the following public constants are now deprecated (and scheduled for removal in ``tox`` 4.0: ``CPYTHON_VERSION_TUPLES``, ``PYPY_VERSION_TUPLES``, ``OTHER_PYTHON_INTERPRETERS``, and ``DEFAULT_FACTORS`` - by :user:`asottile` (`914 <https://github.com/tox-dev/tox/issues/914>`_) Documentation ^^^^^^^^^^^^^ - Add a system overview section on the index page that explains briefly how tox works - by :user:`gaborbernat`. (`867 <https://github.com/tox-dev/tox/issues/867>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/tox - Changelog: https://pyup.io/changelogs/tox/ - Docs: https://tox.readthedocs.org/ </details>
Currently
tox
does not yet appear have support for thebuildsystem.requires
key ofpyproject.toml
as defined in PEP-518.PIP has implemented this: pypa/pip#4144 (will be part of PIP 10.0) and it would be nice if tox could also add support for installing packages defined using that key, instead of having to use the broken
setup_requires
key of./setup.py
.Example of a
pyproject.toml
file that creates a setuptools 34+ environment:Example of a
pyproject.toml
file that creates an environment for buildingflit
packages:(Note that this kind of setup is pretty useless without support for PEP-517.)
Approximate step by step algorithm for supporting both
PEP-517
and "legacy"./setup.py
deployments (based on PEP-518 & how PIP choose to implement it):skipsdist
in thetox
section oftox.ini
istrue
, do nothingvirtualenv
to host thesdist
generation environmentpyproject.toml
exists in thetoxinidir
:pyproject.toml
as TOML file (using the python toml library for instance)build-system.requires
key and treat each entry in the array as anPEP-508
dependency specificationvirtualenv
setuptools
andwheel
into thevirtualenv
./setup.py sdist
within thevirtualenv
to generate thesdist
used for the subsequent testing steps(The build environment may be reused between subsequent testing sessions, as long as the list of requirements doesn't change.)
The text was updated successfully, but these errors were encountered: