Are twine maintainers OK to have a twine build command?

[pradyunsg]

@pradyunsg looks at https://discuss.python.org/t/building-distributions-and-drawing-the-platypus/2062 and sighs.

Hey @pypa/twine-team!

Are y'all fine with twine growing a twine build command in the future? Do twine maintainers feel like twine build would be out of scope for the tool?

The relevant context is in the first post of the discussion that I've linked to above. That discussion quickly diverged from the topic of "how to expose build logic" to "what should a unified tool for Python packaging/development workflow look like", so most replies there are not relevant to the question I'm asking here.

If we'd be doing this, it'd be based on a shared implementation of the underlying build and install logic, that would be the result of refactoring "out" pip's existing build and install logic. This is something that I plan to be working on later this year, and I'm trying to figure out the direction that work should be headed.

Before someone asks, yes, I checked with my fellow pip maintainers -- there's a long on-going discussion within that group privately, about what's "in scope" for pip. It seems to me that we'd be concluding that a pip build would be out of scope, which is why I'm preempting that conclusion and checking here.

[jaraco]

This question interacts with the discussion in #160, but also pypa/pip#6041 and pypa/packaging-problems#219.

The short answer is "no": twine shouldn't have a build command until there's consensus on where that functionality should reside. That issue should be reconciled in pypa/packaging-problems#219. If the resolution there is that twine should host that functionality, then this project will enthusiastically adopt that responsibility.

[jaraco]

So many discussions still open. I just read through pypa/pip#6041, which led me back to your original reference. Please don't take my closure of this issue as authoritative or prejudiced. I encourage your work to try to find a solution to this user-facing workflow issue.

[pradyunsg]

there's consensus on where that functionality should reside.

Right, but for building that concensus, I think it'd be useful to know if the maintainers of twine even want to be responsible for supporting/maintaining this functionality -- which is the question I'm asking here.

[pradyunsg]

FWIW, I am aware that the question is intertwined with many other discussions happening in many locations -- I am actively involved in nearly all those venues of discussion and initiated some of them. #160 also links back to the d.p.o discussion I linked to above.

Anyway, I think what I'm asking here is much more scoped and independent of that. :)

If the resolution there is that twine should host that functionality, then this project will enthusiastically adopt that responsibility.

I'm gonna move ahead assuming that all @pypa/twine-maintainers agree with this, so please holler if you disagree.

I'm not gonna reopen this issue even though I have the permissions to do so, but other folks are welcome to do so if they disagree with this, and would like to discuss the specifics (of the question I've asked in this issue), in further detail.

[jaraco]

I think it'd be useful to know if the maintainers of twine even want to be responsible for supporting/maintaining this functionality.

I do believe I've answered that question; this project will enthusiastically support that need if that's the recommended approach. I welcome others on the project to chime in before closing.

[sigmavirus24]

For years I've had this on the horizon in my mind for twine. I don't have time to commit to writing it though or keeping up with all of the latest standards and such so I never volunteered it because twine's resources are already stretched thin

[sigmavirus24]

That is to say: As a user of twine, I've wanted a twine build for forever, I just haven't needed it enough to do it

[di]

In my experience, users tend to think of twine as "a tool for package publishers" and pip as "a tool for package installers". If the goal is to replace python setup.py ... (which IMO it should be) then I think regardless of the fact that pip needs this logic under the hood to do its job, twine should be the place where this API is exposed to users.

It seems to me that a library like pep517, which knows how to build PEP517/518 distributions, could and should be reused across both tools.

One other thing: I'm hesitant to call this command twine build as it's not clear to me how the creation of source distributions, which are by definition not built, would fall under this command. This area is already pretty confusing for users. Maybe the command should be something other than build, like:

• twine package source
• twine package wheel

[pradyunsg]

puts on bikeshedding hat with an evil smile

@di How about generate -- generating distributions?

[jaraco]

Maybe the command should be something like twine package source

I like package, but I'd recommend following the pattern in pep517.build, namely, one command with --source and --binary options, the default being to package both.

[di]

I like package, but I'd recommend following the pattern in pep517.build, namely, one command with --source and --binary options, the default being to package both.

I think that's more or less what I proposed. Users could just run twine package and get both, or whatever twine's current recommendation is.

Also, note that not all built distributions contain binaries, so twine package binary might be confusing if the goal is to build a pure-Python wheel.

[pradyunsg]

puts on bikeshedding hat with an evil smile

Joking aside, everyone who has chimed in till now has basically agreed with what @jaraco said earlier -- that twine will be happy to host this functionality. That's nice, and I think I can move on to check that the broader community is cool with this happening.

[xavfernandez]

Not a twine maintainer but having twine gain the build command (instead of pip) means that twine will have to implement PEP-517 meaning that it will need to invoke an installer meaning that it will either:

• re-implement parts of pip functionalities
• proxy pip options to the user

to allow pip build . --no-index --find-links=directory_containing_some_build_dependencies (and other special cases with --extra-index-url to work).

[sigmavirus24]

@xavfernandez I wonder if this would be a good opportunity to discover what parts of pip functionality can/should be broken out into a new library for sharing between the two. It seems like you're arguing against this living in twine simply because pip has some fraction of what we're looking for. Are you against working with twine to move some of the logic into a library?

Adding my particular colour choice to the bike-shed, I think build is a more obvious verb for folks who don't speak English as their first or even second language. I'm fine with package but I don't know that it's significantly better. Also, generate has connotations to me of invoking some kind of code-generation tool at the same time, but perhaps I've been working with protobufs and go for too long now.

I personally also dislike (although that might be a more mild word than I actually feel) having --source and --binary be the options because the community may move towards something else in the future requiring a new option. I'd rather have --target or something else that can take a comma-separated-list (or is repeatable, e.g., twine build --target source --target build) of things like source and binary and can then expand to meet the needs of the community in the future. That said, I think --target is imperfect as it might imply building for a specific target (e.g., macOS versus Linux versus Windows).

[pradyunsg]

If we'd be doing this, it'd be based on a shared implementation of the underlying build and install logic, that would be the result of refactoring "out" pip's existing build and install logic. This is something that I plan to be working on later this year, and I'm trying to figure out the direction that work should be headed.

@xavfernandez @sigmavirus24 ^

(not "don't touch this area, MINE!" but more like, "hey, so uh... I flagged this too.")

[xavfernandez]

Are you against working with twine to move some of the logic into a library?

I would not phrase it like that. I'm certainly in favour of moving some pip internals into libraries.

I was just pointing out that, to implement PEP-517 you need a full fledged installer.
So you either subprocess to pip (and need to expose some of pip CLI options) or you reimplement an installer (eventually based on pip internals libraries as mentionned by Pradyun).
Implementing the build command directly in pip seems simpler than both of those options.

[sigmavirus24]

Implementing the build command directly in pip seems simpler than both of those options.

Oh certainly it would almost definitely be easier. It'd probably be a better UX too since someone might download a project and want to build it without ever installing twine or intending to reupload the built package elsewhere. That said, I can see the value in having something external to pip even if it's far less than ideal

[sigmavirus24]

(not "don't touch this area, MINE!" but more like, "hey, so uh... I flagged this too.")

Thanks for reminding me @pradyunsg :D

[sigmavirus24]

Also worth noting, we use entry-points for our subcommands. We could (if it were deemed desirable) distribute this as a separate package that registers itself as twine-build.

[merwok]

For what it’s worth, I am confused by the request.

(I haven’t re-read the whole platypus thread. Also we used to call it the packaging pig, friend of the testing goat! 🙂)

I thought twine’s point was: better tool than python setup.py upload.
Then it was also useful for build tools other than setuptools: use what you want to make a source dist and a wheel, use twine to upload them.
Then there was the discussion: pip has so many things in it that it could upload too; merge twine?
Then some build tools added upload to their features, making things simpler for their users (flit build and flit upload, done).

Now twine should do the build?

[sigmavirus24]

[sigmavirus24]

On a less snarky note, I for one haven't had enough time to start messing with the other tools that do build artefacts. As such, my process is still rm -rf build dist; python setup.py sdist bdist_wheel; twine upload. Simplifying that to just use twine would make me happy. I suspect previously folks were doing $buildTool build; twine upload and that became annoying to them. Frankly this is why I started trying to design an API into Twine to make those $buildTool's lives easier for doing the integration and have upload options that look like Twine. I don't frankly care what people use. And while there should be one obvious way to do it, that's never been entirely true when it comes to packaging. I also view it as twine build could be somewhat agnostic if we wanted it to. It could use flit if folks want it to or whatever other build options exist today. I don't think twine needs to be opinionated in this sense.

[sigmavirus24]

Everything above said, we'll also want to build incremental builds with correctness. We want to avoid some of the unintended consequences of pypa/pip#7882 while also providing a fast and intuitive experience. That fundamentally changes this from "How do we do builds?" to "How do we detect when we can do incremental builds* versus when do we have to copy the entire source to a temporary directory like pip used to?"

*Where 'incremental builds' and doing those well is left as an exercise to the implementer ;)

[pradyunsg]

https://discuss.python.org/t/building-distributions-and-drawing-the-platypus/2062/67?u=pradyunsg

I am no longer investing effort into this directly, and noting the latest developments in this space for completeness. It seems to me that some of the folks who were previously opposed to having an independent tool for this task, are now in favor of doing so.

[pradyunsg]

Closing since https://github.com/pypa/build exists now.