fix version numbering

[phillyc]

I just spent 4 hours with another engineer trying to figure out why our builds weren't producing the same results.

Turns out I had html5lib "0.9999999" and he had "0.999999999" which was nearly impossible to spot on the pip list readout. >.<

http://semver.org/#why-use-semantic-versioning

I'm going to guess that you'll say html5lib isn't ready for production so your versioning scheme is appropriate, but I'm still filing this bug because I'm steamed about losing so much development time. :/

[shaileshahuja]

I didn't even realize I was facing this problem until I read this issue. I was wondering how are the same versions behaving differently!

[phillyc]

I should point out that they don't appear to to be the same versions. I'm running "0.9999999" which is seven nines. While my coworker is running "0.999999999" which is nine nines. :/

[gsnedders]

Alas, it's a bit hard to change now.

For those wondering how we ended up here, in September 2013 when we moved to asymptotically approaching 1.0 I expected a release of 1.0 by the end of 2013. Alas, that didn't happen, for a variety of reasons (primarily the spec having some large changes and not wanting to ship something out of date, and lack of time for me and lack of other contributors). It's worthwhile pointing out that html5lib in its origins was essentially a research project and as a support tool to those suggesting changes to the spec, as such it had some weird features and some weird oddities in it.

Now, perhaps 0.99999999 should have been 0.100000000 to clarify the changes, but that's also a confusing version number.

Also in hindsight, we probably should've shipped one of the releases in 2013 as 1.0.

Ultimately, what's blocking shipping 1.0 now is having something up-to-date with the spec and having documentation.

[phillyc]

Maybe a simple solution would be to use 0.9.7 -> 0.9.9 -> 0.9.10 ??

Then whenever you feel comfortable flipping to 1.0.0 it's a simple and easy read?

[gsnedders]

Maybe a simple solution would be to use 0.9.7 -> 0.9.9 -> 0.9.10 ??

But 0.9.7 is lower version than 0.99 (yet alone the current version!), hence that doesn't work.

[thibaudcolas]

If this project aims to follow SemVer (#278), the minor version number is supposed to be incremented by 1 for each minor release. What should have happened is 0.9, 0.10, 0.11.

It's understandable that html5lib might not have been following SemVer at the time / might not have had much reasons to version releases, but now this is getting hard to read. Better start now even if it still looks silly. What about the following?

• 0.999999999 (current)
• 0.1000000000 (next release)
• 0.1000000001 (next + 1)
• 0.1000000002 (next + 2)
• 0.1000000003 (etc)

[jedie]

It was a very, very bad idea to add a new "9" to the version number in the past!

What's about this:

• 0.999999999 (current)
• 0.9999999991 (next release)
• 0.9999999992 (next + 1)
• ...
• 0.9999999998 (max)
• 1.0

(Release as soon as possible v1.0)

[RichardLukins]

Just put a "." on the end of what we have now and add using SemVer standards there after
ie 0.999999999.1
then 0.999999999.2 ... .8 .9 .10 .11 etc and hopefully 1.0 will happen before long too!! :-)
at least you will see clearly if you are current version or not and how many you are behind

[joernhees]

Ignoring http://semver.org and your fear of version 1.0.0 costs a lot more developer time than necessary.

I remember fixing issues with rdflib's dependency on html5lib a couple of times already and it seems you broke backwards compatibility again: https://travis-ci.org/RDFLib/rdflib/jobs/147499177, this time with default setuptools on travis?!?

Would be cool if you would live up to your own slogan:

Standards-compliant library for parsing and serializing HTML documents and fragments in Python

Why invest so much time to create a great lib, but then make it harder to use than necessary by fucking up the version numbers?

[tonthon]

My solution was to fix the version using the alternative version numbers (1.0b8 in my case) see the full list in the changelog on pypi.python.org :
https://pypi.python.org/pypi/html5lib

[graingert]

pip install html5==0.0.9

[graingert]

(the lib in html5lib is redundant because everything on PyPI is a lib)

[graingert]

Ultimately, what's blocking shipping 1.0 now is having something up-to-date with the spec and having documentation.

You don't need to be up to date with the spec, or have documentation for semver 1.0.0. You just need to be ready to bump the version 2.0.0 when you add a breaking change.

[Chronial]

+1 on graingert's statement – just release 1.0. The confusion from having an imperfect 1.0 is surely going to be less than the confusion caused by the current version system :).

PS: Thanks for this great lib :)

[gsnedders]

@graingert:

You don't need to be up to date with the spec, or have documentation for semver 1.0.0. You just need to be ready to bump the version 2.0.0 when you add a breaking change.

The problem is defining what a breaking change is when you're in a language like Python where everything is public. Pretty sure our options are essentially:

• Underscore prefix everything in one fell swoop and hope people believe anything prefixed isn't a stable API. (Which, to be fair, probably mostly suffices!) As an aside, doing much of this is a large part of what caused breakage in the last release.
• Increment major version number quickly by underscore prefixing things when they're next changed. (And likely this will happen even if the above is done because some will get missed initially despite best intentions.)
• Document everything that is intended to be a stable API (and this doesn't have to be perfect documentation; at some level even a simple list of "these APIs are stable" would suffice here).

As for the naming of the package, I'd argue the "5" is misleading as well, but renaming the package almost certainly causes more pain than it's worth, and there's already a html package on PyPI.

@joernhees:

The problems with setuptools are its own messes; see this for a summary of the different behaviours of different versions of setuptools on different Python implementations… Notably, none of platform_python_implementation, platform.python_implementation, or python_implementation work across all commonly used setuptools versions on all Python implementations. We now have a really, really, really hideous monkey-patch for setuptools in our setup.py from #285 to make it work everywhere.

[phillyc]

So, so sorry for causing all this drama. (even featured on github-drama!)

At least we know now that there are a lot of users experiencing similar grief.

I think you're on the right track; document, prefix, and update.

This would give users a path to update themselves out of the version number problem.

I think @graingert was indicating that bumping from 0.999999 to 1.0.0 is not normally considered a "breaking change", but 1.0 -> 2.0 is. @RichardLukins is right on the money with using 0.9999.1 etc as an interim.

Don't rename the package! :)

[NeilHanlon]

This is actually a joke, right?

[mvasilkov]

Let's just say that it's in hexadecimal, and move on: 0.999999999 => 0.A.

[RReverser]

@mvasilkov 0xA is not bigger than 999999999 though

[mvasilkov]

@RReverser 0.A (as in, A * 0.1) is, however, greater than 0.999999999.

[adamchainz]

0.999999999 version number is not interpreted as decimal 0.99999.. but rather the pieces are separated, into 0 and 99999... , then the comparison is made. Hence version 0.10 > 0.9. http://semver.org/ has an explanation.

[RReverser]

@adamchainz @mvasilkov Well, exactly, it's split. So 0.1 has 1 in second position and 0.999999999 has 999999999 in second position, and 1 < 999999999. If you would have 0.1000000000, then yes, it would be higher. Explicit zeros matter.

[yhaskell]

Why don't you just call it 1.0.0-RC1 For example? All problems would be resolved: It would be "almost ready". And then you could go as far as RC9999 without problems.

[ThYpHo0n]

The problem is defining what a breaking change is when you're in a language like Python where everything is public.

If you're not sure that your code changes are BC just increase the mayor version. Don't be afraid of it.

I like the idea of the RC version.

[graingert]

@ThYpHo0n @yhaskell fyi you can install html5lib==1.0b10

[ThYpHo0n]

@graingert Thanks, that works.
pip install 'html5lib>=1.0' Collecting html5lib>=1.0 Could not find a version that satisfies the requirement html5lib>=1.0 (from versions: 0.2, 0.9, 0.10, 0.11, 0.11.1, 0.90, 0.95, 0.99, 0.999, 0.9999, 0.99999, 0.999999, 0.9999999, 0.99999999, 0.999999999, 1.0b1, 1.0b2, 1.0b3, 1.0b5, 1.0b6, 1.0b7, 1.0b8, 1.0b9, 1.0b10)

pip install 'html5lib>=1.0b' Collecting html5lib>=1.0b Using html5lib-1.0b10-py2.py3-none-any.whl

[adamchainz]

I think @defjaf 's problem is probably the packages that depend on it pinning the version, nothing to do with pip

I've had the problem in the past, and the lesson I learned from pip was that when you upgrade packages, only upgrade those which you explicitly installed.

That's bad advice, some packages have very strict dependencies, like every celery version requires a specific version of kombu. I use pip-compile from https://github.com/nvie/pip-tools to solve these problems, it's much better than pip install --upgrade

[ghost]

@adamchainz I did not mean doing isolated upgrades, but only passing explicitly installed packages to pip and I think dependencies get upgraded wherever they have to satisfy these updates' prequisites.

[adamchainz]

OK my bad, I misunderstood your advice.

Anyway back on topic, still looking forward to the day html5lib 1.0.0 comes out.

[defjaf]

Well, this is happening with projects like jupyter's notebook, which I would imagine tries pretty hard to comply with standards. Just this morning, that was upgraded to 4.3.2, and the upgrade had the following:

 $ pip3 install --upgrade notebook

  <.... many lines removed....>

 Collecting html5lib!=0.9999,!=0.99999,<0.99999999,>=0.999 (from bleach->nbconvert->notebook)
   Using cached html5lib-0.9999999.tar.gz

  <.... many lines removed....>

 Installing collected packages: notebook, html5lib
   Found existing installation: notebook 4.3.1
     Uninstalling notebook-4.3.1:
       Successfully uninstalled notebook-4.3.1
   Found existing installation: html5lib 0.999999999
     Uninstalling html5lib-0.999999999:
       Successfully uninstalled html5lib-0.999999999
   Running setup.py install for html5lib ... done
 Successfully installed html5lib-0.9999999 notebook-4.3.2

(So a subsequent pip list --outdated shows html5lib needing an upgrade, which gets installed by pip install --upgrade html5lib -- and, importantly, doesn't seem to have any actual conflicts with the jupyter notebook...)

[meawoppl]

What should we call the fork? I am thinking html6lib-python to optimize how much people in the future hate us.

[adamchainz]

@graingert has html5 on PyPI: https://pypi.org/project/html5/

[graingert]

Will hand make a little cron job to copy over PyPI versions at some point

[nils-werner]

Incredible, over a year has passed and nothing has changed.

[Engineero]

According to PEP 440: Version Epochs, the current standard provides the epoch option for, for instance, changing the version numbering scheme in a way that would normally break version parsing. If you prefer to avoid version 1.0, use an epoch!

0.999999999 -> 1!0.9.0

From then on, versions are 1!0.9.1, then 1!0.9.2, ... until 1!0.9.10 and so on, and finally 1!1.0.0. From this point forward, I would also recommend following at least the gist of semver.

The "1!" in the lead may be a little bit ugly, but it is a sight better than 0.999999999999999999999 nines for days.

From the docs:

[...] all versions from a later epoch are sorted after versions from an earlier epoch

so I think the "1!" in the front would be a permanent fixture. Still, after we get versions >= 1!1.0.0, it will be clear enough to refer to the version number in conversation without the epoch.

[graingert]

@Engineero ah yes but html5lib do eventually intend to release v1. Plain v1 no epoch

[graingert]

@meawoppl my fork is called html5. no python and no lib. Only python libs exist on pypi, no point in including redundant info in the package name!

[Engineero]

@graingert yes, eventually. They've been "eventually" intending to do that since 2013, and that is how we got here in the first place. They should fix it now, not hold off because 1.0 will be here "eventually".

[willkg]

@Engineero I'd love help working through the 1.0 milestone! Please help in any way you can.

[willkg]

@Engineero @graingert @meawoppl @nils-werner @adamchainz and everyone else who's felt commented in this issue, I really could use some help getting 1.0 out.

You all feel strongly about a 1.0 release and you've taken the time to let your voice be heard as such and mentioned forks and other things, so I know you've got a vested interest in helping to make it happen. I appreciate it's been frustrating that no one likes the version numbering and that this project has slowed way down.

I work on Bleach which uses html5lib-python. I've been experiencing many of the problems you all have. So I offered to step up as a maintainer-ish person to shepherd a 1.0 release. Generally, the experience has been pretty lackluster--the issue tracker is filled with comments from entitled people spewing entitled sentiments and a lack of goodwill, compassion, thoughtfulness, help, or anything else positive. I really don't like working on this project. I doubt I'll continue working on this project after 1.0 is out.

As I understand it, this library has zero full time maintainers. No one is paid to work on it. No company sponsors work on it. It has a couple of volunteers who spend time working on it in their free time with no compensation. Seems like the number of volunteers has gradually gone down over the years.

This library is a huge project covering a complicated problem domain.

This library is used in a vast number of other libraries like Bleach, pip, TensorFlow, Jupyter, and others:

https://libraries.io/pypi/html5lib

Is it a critical library in the Python universe? If so, it's weird it's in this state of total neglect by the community of its users.

Anyhow, long story short, that's where things are at.

I'd love to get 1.0 out by the end of November and solve the version number problem, but I can't do it without help.

Five things you should do to help in the next week:

1. write a positive comment and change this issue from a bunch of entitled kvetching into a cheer squad about the work that's being done
2. test the current master tip of html5lib-python with the libraries you're using and write up new issues for anything you bump into
3. flesh out the docstrings and other documentation and create PRs for those changes
4. work through any of the issues and PRs in the 1.0 milestone to get those resolved
5. think about how you can help other projects you depend on with your time, your money, and your positivity

Thanks in advance!

And for everyone who has helped with 1.0 stuff so far--thank you! I really appreciate it!

[graingert]

@willkg have you thought about creating an opencollective account for html5lib?

[willkg]

@graingert I created issue #361 about that. Let's keep this issue about fixing the version number problem which necessitates 1.0 being finished.

[hugovk]

Here's a handy link to the 1.0 milestone:
https://github.com/html5lib/html5lib-python/milestone/1

Stats from 17th October show html5lib was downloaded 9 513 541 times from PyPI in the preceding 365 days.

[Engineero]

@willkg I will see what I can do! If nothing else, I'm one of those weirdos who likes making sure code is documented.

[willkg]

Today is November 30th. I was hoping to have 1.0 out the door today, but it's not going to happen. Will we get it out the door next week? That's on you.

In the last 9 days, I've gotten some help landing things from @gsnedders, @hugovk, and @jayvdb. Thank you! I really appreciate it!

I've gotten promises of help from other people, but none of that has materialized. We're all busy--I get that--but you all want 1.0 to get released and many of you have expressed horror at the current versioning. Please help!

Here's the milestone: https://github.com/html5lib/html5lib-python/milestone/1

There are a couple of easy fixes that we could/should do. Add a comment, claim the issue, and try to do it in the next couple of days. I'll review and help you land those changes.

I'm going through and documenting the publicly supported API. Unfortunately (or fortunately if you've seen the quality of my work), I can't land any of them on my own--I need someone else to review them. Go through the changes. Do they make sense to you? Are there examples that would help? Did I tpyo?

Please spend some time this week helping me get 1.0 out the door!

[phillyc]

@willkg First off, thank you so so much for stepping up to tackle this problem. I appreciate it more than I can express.

Secondly, I'd like to apologize for the initial negativity I expressed when I opened the issue. I considered editing it, but I want it to stand as is to remind me to be humble and kind when using other's software. No, I'm not in trouble with anyone. Nobody is forcing me to apologize, I just think it's the right thing to do.

Finally, I'll gladly take a look at the milestone to see if I can contribute in any way.

[gsnedders]

With a huge thanks to @willkg for making everything happen, and thanks to everyone who has contributed to this release, 1.0 is now on PyPI.

[graingert]

Amazing

[willkg]

Thank you to everyone who helped out! @gsnedders, @hugovk, @jayvdb, @jvanasco, and others!

Where do we go from here?

Work on #361 could improve maintenance activity and get some of the larger issues fixed. If you or your company depends on html5lib-python, seriously consider sponsoring development through donating money or donating your time.

Libraries that depend on html5lib-python should get updated to work with 1.0. Help us get to a world where we don't have to support pre-1.0 versions.

If you bump into any issues, please add them to the issue tracker preferably with example code showing the problems. I've got iffy availability for the next month and I think @gsnedders does, too, so we'll do the best we can, but additional help from you make it more likely things get fixed timely. As a side note, Bleach had one minor issue that I'll fix in the next couple of days, but otherwise works fine with 1.0.

That's it! Thank you again, everyone!