Build: incompatible Mercurial versions from VCS setup than build commands

[ahal]

Details

• Read the Docs project URL: https://readthedocs.org/projects/taskcluster-taskgraph/
• Build URL (if applicable): https://readthedocs.org/projects/taskcluster-taskgraph/builds/16259866/
• Read the Docs username (if applicable): ahal

Expected Result

The builds should succeed :)

Actual Result

Builds fail as seen here. I don't believe anything changed on the repo side, this was working a week or two ago.

hg identify --id
abort: repository requires features unknown to this Mercurial: sparserevlog!
(see https://mercurial-scm.org/wiki/MissingRequirement for more information) 

[marmoute]

I only skimmed through the logs, but this is weird. It looks like the hg id call is suddently too old to be able to read the repository that just got cloned by that same hg.

So, might this is no longer the same hg and some of the virtualenv play in between downgraded it somehow ?

[ahal]

I don't have any insight into reathedocs infrastructure, though a co-worker pointed out a somewhat suspicious commit to Mercurial which would have been released a couple weeks before this started failing. But yeah, it's weird that the earlier hg commands aren't failing.

Edit: Apparently marmoute ruled out this commit in conversation with said co-worker. I'll leave this comment up just in case.

[marmoute]

The "suspicious commit" (and the related work in this area) only affects the "streaming clone" feature. Streaming clone does not seems to be involved in this build at all. So I would be surprised if these changes play any part in the current failures.

[mattip]

We are seeing this on the PyPy doc builds as well. Is there a way to find all the projects that are using mercurial to find one that is succeeding?

[mattip]

According to #8840 there are ~46 repos using mercurial. Are all their builds failing?

[humitos]

@ahal can you try using a Read the Docs config file with build.os (https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os) declared as Ubuntu 20.04?

[humitos]

I guess the problem here is that we are using a newer version in the hg clone and then an old version for the second hg identify command. I think this issue was introduced in #8930 when we started using the ubuntu-20.04 for the VCS steps.

I think you will be able to fix this problem by using Ubuntu 20.04 in your config file as I suggested in my previous comment.

It would be good to find a way to tell Mercurial to clone the repository to make it compatible with older versions.

[marmoute]

You can use --config format.sparse-revlog=no when cloning (or update the config of the thing doing the clone).
This problem will arise in the future as we turn format variant on by default overtime. Maybe Mercurial need some generic flag to request clone compatible with version X

[ahal]

@ahal can you try using a Read the Docs config file with build.os (https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os) declared as Ubuntu 20.04?

Thanks, this works around the issue for me.

[mattip]

Something is wrong. I cannot build the PyPy docs https://readthedocs.org/projects/pypy/builds/ for the last day: the build fails to get hg id.

[humitos]

@mattip for some reason hg tags is returning Command exited abnormally. I was able to reproduce this error locally when building your project with my Read the Docs development instance, but I don't understand why it's happening yet. However, I'm not able to reproduce this by running the commands manually.

[mattip]

Thanks for looking.

[marmoute]

Do you have any error or traceback? We could look at ?

[mattip]

@marmoute if you mean me, I only have the information in the build log https://readthedocs.org/projects/pypy/builds/16514248/

[humitos]

I've already found the issue and opened a PR to fix it at #9055 --It's a Read the Docs' bug, not related with Mercurial itself.

[ericholscher]

This is now deployed. Please let me know if this solves the issue.

[ericholscher]

Looks like pypy is working properly: https://readthedocs.org/projects/pypy/builds/16518308/