regression: rustdoc resolution changes

[Mark-Simulacrum]

bint-0.1.0: start v. end; cc @folkengine

Looks like resolution for the crate itself changed in some way, not sure.

[Mark-Simulacrum]

• lv2rs-core-0.3.0: start v. end; cc @Janonard
• oysterpack_testing-0.1.2: start v. end; cc @oysterpack (maybe this issue, couldn't test due to openssl v0.9)

[Mark-Simulacrum]

• test-generator-0.2.2: start v. end; cc @frehberg
• trimmer-0.3.6: start v. end; cc @tailhook

[pnkfelix]

triage: P-high.

[JOOpdenhoevel]

What does this issue mean? Do I as a crate owner have to worry about something?

[Mark-Simulacrum]

Presumably, the crates listed here are broken on current stable (1.36+) but this has not been verified. Someone will probably investigate soon and assign this to someone concrete to resolve.

[frehberg]

I will investigate my crate test-generator

[frehberg]

as for the test-generator, I tested various versions, cant see any error as in the log. There is no stdout for the error-case, any idea why it might fail?

[JOOpdenhoevel]

Btw, I found the problem with lv2rs-core and already fixed and updated it!

[nagisa]

Assigning to self to bisect the exact change which caused this regression, but it otherwise looks more like a rustdoc issue than a rustc one.

[pnkfelix]

hey @nagisa , do you think you will get around the bisection at some point? Or shall I assign to someone else?

[pnkfelix]

triage: reassigning to self.

[pnkfelix]

nominating for reassignment (want to get things that can be delegated off my plate)

[lqd]

I can repro on lv2rs-core, but looking at the logs, this was interesting:

Before:

Doc-tests bint
[INFO] [stdout] running 0 tests

After:

Doc-tests bint
[INFO] [stdout] running 1 test

This suggests a rustdoc change that detects more tests rather than resolution changes.

Started failing in 1.36.0.
Bisected to nightly-2019-04-24, and commit 0550766 which is an update to rustdoc's markdown parser.

In this case, I think the old tests did not compile, but that wasn't easy to see until this version of rustdoc actually started testing them.

[lqd]

Looking around similar issues, there's #61562 and this comment mentioning the same theory.

Puzzling, the code in the specific lv2rs-core doctest may have been ... correct ? cargo test still fails on 1.37.0 but passes on 1.38.0, or the latest nightly for example.

@pnkfelix what do you want to do with this issue ?

[pnkfelix]

@lqd it sounds to me like something we can either hand off to the T-rustdoc team, or simply close outright.

[lqd]

OK, I'll keep bisecting the other crates, and also check if the failures reproduce on recent stables/nightlies, to see if we need to open more specific issues, and/or hand this one to T-rustdoc.

[lqd]

Update:

• oysterpack_testing-0.1.2, reproduces on the beta-2019-05-30 tested in this crater run, but does not reproduce for the released 1.36 (or nightly), so it seems already fixed.
• test-generator-0.2.2, the setup seems complicated, there are proc_macros to help writing tests. These doctests rely on the unstable #[bench] feature, and was accepted e.g. on 1.35. It fails on beta-2019-05-30, works on 1.36 and 1.37, fails since 1.38 (and nightly) and is tracked in Tracking issue for future-incompatibility lint soft_unstable #64266. (It's interesting that it seems to be a lint but is actually deny).
  It's likely that the previous rustdoc didn't see these doctests in this case as well. I'll comment on the tracking issue.
• trimmer-0.3.6: works on 1.35, fails on this beta-2019-05-30, and works ever since 1.36, so it's likely close to the now-fixed issuelv2rs-core also had.

As a summary, it seems we have a mix of real rustc bugs, rustdoc effectively-breaking-changes-for-the-better testing new code, and invalid tests. The rustc bugs seem fixed, and the rustdoc behaviour change is #61562, and there might be miscommunication between rustdoc and rustc for the soft_unstable lint (or just its error message also having a future-compatibility lint flavor being confusing).

If need be, I could extract the doctests and bisect those to rustc commits, but I'm not sure we need to ?

[lqd]

As summarized earlier, this issue collects other issues, which were at the intersection of

• the rustdoc behaviour change from 0550766 and tracked in regression: doctest failure display-as-0.4.4 #61562
• and other rustc problems (possibly related to resolution) which have since been fixed, or are already tracked elsewhere (eg #[bench] accepted on stable).

While we could bisect those problems to look for possible missing test cases, it's likely to point at known issues since they were all fixed either in previous betas of the same crater run or 2-3 releases ago, it doesn't seem absolutely necessary to do so. So let's close this one.