Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doctests #5

Open
epage opened this issue Feb 7, 2024 · 18 comments
Open

doctests #5

epage opened this issue Feb 7, 2024 · 18 comments
Labels
C-tracking-issue Category: A tracking issue for something unstable. S-needs-team-input Status: Needs input from team on whether/how to proceed.

Comments

@epage
Copy link

epage commented Feb 7, 2024

We talked about doctests being an area for improvement

@epage epage added C-tracking-issue Category: A tracking issue for something unstable. S-needs-team-input Status: Needs input from team on whether/how to proceed. labels Feb 7, 2024
@weihanglo
Copy link
Member

weihanglo commented Feb 8, 2024

rustdoc is responsible for both compiling doc examples and executing them as tests. Cargo current has no way to control the behavior. There are a handful of issues around this:

Bonus

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

@epage
Copy link
Author

epage commented Feb 8, 2024

If we can put as many doctests into a single binary as possible, that would also improve compilation times

We also want to make it so its as natural to get coverage for doctests as other tests.

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

That is the direction I would want to go as well.

@weihanglo
Copy link
Member

Yeah we can push it further.

rustdoc, as a doc comments analyzer, provides source information via a flag. Cargo invokes rustdoc with the flag, so it knows where the source is and how to compile and test it.

@GuillaumeGomez
Copy link
Member

I'm very fine with this approach as it would remove code from rustdoc. Problem is that the code examples originally were written to allow crate writers to add code examples for their users, meaning they needed to import the crate as if it was a dependency. I think this approach is still good, but should not be the only one (for example we can't run doctest in binary crates, which is bad). If cargo can solve both issues, that'd be awesome. :)

@epage
Copy link
Author

epage commented Mar 26, 2024

Crazy idea: could we change the compilation of #[doc] to generate #[test] fns for each code fence.

  • If the test lib builds against the regular lib, then the import names still work
  • Otherwise, the doctests would be usable on private items, having access to everything where they are located.

Challenges

  • The compiler knowing enough about doctests to do this
  • The existence of compile tests (maybe an unstable feature to assert on compilation state?)

@GuillaumeGomez
Copy link
Member

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails. More information here.

@BurntSushi
Copy link
Member

Would it be possible to resolve how the doctest should be built with a new attribute? Maybe something like internal to opt into "compile this test as a normal unit test that uses crate internal APIs."

@epage
Copy link
Author

epage commented Mar 27, 2024

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails

Yeah, I was aware of and called out compile_fail. Being able to set the edition is an odd one. Where can I read up more on the use case for that? As I don't personally have a use case for it, my automatic reaction is "can this be deprecated?" but I recognize there are likely cases I'm not aware of.

Likely switching to this style of doctests would require an opt-in with it being the default in a new edition, so we could get away with changing things.

@epage
Copy link
Author

epage commented Apr 23, 2024

btw rust-lang/rust#123974 is a nice improvement to performance.

@GuillaumeGomez
Copy link
Member

It's not finished yet. A few things need to be added:

  • only enable it starting 2024 edition
  • remove chunking to run all tests at once (and improve even further the performance improvement) and potentially fix libtest being blocked

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

@epage
Copy link
Author

epage commented Apr 24, 2024

only enable it starting 2024 edition

Sounds like this is because the tests were building and running as separate processes and now they are being combined, causing changes in behavior through any interactions with globals?

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

Could you clarify what you mean here?

Overall, I'm excited by these changes. The changes in performance will hopefully mean people will stop avoiding doctests. That said, I still feel like we need to do something to make these less special. The more we align with how things are normally built and run, the easier it is to take part in a more unified CLI, reporting, and coverage tracking.

@GuillaumeGomez
Copy link
Member

Sounds like this is because the tests were building and running as separate processes and now they are being combined, causing changes in behavior through any interactions with globals?

Exactly! Should have precised. That's also why we'll need to add a new code attribute (named standalone or something along the line).

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.

Could you clarify what you mean here?

Absolutely: if your crate is a binary, then I plan to apply the same transformation to doctests as we do for #[test] tests (as described here). It's kinda what I already to merge all compatible doctests, but this time I would do it so they are generated in the crate source code directly, allowing users to test their internal code (which they can't currently as doctests are currently using the crate as dependency).

Don't hesitate if my explanations are not clear enough.

@epage
Copy link
Author

epage commented Apr 26, 2024

Absolutely: if your crate is a binary, then I plan to apply the same transformation to doctests as we do for #[test] tests (as described here). It's kinda what I already to merge all compatible doctests, but this time I would do it so they are generated in the crate source code directly, allowing users to test their internal code (which they can't currently as doctests are currently using the crate as dependency).

Is this a variant of the idea mentioned earlier of generating #[test]s for binaries or something different? What is the path looking like for this?

@GuillaumeGomez
Copy link
Member

Is this a variant of the idea mentioned earlier of generating #[test]s for binaries or something different?

Not that I know.

What is the path looking like for this?

Me doing it. First I want to finish the first PR (the speed up of doctests by merging them).

@epage
Copy link
Author

epage commented Aug 19, 2024

FYI @GuillaumeGomez posted a status update on bins: rust-lang/rust#50784 (comment)

@epage
Copy link
Author

epage commented Sep 3, 2024

For overriding file/line, lang is interested, it just needs someone to champion and workout corner cases, particularly if we also support column. We need to harvest Internals for feedback and respond.

e.g. https://internals.rust-lang.org/t/pre-rfc-enable-setting-of-source-file-line-column/19156

One idea: punt on column and code-generate to the right column

@epage
Copy link
Author

epage commented Sep 24, 2024

@GuillaumeGomez in your update at rust-lang/rust#50784 (comment) you mention either having rustc translate doc-comments into #[test]s or generating the code for it. The former aligns with some of my ideas above for un-specializing doctests but only for tests for unit_test attributes. The latter doesn't immediately help the un-specializing. iirc you are starting your experiments without involving rustc (if nothing else, its much easier to prototype in only a single code base).

The part I don't remember is how much we talked about rustdoc generating, compiling, and running this code or only generating it and having cargo compile and run the code. The latter would help us un-specialize doctests.

@epage
Copy link
Author

epage commented Sep 24, 2024

Doesn't have to be in the first release but an interesting spin off from that is if rustdoc emits json messages for these generated files, some of those could include that they are compile_fail and cargo could do the compile-fail part.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-tracking-issue Category: A tracking issue for something unstable. S-needs-team-input Status: Needs input from team on whether/how to proceed.
Projects
None yet
Development

No branches or pull requests

4 participants