RFC: Relative link syntax for rustdoc

[JohnHeitmann]

Rendered

RFC Summary

The short version is that instead of this:

[`Ord`]: ../../std/cmp/trait.Ord.html

we should be able to write this:

[`Ord`]: ::/std/cmp/trait.Ord.html

Feedback Summary (through July 3)

• Strong sentiment that we should head straight for semantic links.
• Strong sentiment that the originally proposed spelling isn't a winner.
• A scheme-style link spelling like rust:std::cmp::Ord has a lot of support.
• ... even the CommonMark forums suggest using Markdown links with a custom scheme.
• Several suggestions to switch away from Markdown entirely, but the weight of opinion is that Markdown has won.

[killercup]

Nice RFC, I've wanted that feature for a long time. But like you mention in Drawbacks, this is a temporary fix. I don't like RFCs as temporary fixes.

Before adding band-aids, we should rather see how much needs to be done to refactor the necessary tools (rustdoc, in this case) and when the final CommonMark extension proposal is expected. Our RFCs usually have a few month of lifetime before being deployed :)

For me, this is not enough of an improvement:

The [`Ord`] trait is the grooviest trait there is!

[`Ord`]: ::/std/cmp/trait.Ord.html

We should aim for something like

The <std::cmp::Ord> trait is the grooviest trait there is!

And actually, if we were to specify a URI schema for rust items, we could introduce something like <rust://std::cmp::Ord> without waiting for CommonMark at all. (This has its own drawbacks, of course.)

---

FYI, there has been talk about switching from Hoedown to the pulldown-cmark crate.

[flying-sheep]

…and the markdown dialect begins to diverge. we’ll end up with somthing that resembles, but isn’t markdown.

i predicted that and proposed solving it properly by switching to ASCIIdoc or rST, but nobody listened 😢

[JohnHeitmann]

…and the markdown dialect begins to diverge.

My take on this is that it's not changing the markdown dialect, it's just adding a post-processing phase to links. It's closer to code pretty printing than something like a new table syntax.

[JohnHeitmann]

FYI, there has been talk about switching from Hoedown to the pulldown-cmark crate.

Context on that: rust-lang/rust#29329

Context on more sweeping markup changes: https://internals.rust-lang.org/t/rustdoc-restructuredtext-vs-markdown

[flying-sheep]

as i’ve said from the first minute rustdoc was announced: we want a naturally extensible markup format, NOT markdown.

what @killercup wants is e.g. trivially possible with rST’s text roles. :trait:std::cmp::Ord`` would give us a link with customizable prefix, and a trait CSS class, i.e. easily colorable and formattable.

[BurntSushi]

I think discussing the entire markup format used by rustdoc is out of scope for this RFC.

[flying-sheep]

yes, but this RFC is a symptom of the problem i outlined. @killercup’s idea on what more he wants to have further underlines this problem, as it is even more custom syntax.

AFAIK this is the first proposed markdown extension. i say we leave this can of worms closed.

[SimonSapin]

I’ll join other comments in preferring proper cross-referencing (whatever the syntax, even abusing links with something like [the Ord trait](rust:std::cmp::Ord) or [the Ord trait](rust:/std/cmp/Ord)).

But if we go with hacks on HTTP(S) URLs, we could have rustdoc rewrite "host-relative" ones (those that start with a single /) to be relative to the root of the documentation. (E.g. https://doc.rust-lang.org/stable/ for https://doc.rust-lang.org/stable/std/cmp/trait.Ord.html)

[louy2]

@flying-sheep Kinda feeling your frustration, especially when rST was specifically invented to handle Python documentation. That said, can you link to your RFC?

Recommend read: Common markup for Markdown and reStructuredText

[louy2]

Quoting @gankro:

For those showing up these days:
This thread is super old. Like pre-1.0 when breaking all the documentation was sort of maybe a possibility. 1.0 has landed, we have thousands of pages of documentation using markdown.
It's done. Markdown won.
Any shiny features we want will almost certainly be grafted on top of markdown, hopefully using the very nice API provided by https://github.com/google/pulldown-cmark59
If you're really interested in improving rustdoc, you should ping @alexcrichton about what needs to be done to land pulldown-cmark in-tree.

[JohnHeitmann]

I'll take a stab at designing out the rust: link idea to see how it compares. Two questions occur to me off the bat that I'd happily take early help on:

First, what do we do when someone runs cargo doc --no-deps? Why do people run --no-deps anyway?

Second, we'll still need syntax for non-items. Fragments are probably easy, just use rust:std::cmp::Ord#fragment, but what about non-item docs? It feels like this comes back around to my original proposal.

[JohnHeitmann]

But if we go with hacks on HTTP(S) URLs, we could have rustdoc rewrite "host-relative" ones (those that start with a single /) to be relative to the root of the documentation. (E.g. https://doc.rust-lang.org/stable/ for https://doc.rust-lang.org/stable/std/cmp/trait.Ord.html)

I like that idea. It transforms what would today be allowed, naive, and wrong into slightly surprising but correct.

Folks who thumbs-up'ed @SimonSapin 's comment would you mind voting here if this was an idea you liked? It's not clear which of the two ideas you were nodding at.

[joshtriplett]

I'd like to see the "rust:" URL scheme as well. This would avoid introducing an extension to markdown, and would allow the use of standard Rust syntax (e.g. rust:std::io::stdout), without mention of .html or similar. That would also make it easier to produce documentation that doesn't include .html in its links, and instead just uses bare paths (e.g. https://doc.rust-lang.org/std/io/fn.stdout instead of https://doc.rust-lang.org/std/io/fn.stdout.html).

That would also make inline links like rust:std::io::stdout easier.

[flying-sheep]

That said, can you link to your RFC?

there is none, due to @gankro’s reply quoted by @louy2. i figured that i need them to at least tell me that the idea has a chance.

but @louy2 you forgot to also quote my reply, which IMHO invalidates @gankro’s point:

nah, converting it all isn’t a problem thanks to pandoc. markdown’s functionality is a straight subset of [reStructuredText], and pandoc supports all of markdown so i doubt very strongly that there will be issues.

[…]

i propose the following:

1. make rustdoc do rST and/or asciidoc as well. both are extensible by default (directives&roles/macros) and therefore adequate
2. add a Cargo.toml field like doc-format
3. emit non-markdown (rST or asciidoc) by default. everyone choosing to keep using markdown just has to add the doc-format key.

[…]

[steveklabnik]

Here are my priorities here:

1. Not further fragmenting the markdown ecosystem by having "special" markdown syntax that's just our own.
2. Making such linkage as easy as possible for doc authors.
3. Making links in a way that allows us to refactor rustdoc's URL structure without breaking the universe

As such, I'm 👎 on the RFC as written, as it violates 1, and arguably 2. (It's not an improvement on 3, but it's also not worse than the status quo.)

The URL idea helps mitigate 1, helps a lot with 3, and isn't bad on 2. But it does mean that some sort of post-processing tool needs to happen to convert our URL scheme into HTTP URLs, which is kind of a violation of the spirit of 1, if not the rule.

Historically, my position has been "Let's wait until CommonMark has an extensibility mechanism, and then use it," but it does seem like that might still take some time, and I'm very sympathetic to the pain here today. If we want people to write docs, we need to make it easy; that was the whole reason for Markdown in the first place!

I've posted on the commonmark forums to see what they recommend.

[nagisa]

I feel like adopting anything that’s not as good as rST’s text roles will dig us into a hole we cannot get out from. xrefs are probably the most obvious thing that does not fit well to Markdown that would be easily solved by roles.

We avoided problem of deprecated/stable/unstable annotation by having language features for them, however that only covers a very limited set of annotations and you cannot annotate different things this way (e.g. version in which the API gets removed). I feel like many more cases will crop up over time for which we won’t have any solution if we adopt anything less than fully extensible version of markdown (i.e. the ::ext thing from some link in Steve’s post) and allow people to add their own ::things.

That being said xreferencing is important and thus what I’d propose is to add support for the rust: url scheme under a (possibly unstable) flag and then wait for an extensible CM.

[JohnHeitmann]

In this discussion we're ignore non-type relative links like book links. If we're happy saying either "use absolute links for them", or "don't link to them from re-namespaced types", or even "there's not enough to care", then that's fine. Otherwise, it's a corner of the problem that needs a little help even if we solve type linking perfectly.

[nagisa]

That is also something that is sufficiently solved by either using a new URI schema (rust-book:) or extensible markdown.

[JohnHeitmann]

That is also something that is sufficiently solved by either using a new URI schema (rust-book:) or extensible markdown.

Maybe that is what this RFC should be narrowed to focus on.

Step 1: Design and implement a link scheme: with the intention it lives forever as a mechanism to relative link to in-tree non-rustdoc documentation from rustdoc documentation.
Step 2: Abuse that scheme to work around today's type linking problems
Step 3: Design and implement true rustdoc cross-links
Step 4: Undo the abuse of Step 2 with a script

That makes the pessimistic assumption that Step 3 is going to take a while. If it looks like there's a smooth path towards true cross-links available to us today, then we can skip the abuse in Step 2.

To make my position clear since I'm pushing back a bit: I love the idea of true type links, and I'm happy to spend time making them happen, but there's a pragmatic path to get some nice doc improvements out to rust users that doesn't need to wait.

[llogiq]

How about auto-generating links to all items in module scope (which is what the commonmark people suggest, incidentially)? Admittedly, that list would become rather long & we'd need to weed out things that were already linked manually, but if [Ord] automagically was linked to https://doc.rust-lang.org/std/cmp/trait.Ord.html (it would be in scope anyway, as it's in the prelude), we'd make linking to stuff so much easier.

[joshtriplett]

@llogiq That's a nice idea.

[llogiq]

If we used pulldown-cmarks, we could simply prepopulate the links HashMap. Or we could even ask them to allow us to specify the map type and use a specialized ScopeMap that will lazily compute the links whenever the map doesn't have a key.

[durka]

@steveklabnik given that the current system for linking is "know rustdoc's URL structure by heart" your point 3 means we really do need some system for this.

[quodlibetor]

To address the "don't fragment the markdown ecosystem": If we want to make a rust URL scheme correct and generic across implementations all that really needs to be done is allow "absolute" links -- where links must include the crate name -- and publish a relative mapping from rust links to http links inside of any given documentation tree. Then anybody could link to arbitrary rust urls if they have some metadata about where to collect their required indexes. This would be a whole lot easier if crates.io automatically published docs.

AKA do what intersphinx does for Python/rST.

[steveklabnik]

This would be a whole lot easier if crates.io automatically published docs.

crates.fyi is making progress on this.

[jan-hudec]

@steveklabnik,

1. Making such linkage as easy as possible for doc authors.

Well, the easiest, for doc authors, would automatic linking like doxygen does, i.e. every word followed by (), containing :: or containing uppercase letters (for types) is automatically made into a link if there is corresponding target.

Would that be a special syntax to you to conflict with the first point or not? Whatever is done will be specific to rustdoc anyway. So the point is just to make it not conflict with future improvements to commonmark.

Admittedly, that list would become rather long & we'd need to weed out things that were already linked manually, but if [Ord] …

[Ord] — all items are implicitly available as references — or <Ord> — resolvable item name is considered a valid link?

I would, perhaps, argue the later makes more sense, because that's how linking to arbitrary targets works in generic commonmark. Either way, one of those, or fully automated linking like doxygen does, should clearly clearly the goal.

Advantage of explicit request with [] or <> is that it can be an error if the target is not found. For bare words that would be more problematic.

Or we could even ask them to allow us to specify the map type and use a specialized ScopeMap that will lazily compute the links whenever the map doesn't have a key.

… or adding a callback function. In any case lazy seems better option. Not all code uses its dependencies, so linking needs to work on absolute paths too and that list would be rather large to copy around all the time.

[llogiq]

@jan-hudec I think commonmark prefers [..] to ≪..> The latter also risks clashes with valid HTML, which creates ambiguities.

Apart from that, I fully agree that using lazy lookup would be preferable.

[nrc]

I'm roughly behind the suggestion of having Rust URLs to handle cross-referencing. However, then adding further custom URLs for the book (and presumable other resource in the future) makes me nervous.

Could someone explain in more detail how the suggestion for auto-generating a link for every item would work? I'm not sure I understand the implications nor how it differs from other proposals.

I think a pre- or post-processing step for URLs on top of markdown processing is totally fine and seems like a minimally invasive way to extend it to our needs until a proper extensibility mechanism is added. It also makes automatically changing to a new system in the future fairly easy (c.f., adding our own extensions to Markdown).

I'd like to see a proper RFC or pre-RFC or whatever for Rust URLs, there are a number of issues that are not trivial to decide upon:

• type vs value (and macro, in the future) name spaces
• case sensitivity
• relative paths
• aliases via use ... as and type
• do we ever need to refer to items within functions or to impls? (Probably not for Rustdoc, but perhaps in the future if we want to use the same URL scheme for other purposes).

However, I would like to see this, currently Rustdoc URLs are a hell-ish mess.

Finally, is there consensus that the motivation for this RFC is best addressed in other ways or are some still in favour of the current proposal?

[joshtriplett]

Could someone explain in more detail how the suggestion for auto-generating a link for every item would work? I'm not sure I understand the implications nor how it differs from other proposals.

Markdown handles a "reference" style of links, written like this: [Ord][]. (Or [with different link text][Ord].) Normally, those require adding the [Ord]: url somewhere below. Rustdoc could supply those URL references for all identifiers in scope, and then rustdoc documentation could reference them without including the URL reference itself. That would mean any documentation for code that has Ord in scope could just write [Ord][].

Note that Markdown does not treat those reference names as case-sensitive, so if you have both Foo and foo, standard Markdown processors will not distinguish them. Also, this would potentially conflict with reference links provided by the user.

The other alternative would be to write <rust:Ord>.

Either syntax eliminates the .. and .html from existing Rust links, and would make it easier to substitute external links to documentation hosted elsewhere.

[JohnHeitmann]

I'd like to see a proper RFC or pre-RFC or whatever for Rust URLs, there are a number of issues that are not trivial to decide upon:

What's the best way to float a pre-RFC? A post to internals.rust-lang.org?

[llogiq]

@JohnHeitmann Been there, done that

[nrc]

@JohnHeitmann yes. Or an issue on this repo (seems to be a bit hit and miss which gets better visibility/discussion).

[nrc]

@JohnHeitmann and @llogiq I was talking specifically about URLs, if there is enough discussion in that discuss thread, then going straight to RFC would be fine, but it might be worth discussing some of the finer points about URLs ahead of time.

[JohnHeitmann]

@nrc Let's postpone or close this RFC. I think there will be a few narrow problems left behind by rust:<rust thing> style links and something like this RFC may address those problems, but we should design out the proper primary link style first and revisit this afterwards if there's a demonstrated need.

I'm working on two things in parallel: an RFC for rust links that take the bigger steps everyone is pushing for in this feedback, and a prototype to validate that I'm not missing novice problems. I'll re-start discussion for that new RFC at internals.rust-lang.org on ~Friday once I've gotten far enough prototyping to make sure I'm not going to embarrass myself with glaring design holes.

If you want I can overwrite this RFC, but I'd slightly prefer to start fresh.

[nrc]

@JohnHeitmann sounds good. I've been doing some refactoring to Rustfmt to make the existing URLs better, sounds like we might overlap somewhat? Could you ping me on irc so we can discuss our plans?

[steveklabnik]

I would also like to point out that the markdown thread has a fantastic response posted to it.

On Jul 6, 2016, 15:54 -0400, Nick Cameronnotifications@github.com, wrote:

@JohnHeitmann(https://github.com/JohnHeitmann)sounds good. I've been doing some refactoring to Rustfmt to make the existing URLs better, sounds like we might overlap somewhat? Could you ping me on irc so we can discuss our plans?

—
You are receiving this because you were mentioned.
Reply to this email directly,view it on GitHub(#1661 (comment)), ormute the thread(https://github.com/notifications/unsubscribe/AABsiviQT7rd3iqZIn-4xfjFU3sNAzkNks5qTAf6gaJpZM4I_YSK).

[brson]

I’ll join other comments in preferring proper cross-referencing (whatever the syntax, even abusing links with something like the Ord trait or the Ord trait)

Something like this (particularly the former) seems right to me. Personally, I always anticipated we would need to extend markdown like this for our own purposes.

[alexcrichton]

The tools team discussed this recently and we concluded to close in line with @JohnHeitmann's comment.

@JohnHeitmann if you need any help starting a new RFC though @nrc is more than willing to help, so feel free to reach out to him (or me)!

[matthew-piziak]

What is the status of the replacement RFC? The deficiencies in the current linking system are currently my biggest pain point when writing docs.

[matthew-piziak]

I'm not the most qualified but if a replacement RFC isn't in the works then I'm happy to take a crack at it.