proposal: cmd/doc: module documentation

[carnott-snap]

What did you see today?

Currently there is no place to put module level documentation that is consumable via the toolchain.

What would you like to see?

We should standardise a doc string format that reads like the // Package xxx ... syntax. This will follow the current godoc rules and allow for code blocks, etc.:

// Module golang.org/x/sys provides OS specific interfaces for low level operations.
//
// ...
module golang.org/x/sys

go 1.13

We should add a new -m flag to put go doc in module mode, much like go list -m vs go list. This will only accept module import paths not packages, again like go list, and will display the module doc string, and full package listing.

$ go doc -m golang.org/x/sys
module golang.org/x/sys

Module golang.org/x/sys provides OS specific interfaces for low level operations.

...

package cpu      // import "golang.org/x/sys/cpu"
package plan9    // import "golang.org/x/sys/plan9"
package unix     // import "golang.org/x/sys/unix"
package main     // import "golang.org/x/sys/unix/linux"
package windows  // import "golang.org/x/sys/windows"
package main     // import "golang.org/x/sys/windows/mkwinsyscall"
package svc      // import "golang.org/x/sys/windows/svc"
package debug    // import "golang.org/x/sys/windows/svc/debug"
package eventlog // import "golang.org/x/sys/windows/svc/eventlog"
package main     // import "golang.org/x/sys/windows/svc/example"
package mgr      // import "golang.org/x/sys/windows/svc/mgr"
package registry // import "golang.org/x/sys/windows/svc/registry"

What alternatives did you consider?

README

The forthcoming discovery service, #33654, currently uses the repo's README for top level documentation, since this is a fairly strong pseudo-convention for most repos. Unfortunately, while this works well for a web ui, this syntax may not be as readable on the command line, or format well with godoc.

versions

It may be useful to list all known versions, but this implies that the list is up to date, and I would rather not impose a full go get flow. This could be gated behind -all or a new flag.

packages

noise

For complex packages, say k8s.io/kubernetes, listing all the packages may be really noisy. As such it may be useful to gate this data behind -all. That being said many protobuf generated packages suffer from the same noise.

stutter

The current syntax also suffers from stutter, because all package lines start package xxx // import .... This format borrows from the package declaration syntax to be more legible to readers. It also feels like the type declarations that can be found in regular godoc:

package unix // import "internal/syscall/unix"

const AT_REMOVEDIR = 0x200
const AT_SYMLINK_NOFOLLOW = 0x100

submodules

While they are not contained within the module, many customers may be confused about why they cannot find the cloud.google.com/go/datastore package within the cloud.google.com/go module. This could be displayed like so, in addition to being guarded behind -all or a custom flag:

module cloud.google.com/go

package cloud // import "cloud.google.com/go"
package asset // import "cloud.google.com/go/asset/apiv1"
...
module cloud.google.com/go/bigquery
module cloud.google.com/go/bigtable
...

[jayconrod]

cc @dmitshur

[dmitshur]

Thanks for making this proposal.

I have considered this problem space in the past and wrote a proposal draft related to "module comments", but upon review and thinking more about it, I've realized there were reasons not to propose the change at that time.

There may be useful and relevant information in my proposal draft, so I've posted it so that it is available for reading. See issue #35546.

[carnott-snap]

I created this proposal to resolve a question I had from the new pkg.go.dev site:

Should we use the README on the module Overview page?

I felt like for a web ui, use of README was an understandable fallback, but only worked for web. This proposed improvement attempts to provide an interface agnostic answer to these user questions: "what does this module do?" and "where is the injection point?".

READMEs are a useful convention that github has built up, but they tend to read like manuals, and are highly unstructured. I could not find any better place to store the package author's response to "what does this module do?", let alone display it with the Go toolchain. Furthermore, many repos have top level doc.go files that create a documentation package at the root of a module without any symbols, something that strikes me as an anti pattern.

In order to revive this proposal, there needs to be an answer to the question of "why document a Go module."
I would propose the following divide: Packages document how they should be used, modules document why they should be used. When evaluating a module for use one should look to the module documentation, when developing one should look to the package documentation.

Currently package docs frequently do both, especially at the repo root. With this, a package would document how the exported symbols are to be used, which ones are important, maybe some code samples, and relationships with other packages. E.g. cloud.google.com/go/iam.

A module should document why you should use the features within. It should describe what it was built to do, call out important packages and note shared features/concern between packages. E.g. cloud.google.com/go.

This works better for complex libraries that have many packages, like cloud.google.com/go, but I think a short, human readable description of a module could be a boon, even if it only contains a single package. One optimisation would be to have the module docs fallback to the package docs if they do not exist.

The other feature is the ability to list out the packages in a module. There may be a way to do this with go list, but I cannot make it work, and I think that discovery should be as easy as go doc -m example.com/path/to/module.

[rsc]

Go programs import packages. It makes sense to ask for the documentation for a package.
A module is just a group of packages that happen to be versioned together. The module concept supports the package concept, not the other way around.

It is a mistake to place emphasis on the module. Focus on the package. Especially for a single-package module, the documentation has always been on the package and it should remain there. A large module like cloud.google.com/go may benefit from a brief overview in a README, and that seems fine.

I'm reluctant to formalize module documentation any more than that.

[carnott-snap]

A module is just a group of packages that happen to be versioned together.

My goal is to standardize a location to document why they are versioned together, and interface(s) to consume this data. This is frequently information that I want to extract from a module, without cloning or poking around in ~go/pkg/mod/....

A large module like cloud.google.com/go may benefit from a brief overview in a README, and that seems fine.

What is your view on empty doc.go packages?

I'm reluctant to formalize module documentation any more than that.

pkg.go.dev is formalizing this, and in a way that is not very compatible with CLI tools. Is the path that the language would like to take? I am a very large fan of go doc being a first class interface.

[carnott-snap]

Is any additional information required, or is this proposal on Hold, but not correctly labeled? It appears to have missed the past few proposal review meetings: #33502.

[rsc]

It's still very unclear what it means to "document a module". This is why we didn't move forward with #35546.

I understand your point that pkg.go.dev is making this unclear as well. (For example, https://pkg.go.dev/golang.org/x/crypto/ssh/terminal?tab=overview shows the module-level README but isn't really about the terminal package despite the framing.) It's all still up in the air.

I'm not sure there's more information needed or available right now to move forward, but I've moved this into the active proposal list so we keep looking at it.

[rsc]

Moving it into the active proposal list did mean we kept looking at it, but I still don't see that there's much to do here. Even if we added a module comment like the original suggestion, it would be awkward to have to run go doc -m to see it. (Most directories with a go.mod also have a package.)

Putting on hold until we have a better idea of what any of this means.

[carnott-snap]

Makes sense; the impetus for this proposal is the recently released pkg section of the discovery site. I think some decision about module documentation should be made before that site hits GA.

That being said, I agree nobody is hankering for this information via go doc; though, I would appreciate a cli interface for this data, since I prefer those to a web ui.