Planning for hosting documentation archives for all packages

[daveverwer]

I’ve done some very preliminary tests with DocC on how we might approach mass-generating documentation for all Swift packages in the index.

So, taking a docbuild command like this:

xcrun xcodebuild docbuild -scheme Lindenmayer-Package -derivedDataPath ~/Desktop/Lindenmayer-DocC -destination platform=macOS, arch=arm64

We end up with a Build directory that includes several doccarchive files. In the case of the Lindenmayer package, we have five:

• Lindenmayer.doccarchive
• LindenmayerViews.doccarchive
• MeshGenerator.doccarchive
• SceneKitDebugTools.doccarchive
• Squirrel3.doccarchive

I also don’t fully understand why we need the -destination platform=macOS, arch=arm64 parameter, but we do. This needs more investigation.

We also need to make use of these enhancements coming with Swift 5.6. They are already in the open-source DocC project and after a quick chat with @macdevnet he also advised we could potentially combine the documentation build process with the conversion to a static site and do everything in one step. For now, we have this command which should convert a single doccarchive:

swift run docc process-archive transform-for-static-hosting ~/Desktop/Lindenmayer-DocC/Build/Products/Debug/Lindenmayer.doccarchive --hosting-base-path some/path/Lindenmayer --output-path ~/Desktop/Lindenmayer-HTML/

To run this, you also need to set an environment variable DOCC_HTML_DIR to the template generated as the output from this project. I haven’t managed to get this working yet but @macdevnet has been extremely helpful with figuring out what we have so far.

What packages should we generate documentation for?

I was half hoping that a package without any documentation comments would cause docs or docbuild to fail so we’d know which packages had any docs at all.

Taking a package like the trusty old LeftPad the docbuild command gives a triumphant ** BUILD DOCUMENTATION SUCCEEDED ** and generates a valid DocC bundle, but of course that bundle is effectively empty:

Maybe that’s OK, and we should host that type of documentation in the hope it will inspire people to start adding documentation comments as a first step. That said. If we launch this feature with a huge set of mostly blank documentation, it won’t look great.

Hosting

Ideally, we would host package documentation under the SPI package URL. For example https://swiftpackageindex.com/daveverwer/LeftPad/documentation.

There are some significant advantages to this approach, including having navigation and context as people browse package documentation. If someone ends up in some hosted documentation via a Google search (which is likely), they’ll easily be able to navigate to package metadata if we have a site header and navigation back to a package page available. It’s likely that if we manage to get the static transform mentioned above working that we could host the docs inside an iframe element, similar to what we do with README files.

Other notes

We will probably want to:

• Store the logs from the xcodebuild docbuild command.
• Store the output of the transformed documentation archive so we can access it from within the Vapor app.
• We may also want to think about how we might index some of this information for potentially including in search results later.

Next Steps

• Investigate the implications and effect of the destination parameter.
• Investigate whether we need to process all doccarchive files in the Build directory for a package and how the output of each relates.
• Get a static documentation build working using the open-source docc tool.
• Decide whether to host blank documentation sets, and if not, how to detect them.

[Sherlouk]

This is super exciting.

A couple initial thoughts/questions/etc

• Would we have a https://swiftpackageindex.com/daveverwer/LeftPad/documentation?download=true endpoint which allows users to download an archive to explore locally?
  • Could/would we inject a "Generated by Swift Package Index" notice into the documentation, if downloaded?
• Would we add a new value to the SPI.yml to allow folks to choose which scheme is used to generate documentation? or is it generally accepted that the default package scheme will work universally for documentation (just not builds).
• Would we (and if so, what) information would we show on the package page with regards to documentation?
  • Could we add a documentation percentage?
  • Could we improve the search score if the package is well documented and not empty?
• Could search include results from documentation?
• Does DocC only include functions, variables, etc that are documented?
  • This links onto the whether or not we should host blank documentation as if the docs at least includes a list of the functions (especially the public API) then that could be valuable.
• Is DocC public interface only by default?

[franklinsch]

Hi @daveverwer, @Sherlouk! This is really exciting work indeed.

Regarding the build workflow you’re proposing, note that you can use the OTHER_DOCC_FLAGS build setting to specify additional parameters to docc during the build. If you want to generate a documentation archive that’s suitable for hosting in static environments, you’ll want to set OTHER_DOCC_FLAGS=--transform-for-static-hosting --hosting-base-path [hosting-base-path] (note that this will require building with Xcode 13.3). You will find more info here: https://apple.github.io/swift-docc-plugin/documentation/swiftdoccplugin/publishing-to-github-pages/.

Regarding other questions you might have regarding Swift-DocC, I would love for us to discuss further in the Swift Forums where the Swift-DocC community can pitch in. Cheers!

[daveverwer]

So this thread has not been kept up to date (but you should see the DM conversation between Sven and me on this topic! 😂), but here's where we are and what we're going to ship soon:

• Generated documentation from the project's default branch, as configured in a .spi.yml file, published to a URL that looks something like the one below.
• Proxying through the SPI Vapor infrastructure so we can add some navigation (and for a couple of other reasons listed below).
• Multiple documentation target support.
• Links from the project's package page to each documentation target.

https://staging.swiftpackageindex.com/SwiftPackageIndex/SemanticVersion/main/documentation/semanticversion

This is a "v1" release of the feature and is the entire scope of what we'll make public in the next couple of weeks. But I also wanted to detail what we've discussed about a more complete version of this feature:

We want to prioritise tagged release documentation rather than default branch documentation. URLs are more stable for tagged releases, and it's much clearer what is actually being documented if it's built from a release. So, we will make the following changes after shipping the v1 feature.

• Generate documentation for every tagged version and potentially for n-1 major or minor (TBD) releases, too.
• Change the documentation links on package pages to point to the latest stable release's documentation rather than the default branch. This will need a new layout for the links, potentially merged with the "Versions" section of the sidebar. Design TBD.
• Add redirects between outdated minor releases. So when 1.2.3 of a package replaces 1.2.2, documentation URLs that are in the wild that pointed to 1.2.2's reference automatically update to 1.2.3's reference.
• On generated DocC documentation pages that are not the latest stable release, add extra metadata inside the SPI header that points out that people are not looking at the latest version of documentation, with a link to the latest stable version. Also, when viewing default branch documentation, it will guide people towards the latest stable using the same area.
• Include the date of the release of the documentation being viewed in the SPI header.

Key points:

• URLs to DocC documentation will always be stable and permanent but may redirect people to a later patch version of the release they're looking at.
• Default branch documentation will be de-emphasised in favour of documentation for the latest stable release.
• It will always be clear if people are not viewing the documentation for the latest stable branch, and there will always be a prominent link to the latest docs.

[daveverwer]

URLs to DocC documentation will always be stable and permanent but may redirect people to a later patch version of the release they're looking at.

Well "always" up until the point that package authors delete version tags in their repository, but that's a bigger issue more related to SwiftPM than documentation generation.