Consider using AsciiDoc instead of Markdown for F Prime documentation

[LeStarch]

Pros

• For anything Markdown can do, the AsciiDoc syntax is similar.
• AsciiDoc is fully supported in GitHub READMEs and wikis.
• AsciiDoc does some common things better than Markdown, e.g., multilevel outline lists and tables.
• There is a tool (asciidoctor) for locally rendering AsciiDoc source to HTML and PDF. There is no such tool for GitHub-flavored Markdown. You have to ask GitHub to do the rendering. In practice, this means pushing to GitHub to see the rendered version.
• AsciiDoc has many features that Markdown completely lacks, e.g., auto-numbered sections. In Markdown you have to add section numbers manually (tedious and error-prone) or write a script in Python or Awk.

Cons

• Markdown is familiar to many users. E.g., it's the default on GitHub (but not the only choice) for README and wiki pages.
• Markdown is supposed to be more readable. See https://github.github.com/gfm/#what-is-markdown-. However, this point seems debatable. Since GitHub-flavored markdown is just one of many flavors, and not a universal standard, standard text editors such as vi do not highlight it correctly, impeding readability.

Alternatives

ReStructured Text is another popular choice for documentation. It’s similar to AsciiDoc, with similar pros and cons vs. Markdown. It’s widely used in the Python community.

[LeStarch]

@bocchino here you are

[LeStarch]

I'd like to note, we do feed Markdown formatted SDDs into our Doxygen documentation for completeness of the Code/C++ reference. We should ensure Doxygen can pull in asciidoc.

[bocchino]

Here is some (rather old) discussion of the topic: https://discuss.asciidoctor.org/Doxigen-support-td1359.html.

[SterlingPeet]

@LeStarch, this is the call for comments item that compelled me to comment 😄

When @zeroping and I were on lab in the spring of 2019, I believe we had a discussion with @bocchino about this very topic. Specifically, the desire to have a single source of documentation that was flexible enough to provide things like aforementioned auto-numbered sections, but not so complicated to require a specialist to write it (i'm looking at you LaTeX 😛 ).

At the time, I used Pandoc to convert the MS Word formatted Users Guide to RestructuredText, and then rendered it to a local paginated html folder and a PDF. The existing PDF didn't have any bookmarks, so I was ultimately taking a long round-trip to get a document I already had albeit in a reasonably navigable format.

The one Pro that I will add to the above list, is that this project already has a fair amount of Python code, and that could put slight favor towards utilizing a rich text format that is uniform across the multi-language project.

Personally, I do relatively heavy Python development. My Python projects are all documented in RST because it is both the prevailing defacto standard and the Sphinx rendering engine is nicely integrated/interoperable with Python development tools. I use RST for my F Prime tools like fprime-extras and cookiecutter-fprime-deployment. We even use RST at GT for some of the mission specific README pages and SDD/RDD documents already. So, I would find it convenient to remember less rich text syntaxes.

That said, Asciidoc is just as good when you look at the main concerns. It does have more expressive syntax, it is not subject to "flavored renderers", and it does have the tooling to produce multiple output formats from the same source. So, I will be happy with either choice.

[LeStarch]

This is a good note on Python. We do have a non-negligible amount of python and are already using sphinx in parts of our documentation. Switching to ReStrctured Text would bring that our Python and other documentation together.

[timcanham]

My main concern is that the documentation be renderable on GitHub so users can easily browse the documentation. It sounds like either AsciiDoc or RST does that. If we pick one going forward, I wouldn't want to go back and redo the old Markdown. As it gets updated, we can convert it. So, I would propose developer's choice.

[SterlingPeet]

OK, so it sounds like your main concern (renderable on GitHub) is satisfied, but you are still concerned that mandating a switchover would generate unnecessary busy work?

And your solution is to have developers convert the documentation only for components/code that they actually touch, is that accurate?

[bocchino]

Developer's choice sounds good to me.

[timcanham]

@SterlingPeet That would be my opinion as one of many. :) It would make sense to allow users to pick, but for the sake of everybody's time, I wouldn't recommend issuing a bunch of Git issues to go back and change the old Markdown.

[LeStarch]

I don't follow "developer's choice". Does this mean that each developer would pick their documentation standard? As the team that has to clean-up and keep in order all the documentation, I want one chosen format to maintain. If we are in transition, we might have two....as we move from one to the other....but I don't want one component in RST, one in AsciDoc, one in PDF...

[SterlingPeet]

OK, so I see an argument for "developer's choice" where there might be some resistance to learning a rich text syntax. However, I do think it is a weak argument, especially when bounded to the formatting that correctly renders a page on GitHub. This is still a superset of the features provided by Markdown, but a subset of features available to a bigger documentation effort in Asciidoc or RST.

I would like to see the F Prime team decide on a standard rich text format that fits the needs (in this case better than Markdown), and commit to it. If there are concerns about a learning curve, its not to hard to come up with a standard cheatsheet, like this one for RST, and distribute it to the team.

I also agree that submitting potentially hundreds of formatting change PRs is just silly. It adds a lot of noise to the git history too. A very, very strong commitment to a new rich text format does not need to generate tons of effort, it can be done in one strategic commit. Pandoc can do the conversion for you, converting everything to a different format programmatically while keeping all your formatting. A conversion from Markdown to RST should be a strict superset of features, so there should be no loss of formatting.

I can go try this on my own and submit a PR with a single commit to change all of the documentation formatting, just so that the team can see an example of a full conversion and talk about it.

[astroesteban]

In some projects I've worked on we used ReStructured Text for more thorough documentation. We found that Markdown was great for simple to slightly advanced use cases like a README or a CHANGELOG but fell short with complex or long-form documentation. It was also annoying trying to edit Markdown in VS Code or Vim and then there were issues of some developers introducing unsupported tags or flavors into the Markdown and then have it not render correctly for everyone else.

Alternatives like ReStructured Text, and by the looks of it Asciidoc, provide great functionality to create even better documentation.

My vote is in support of a transition to something like ReStructured Text or Asciidoc.

[gjwatney]

I vote for AsciiDoc
I switched from Markdown to AsciiDoc out of frustration in trying to do tables. AsciiDoc does them so much better.

[SterlingPeet]

This is a very real problem with Markdown

[SterlingPeet]

we actually wrote some code to auto-generate the GitHub-flavored tables of commands and telemetry, but it was still limited

[bocchino]

We found that Markdown was great for simple to slightly advanced use cases like a README or a CHANGELOG but fell short with complex or long-form documentation.

This is also my experience.

On a related point, I've seen evidence that the lack of key features in Markdown leads to bad practices in writing and maintaining docs. For example:

1. Because there's no auto section numbering, developers write the numbers in by hand.
2. Because there's no way to include text, developers copy-paste code examples.

As F Prime matures and develops, it would be good to improve this situation.

[bocchino]

Also both asciidoctor and the GitHub AsciiDoc renderer will auto-generate a hyperlinked table of contents. This is extremely useful.

[Joshua-Anderson]

I general I have no strong opinions on the matter, however, I am slightly biased towards markdown since we're already using markdown and in general it seems more popular than restructured text. If we wanted to convert from markdown to restructured text we may be able to use a tool to convert everything over like pandoc. I'd be strongly opposed to splitting our documentation between multiple formats.

There is a tool (asciidoctor) for locally rendering AsciiDoc source to HTML and PDF. There is no such tool for GitHub-flavored Markdown. You have to ask GitHub to do the rendering. In practice, this means pushing to GitHub to see the rendered version.

I don't believe this pro is accurate.

There are many local tools that can render markdown. For example, VSCode has a nice side by render view for markdown (including github flavored markdown. In practice most editors I've used have markdown support.

Additionally, you can trivially render markdown to pdf with pandoc. I've used that for writing formal papers with markdown before.

[bocchino]

These are good points, but I stand by the Pro. I just ran pandoc on Svc/CmdDispatcher/sdd.md. It generated some reasonable HTML, but it sure doesn't look as nice as the output of asciidoctor. There's some junk at the top due to some text in the source that the GitHub renderer recognizes as metadata (apparently) but pandoc doesn't recognize. Pandoc is good for what it does, but in my experience it's a best-effort tool that gets about 80-90% of the job done; the rest you have to fix up yourself. It's not a substitute for a tool like asciidoctor. Side-by-side rendering etc. is not a direct substitute for source to HTML conversion.

[ThibFrgsGmz]

Personally, I am in favor of abandoning Markdown for a more elaborate language to improve the user experience when writing documents.

I must admit that I am not familiar with Asciidoc, having only worked with MD and RST. And, as I am currently overloaded with work, I don't have the time to conduct a thorough research on this technology.

What saddens me is losing the real-time side-by-side rendering functionality we have in VS Code with Mardkdown.
Some may consider this a minor detail, but I think it is essential.

According to the discussions, this real-time rendering functionality is not supported?

---

Side note: I would like to share the use case of F Prime that we have in my company/team. I think it might be interesting to know how a stakeholder uses it in a private/industrial context. My anecdote applies to any open source repository, on GitHub, GitLab, etc.

Since my company has very limited access to the internet (including GitHub), we cannot clone GitHub repositories directly to our development machines using "git clone".
Therefore, we have to ask our IT department to download the repositories so that we can have internal copies to develop/operate with.

We use Tuleap ALM (https://www.tuleap.org/fr/) internally to manage our projects and software, and this tool only supports Markdown rendering, which gives us a relatively similar user experience to GitHub for Markdown documentation.
Switching from Markdown to another document language would therefore degrade our internal user experience when viewing F Prime documentation.

However, these are issues specific to my company and its infrastructure. This should not hinder the evolution of the F Prime project; it is up to the stakeholders to adapt in my opinion, at least for this topic. Especially since we are currently using it for PoC rather than marketable projects.

---

[bocchino]

What saddens me is losing the real-time side-by-side rendering functionality we have in VS Code with Mardkdown.

It looks like this is fully supported in AsciiDoc:

https://marketplace.visualstudio.com/items?itemName=asciidoctor.asciidoctor-vscode

It looks like the plugin uses asciidoctor to do the real-time rendering.

[bocchino]

Looks like there is RST support too:

https://marketplace.visualstudio.com/items?itemName=lextudio.restructuredtext

[ThibFrgsGmz]

Oh, you're right, these plugins support real-time rendering in VS Code... Well, for Asciidoc yes, for Rst the rendering has trouble to update in real time. But I'm still relieved on that side!

Thanks for taking the time to do this research; I could have done it myself at the time, sorry...

[LeStarch]

I was unable to find any support in Doxygen for importing asciidoc nor restructuredText. I followed the above links/discussions but is seems like nothing came of the disucssions. Currently it looks like only Markdown and Doxygen comments are support through Doxygen.

The effect of this means we would be unable to generate documentation for our code using Doxygen. Currently, we run Doxygen on the C++ code and it brings in the Mardown-formatted SDDs for the components. This gives us fairly good code documentation for little overhead. That is, we get the SDDs and C++/API documentation in a standard package.

I would be hesitant to adopt a different format for these SDDs, because we'd then need to maintain a new tool/workflow for producing the SDD/API documentation we need, which would still require doxygen for the C++ comments as that is our established style.

Markdown, for all its irritations, has always won-through thus far because it always has support in XYZ tooling that saves us maintenance or other costs.

[LeStarch]

I should also note that our public webpage is built on Markdown -> .HTML translation using Github Pages. Github pages, does not have the jeykll-asciidoc plugin on its built-in list, which would be required for building our webpage.

https://pages.github.com/versions/

[bocchino]

Due to the showstopper-type issues raised by Michael, for now I have converted the Math tutorial from AsciiDoc to Markdown. :^| Here is the PR:

#1143

It is an interesting case study in using ad-hoc scripts to work around Markdown limitations. Maybe scripts like this can be a solution, until we can get the tools to support AsciiDoc. We just need some way to write documents with basic features such as numbered sections, hyperlinked tables of contents, and proper internal cross links that are checked for validity.