VIP 18: RST specification for code generation

Synopsis

Replace .vcc file format with .rst

Change the VMOD development model so that VCC files can be retired, and an RST file becomes the sole source of information about a VMOD.

Rather than converting VCC to RST (and then having issues like having to maintain documentation in two places on web-hosted repositories like github or gitlab, which expect a README in the repo root), the RST file should be able to serve for documentation without changes, and RST/python directives would be used for the VMOD declarations currently written in VCC.

This topic has come up in conversations on IRC, and @Dridi and @Geoff have expressed interest in getting something going, but it hasn't gotten far past the stage of "we ought to do that".

For VMOD authors, this change would be "wreckage-complete" (that is, as comprehensively wreckaging as anything could be) if VCCs immediately become obsolete in 6.0. So if we do it, we should do it so that VCCs files are deprecated but still work for at least another release, maybe for the next two releases (i.e for another year).

Why?

The .vcc formats use of "\n$" is a gross hack and not very friendly to anybody.

Extensible VSC's will need a facility similar to .vcc to produce .c and .h files.

How?

The proposal at hand is to ditch the .vcc format and properly embed the necessary information in a .rst file.

By properly we probably mean adding new varnish-specific directives to Sphinx.

(Tutorial: http://www.sphinx-doc.org/en/stable/extdev/tutorial.html)

PHK comments

I'm not very keen on putting private Sphinx extensions in front of code compilation, both from a developer time point of view, but also as a matter of general stability.

As far as I can tell, the sphinx directive syntax is \n..<whitespace><directive>::<whitespace> and the directive continues until but doesn't include the next \n<nonwhitespace>

That shouldn't be too hard to parse in a stand-alone python program for code generation.

Github seems to render unknown directives as a code block: https://github.com/bsdphk/sandbox

That means that whatever we "feed" the directive, has to be very human readable, if the idea of using the VCC's controlling .rst file as github README.rst -- This may be the biggest challenge.

As for the actual sphinx extension, it seems to be possible to store it locally in the tree, using something like this::

diff --git a/doc/sphinx/conf.py.in b/doc/sphinx/conf.py.in
index 30c527a50..ab867eecd 100644
--- a/doc/sphinx/conf.py.in
+++ b/doc/sphinx/conf.py.in
@@ -20,9 +20,13 @@ import sys, os
 
 # -- General configuration -----------------------------------------------------
 
+import sys, os
+
+sys.path.append(os.path.abspath('sphinxext'))
+
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ['sphinx.ext.todo']
+extensions = ['sphinx.ext.todo', 'varnish.ext']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['=templates']