Post-0.3 Docile/Lexicon roadmap.

[MichaelHatherly]

After Julia 0.4 is release there will be little need in keeping much of the current features around. The focus will be on docstring/helpsystem extensions that build on top of the infrastructure in Base. To that end the following will serve as a working list of features that may still be useful:

• Custom docsystem hooks. register!.
  • Hooks.track, mostly for debugging docstring capture.
  • Hooks.__doc__ and @__doc__, documenting macro-generated code.
  • Hooks.directives, parsing directives from inline docs.
  • others?
  • doc!* variables, interpolate markdown formatted metadata such as signature and per-argument docs:
    • doc!sig
    • doc!args
    • doc!kwargs
    • others?

ie.

map(register!, (Hooks.doc!args, Hooks.doc!kwargs, Hooks.doc!sig))

"""
$doc!sig

$doc!args

$doc!kwargs
"""
function foo("x docs", x, "y docs", y; "k docs", k = 1)
    # ...
end

• Directives. @{...}:
  • docs, default
  • module
  • esc
  • ref
  • anchor
  • repl
  • code
  • more?
• Build external docs. makedocs:
  • integration with 3rd-party tools, mkdocs, gitbook etc.
  • other features / output formats? (Limited by what Base.Markdown can do.)
• Docsystem query extensions:
  • separate REPL mode, entered using ] key after using Docile.
  • full-text search
  • regex
  • objects
  • method signatures
  • method return types
  • logic, &, |, and ! for combining other queries
  • improve look-and-feel of the webserver query mode on hold for a the moment.

[hayd]

Should we update/create a new issue for build related changes. Specifically:

• @{..} directive syntax (Note: I much preferred directive".." to D"..")*
• repl directive
• docs directive
• generate mkdocs (maybe)
• using directives within docstrings

* in the latest rewrite I can't get the

@{
    foo
    bar
    baz
}

syntax working. Specifically it should work here, and we should add in the --- when rendering. :)

The stacktrace if you can't replicate:

Building: 'src/internals.md'   --> 'build/internals.md'
ERROR: LoadError: ParseError("extra token after end of expression")
 in parse at parse.jl:195
 in call at /Users/andy/.julia/v0.4/Docile/src/Docs/directives.jl:61
 in exec at /Users/andy/.julia/v0.4/Docile/src/Docs/directives.jl:17
 in process! at /Users/andy/.julia/v0.4/Docile/src/Docs/process.jl:17
 in process! at /Users/andy/.julia/v0.4/Docile/src/Docs/process.jl:11
 in process! at /Users/andy/.julia/v0.4/Docile/src/Docs/process.jl:4
 in call at /Users/andy/.julia/v0.4/Docile/src/Docs/doctypes.jl:10
 in anonymous at /Users/andy/.julia/v0.4/Docile/src/Build/makedocs.jl:69
 in cd at ./file.jl:22
 in makedocs at /Users/andy/.julia/v0.4/Docile/src/Build/makedocs.jl:51
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:264
 in process_options at ./client.jl:308
 in _start at ./client.jl:411
while loading /Users/andy/.julia/v0.4/Docile/doc/build.jl, in expression starting on line 3

Should I/we start posting separate issues?

[MichaelHatherly]

in the latest rewrite I can't get the

Yeah, that was just an oversight from some changes made yesterday. Pushed fix in 6beba76, hopefully that works a bit better. Instead of splitting on new lines, it uses parse to read complete expressions allowing for multiple expressions per directive.

I much preferred directive".." to D".."

Both of those are actually gone now :) since it made it possible to overwrite defined directives by just redefining the method signature outside of the module, not cool. I've changed to a manual cache and dispatch here. That file has some examples of how the directives are defined now.

using directives within docstrings

That should work, for the most part, though there's some issues with relative paths for the auto cross referencing using @{ref:...} in docstrings that I've not worked out just yet.

Should I/we start posting separate issues?

Yes, if something doesn't fit here then feel free to open an issue specific to it.

[MichaelHatherly]

repl directive

The basics of that one are implemented, here, but if you'd like additional features for it then they'd definitely be welcome improvements.

[hayd]

Thanks for the fix!

One remaining directive idea is parameter this would be really useful.

I really dislike the current style for posargs/keyword (everyone has different conventions) e.g. https://github.com/MichaelHatherly/Docile.jl/blob/6beba76a739fc5e81e138fb2c7b8e24d8de95c8f/doc/build/public.md#functions

I think would be nice if there was some way to grab the default arguments too:

@{parameters
    # source
    Directory to collect markdown files from. The provided path is treated as being
    relative to the directory in which the build script is run.
    # build
   ...etc
}

Headers (regardless of size) become arguments, if there are keywords the default arg is grabbed (not sure whether that's possible), the subsequent markdown is attached to that parameter.

would become (and could be tweaked later for everyone :) ):

#### Parameters

##### `source = "src"`

Directory to collect markdown files from. The provided path is treated as being relative to the directory in which the build script is run.

##### `source = "build"`

...etc

This is already doing so much (I'm being "Julian greedy").

Edit: tried to implement but is messy. Will pr if I get it working...

[MichaelHatherly]

One remaining directive idea is parameter this would be really useful.

If we can access the object then .source in the FuncDoc will give us the parameter expression. Should be possible I think.

I'm not a huge fan of the output #### and ##### for this, how about a table instead?

[hayd]

how about a table instead

I think that should work and will be much better! I wasn't a fan of anything before but conscious that h1-3s screw up formatting... and pretty much everyone tries to use those atm. (The parameters title needs to be h4 I think.)

[MichaelHatherly]

The parameters title needs to be h4 I think.

Sounds reasonable.

Maybe these kind of "rendering" directives should wrap their output in a <div class="..."> to allow for better customised styling?

[hayd]

Sounds good, we can even add some style to our default mkdocs css. (depending on other PR) :)

Is this going to be a problem in the repl? (which can't parse divs/classes etc.).... Aside/related: the julia devs wanting julia to host julia docs may be tricky too for the same reason. ?

[MichaelHatherly]

Is this going to be a problem in the repl?

julia> Markdown.parse("""
       <div class="skipme">
       oh...
       </div>
       """
       )
  <div class="skipme"> oh... </div>

Could just have a flag to only show when !isinteractive() perhaps?

[hayd]

Maybe this needs to just be solved in Markdown. Each tag could just be a Markdown object:

tag::AbstractString
class::AbstractString
content::MD

and deal with some e.g. links, images, headers. and let all others pass through (e.g. divs).

:/

Edit: class should probably be attrs::Dict{AbstractString, AbstractString}.

[hayd]

Each tag could just be a Markdown object:

lol. This is not easy... at all.

[MichaelHatherly]

Each tag could just be a Markdown object:

lol. This is not easy... at all.

Probably best to just add some logic into the writemime methods here to print the necessary tags only for file output rather than REPL as well.

[hayd]

Would be great to release something soon now we have 0.4rc. I haven't had much time to look recently, but it seems (from all the ticked checkboxes) like it's coming together?

[MichaelHatherly]

I might have some time to sort things out soon, but there's still much polishing up of things to do. There's also a couple of things that need to be redone, like the cross-referencing, which I may get to today.

[MichaelHatherly]

Just an update on the current state of the rewrite branch:

• Directive syntax is now closer to macro syntax, @ref{...}, rather than @{ref: ...}. I'm finding that a lot nicer to read and write, and it's one character shorter too. The behaviour of @{...} is still the same, expanding to @docs{...}.
• Other than that most of the other changes are just code churn.

[hayd]

I did a brief talk on this last night at SF julia meetup, I think it went down pretty well.

Is the idea going forward that:

1. old Docile becomes 0.3 only
2. Docile becomes Docile rewrite
3. Lexicon deprecated in favour of new Docile.

I think the main things to do are:

• only using docstring from your package (I haven't quite been able to get that working with getdocs, though it seems achievable via filtering out based on the MD's meta attribute). I think there is something a little subtle there.
• some cleanups to the generated output (I have some ideas for that, will make a PR over the next few days). Specifically enumerating signatures. We should hide/configure some of this.
• Fix up the mkdocs generator. 🚢
• Explore exporting to other things e.g. Judo. (Perhaps we can show off the different exports for the Docile docs). Judo may have issue with $ syntax, as it uses that internally for variables... I'm yet to find a simple Judo example yet unfortunately.
• warn about h3- (I had a branch doing this, but the code has diverged quite a bit).
• The parser is very sensitive, I'm not sure if that's terrible in itself, but the errors are difficult to reason about (since the lines aren't referenced).

---

Aside: I'm still not a fan of the hook syntax... I'd prefer:

# one-line
\n{{directive_name:content}}\n

# multi-line
\n{{directive_name:?\ncontent_with_newlines\n}}\n

# without directive_name it's docs

I think that's clearer, enforcing these newlines is ok (as Markdown won't break the text with a newline) and I that makes the parsing strict enough to avoid ambiguity.

I don't think the @ syntax adds any value here (whilst I see that it's similar in spirit to julia macros), whilst {{...}} is a standard for templates (and directive is similar to code flavour).

Granted that's bike shedding for the main functionality, which is excellent, and everyone is excited about!

[MichaelHatherly]

I think it went down pretty well.

Great to hear!

Is the idea going forward that:

Yes, that's hopefully the plan. http://pkg.julialang.org says that 172 packages depend on Docile... so the switch and deprecations will need to be done carefully.

filtering out based on the MD's meta attribute

That's probably the best way to go at the moment I think.

show off the different exports for the Docile docs

That would be really nice to have.

Judo may have issue with $ syntax, as it uses that internally for variables

The $ syntax for latex has been bugging me for a while. I think ScholarlyMarkdown has a nice approach to this, http://scholarlymarkdown.com/Scholarly-Markdown-Guide.html#math. Might not be too difficult to add it to the parser.

warn about h3-

Having thought over this for a while I think it would probably be best to let authors use h1 for different sections in their docstrings. This is really just a presentation issue that could be solved by Docile automatically lowering all headers by 2 (or some other count, possibly configurable) when outputting docs to file. Rust appears to go with h1 for different sections in docstrings, https://doc.rust-lang.org/stable/book/documentation.html#special-sections, for what it's worth.

but the errors are difficult to reason about (since the lines aren't referenced)

Yeah, threading some kind of context object through the parser to track where we are would be useful.

enforcing these newlines is ok (as Markdown won't break the text with a newline)

Requiring \n before and after is going to look odd for inline directives such as @ref{...}, \n{{ref:...}}\n, even if they do get parsed correctly by Markdown.parse.

standard for templates

They aren't really templates in every case though. Something like @module{...} doesn't expand to anything in the output. Granted, in the common case, @{...}, does act kind of like a template.

[hayd]

xlink for h1/3 thing JuliaLang/julia@3160e54
rst gives a fatal error, so let's definitely not do that.

If we lower, we should lower all headers (or lower h1 and h2s to h3s??), I hate APIs where you can change one part of the docstring and it to change the behavior of the entire thing. In this case adding a h1 somewhere changes h3s to h5s, and so an h3 in one docstring may render differently to an h3 in another.

[MichaelHatherly]

we should lower all headers

Agreed, it would definitely be all of them, except for h6 I guess. Though perhaps h6 could be turned into bold or italic.

[hayd]

Just to clarify, I think what would be consistent if h1 -> h3, h2 -> h3, h3 -> h3, h4 -> h4, which would solve the issue without error whilst also keeping an h4 always an h4. :)

[MichaelHatherly]

That would lose some of the document structure by flattening to h3, though maybe that would be fine. I'll probably only get a chance to push on with this in early December though, so feel free to hack away at it if you have the time.

[hayd]

Bump! Anything that can be done to help here?

[MichaelHatherly]

Yeah, I've been meaning to revisit this for some time. There's a few bits, such as @__doc__, that aren't needed any more and some others that, after using for a while, could be better thought out:

Such as the @{...} syntax which doesn't look great in unparsed docstrings. Some less obtrusive syntax would be nice, perhaps using the idea from the "ref" links we're currently using in base, ie. [...](:ref:...), but less RST-like. I'll try come up with something consistent today. Things like @code{ and @repl{ could just be code blocks with a special ".language" field names.

Also, given the number of packages that appear to depend on current Docile, 177 according to http://pkg.julialang.org/, making a breaking change could be quite problematic. It may actually be best to split out the document generation code into a separate package, and leave Docile and Lexicon as is for the moment. Something like the following for the future uses of each package:

• MysteryPackage.jl (needs a good name): generate documentation from docstrings (doctest also?)
• Docile.jl: experimental extensions to the docsystem, such as the docvars doc!sig, etc.
• Lexicon.jl: extended query mode for help in the REPL, ie. query>.

[hayd]

What about something lazy / "say what you see"-y: MakeDocs.jl ?

+1 to code and repl being code blocks.

I was going to say that you could set multiple words here (and it should render ok)

```julia repl
some coce
```

but looks like that's only in commonmark http://johnmacfarlane.net/babelmark2/?text=%60%60%60julia+repl%0Acode+block%0A%60%60%60

Sounds great to move to a new package moving forward, hopefully I can get my hands dirty once it's there :) It'll also make dev/status a little less opaque/easier (than being on a branch) so 💯

[MichaelHatherly]

I was going to say that you could set multiple words here (and it should render ok)

Yes, something like that would probably be best. For the current @docs{...} / @{...} maybe write that as

```docs
foo
bar
```

which would make the raw markdown files in src/ quite readable prior to parsing as opposed to their current look. Also could extend this kind of thing to metadata, table of contents, etc. perhaps.

[hayd]

Thinking about it DocTests.jl should probably be a separate package too. IMO The more modular packages are the clearer/better: do one thing and do it well. :)

MysteryPackage.jl/MakeDocs.jl : generate documentation from docstrings
DocTests.jl run the jldoctest code blocks for a package (e.g. can be run as CI line)
Docile.jl: experimental extensions to the docsystem, such as the docvars doc!sig, etc.
Lexicon.jl: extended query mode for help in the REPL, ie. query>.

I wonder if at some stage these should be "blessed" as JuliaTests/JuliaDocumentation groups like others.

A related project I was thinking about was multi-language support: now we have a docstrings these can be translated (there are free services to source translations for open source projects), these are looked up and help is in that language... One problem is that this would have to be for each package rather than globally.

[MichaelHatherly]

Thinking about it DocTests.jl should probably be a separate package too.

Yeah, probably best to do that. Though I'd like it if building docs with makedocs (or whatever name) also does a doctest on everything it finds. Having "MakeDocs" depend on "DocTests" would be fine though.

I wonder if at some stage these should be "blessed"

Sure, once things are passed the initial teething stage then that would be best for long-term survival, though I'm not planning on going anywhere.

multi-language support

That could probably work quite well I think. Shouldn't be too difficult to replace, at runtime, any specific docstring with another from a package. A task for another day though.

[MichaelHatherly]

MysteryPackage.jl package is here, https://github.com/MichaelHatherly/Lapidary.jl. Mostly a prototype at the moment, but it does manage to do most of what's needed. The readme outlines most of what's been implemented so far. Issues can be opened over there if you happen to try it and run into problems. Doctesting is currently built-in, but could be split out into another package relatively easily.

[hayd]

to the name... But awesome, I will give it a whirl!

[MichaelHatherly]

to the name

Heh, yeah just a working name for the moment. Easy to change anytime really.