Line-wrapped Latex string isn't recognized as Latex

[kbarros]

An example of the error appears in this doc string:

https://github.com/SunnySuite/Sunny.jl/blob/47aad8c2c560f233b59441b9d9e3b48edd48e9ff/src/Operators/Stevens.jl#L230-L231

which is rendered like this:

https://sunnysuite.github.io/Sunny.jl/v0.7.1/library.html#Sunny.stevens_matrices-Tuple{Any}

Note that the Latex ``N = 2s + 1`` gets wrapped across lines as follows:

The problematic wrapping is ``N = 2s
+ 1`` ...

and then Documenter picks up + 1`` ... as the start of a new bulleted list.

[singularitti]

I'm trying to parse this code:

using Documenter
using Documenter.LaTeXWriter

function _dummy_lctx()
    doc = Documenter.Document()
    buffer = IOBuffer()
    return LaTeXWriter.Context(buffer, doc)
end

function _latexesc(str)
    lctx = _dummy_lctx()
    LaTeXWriter.latexesc(lctx, str)
    return String(take!(lctx.io))
end

function _md_to_latex(mdstr)
    lctx = _dummy_lctx()
    ast = Documenter.mdparse(mdstr; mode=:single)[1]
    LaTeXWriter.latex(lctx, ast.children)  # should use latexesc internally
    return String(take!(lctx.io))
end

str = raw"""This will produce an ``N×N`` matrix where ``N = 2s
      + 1``. Linear combinations of Stevens operators can be used as a "physical
      basis" for decomposing local observables.
      """

and get:

_md_to_latex(str) |> print
┌ Error: mode == :single requires the Markdown string to parse into a single block
│   s = "This will produce an ``N×N`` matrix where ``N = 2s\n+ 1``. Linear combinations of Stevens operators can be used as a \"physical\nbasis\" for decomposing local observables.\n"
│   mdast =
│    @ast MarkdownAST.Document() do
│      MarkdownAST.Paragraph() do
│        MarkdownAST.Text("This will produce an ")
│        MarkdownAST.InlineMath("N×N")
│        MarkdownAST.Text(" matrix where ``N = 2s")
│      end
│      MarkdownAST.List(:bullet, true) do
│        MarkdownAST.Item() do
│          MarkdownAST.Paragraph() do
│            MarkdownAST.Text("1``. Linear combinations of Stevens operators can be used as a \"physical")
│          end
│        end
│      end
│      MarkdownAST.Paragraph() do
│        MarkdownAST.Text("basis\" for decomposing local observables.")
│      end
│    end
│
└ @ Documenter ~/.julia/packages/Documenter/C1XEF/src/utilities/utilities.jl:526
ERROR: ArgumentError: Unsuitable string for mode=:single
Stacktrace:
 [1] mdparse(s::String; mode::Symbol)
   @ Documenter ~/.julia/packages/Documenter/C1XEF/src/utilities/utilities.jl:527
 [2] mdparse
   @ ~/.julia/packages/Documenter/C1XEF/src/utilities/utilities.jl:512 [inlined]

[singularitti]

I found it is not only a problem that MarkdownAST.jl (which Documenter.jl uses), Julia's stdlib Markdown.jl has this problem, too:

using Markdown: parse, @md_str

julia> Markdown.md"""This will produce an ``N×N`` matrix where ``N = 2s
              + 1``. Linear combinations of Stevens operators can be used as a "physical
              basis" for decomposing local observables.
              """
#  This will produce an N×N matrix where ``N = 2s
#
#    •  1``. Linear combinations of Stevens operators can be used as a "physical
#
#  basis" for decomposing local observables.

julia> str = raw"""This will produce an ``N×N`` matrix where ``N = 2s
         + 1``. Linear combinations of Stevens operators can be used as a "physical
         basis" for decomposing local observables.
         """

julia> parse(str)
#  This will produce an N×N matrix where ``N = 2s
#
#    •  1``. Linear combinations of Stevens operators can be used as a "physical
#
#  basis" for decomposing local observables.

[goerz]

It's the same underlying problem (MarkdownAST.jl still uses the stdlib-Markdown): "Julia Markdown" isn't actually markdown, but a (poorly specified) subset of markdown. One of the ways Julia Markdown differs from actual markdown is that it can't handle inline math with line breaks. For that matter, it also can't handle linebreaks in links or reference links, which is even more annoying.

You have to adjust your writing process to accommodate Julia-markdown's idiosyncrasies. The "correct" solution is to add the long-planned support for CommonMark documents, see #2519 and references therein. Until then, this is basically an "upstream" issue with the stdlib-Markdown library. Hence, I'll close this here (even though I share your frustration).

[kbarros]

I guess the answer is "no", but I will ask just in case -- Is there a way to request that Documenter should throw an error if it detects the two characters ``, but cannot form a full Latex string? Such an error would cause our Github CI/CD doc builds to break, and the maintainers would be notified that a fix is necessary.

[goerz]

Not currently, as far as I’m aware, but maybe that sort of check is something someone could explore in a plugin

[mortenpi]

subset of markdown

Just a nit: not a subset. It also has features (e.g. admonitions, tables) that are not part of standard Markdowns (CommonMark or others). It's just yet another flavor 🙃

In any case, yes, #2519.