latex macros not in $$s not applied to math environments for markdown -> html

[petrelharp]

I am converting from markdown to html and to pdf, with \newcommands up top. I can either wrap these in '$$'s or not, but both options are broken in different ways:

This example:

\newcommand{\E}{\mathbb{E}}

I expect $\E[X]$ but get
\begin{align}
    \E[ X ] .
\end{align}

produces pdf output fine, but the second \E, inside the \align, is not expanded.

On the other hand, this example:

$$
\newcommand{\E}{\mathbb{E}}
$$

I expect $\E[X]$ but get
\begin{align}
    \E[ X ] .
\end{align}

produces the correct html, but producing a pdf dies with

! Undefined control sequence.
l.68 I expect \(\E

This seems to be part of a larger conversation, e.g. #1426?

[jgm]

Pandoc only expands macro definitions if they're not inside a raw latex block (which is what your
\begin{align}...\end{align} gets parsed as).

So, the fix is to put this into a proper math environment, and use aligned instead of align:

$$
\begin{aligned}
\E [ X ] .
\end{aligned}
$$

[petrelharp]

That works for html output, but not for pdf:

! Package amsmath Error: Erroneous nesting of equation structures;
(amsmath)                trying to recover with `aligned'.

See the amsmath package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              

l.69 \end{align}

It seems we're being hit by the curse of having too many ways of doing one thing.

[jgm]

You're getting an error message that says you have \end{align}. I suggested aligned. Are you sure you did what I suggested?

[petrelharp]

Oops; I missed that: you're right, I just put $$ around the align.

I am used to using the amsmath environments, and would like to continue using them, but $$ + aligned seems nearly as good (although, equation numbering?).

More seriously, this seems like a major point of confusion for users unfamiliar with how pandoc parses the document (like me): macros are expanded in some of the latex, but not all of it. For mathjax or pdf output, passing the macros on downstream in the right way would do the trick, no?

[jgm]

+++ Peter Ralph [Sep 01 15 23:09 ]:

Oops; I missed that: you're right, I just put $$ around the align.

I am used to using the amsmath environments, and would like to continue
using them, but $$ + aligned seems nearly as good (although, equation
numbering?).

More seriously, this seems like a major point of confusion for users
unfamiliar with how pandoc parses the document (like me): macros are
expanded in some of the latex, but not all of it. For mathjax or pdf
output, passing the macros on downstream in the right way would do the
trick, no?

Yes, that's probably right.

[jgm]

Suggestion: try

pandoc -f markdown-latex_macros -t html --mathjax

This should cause macros to be passed downstream as they already are for PDF.

[petrelharp]

My two intial examples differ in that one wraps the macros in '$$'s. Using -f markdown-latex_macros does indeed pass along the unwrapped macros in the version without '$$'s; but still, mathjax doesn't recognize them, because mathjax needs macros to be inside a math environment.

Since pandoc recognizes macros, couldn't it wrap the macros in $$s before writing out to html with mathjax enabled?

[jgm]

+++ Peter Ralph [Nov 24 15 21:26 ]:

My two intial examples differ in that one wraps the macros in '$$'s.
Using -f markdown-latex_macros does indeed pass along the unwrapped
macros in the version without '$$'s; but still, mathjax doesn't
recognize them, because [1]mathjax needs macros to be inside a math
environment.

Since pandoc recognizes macros, couldn't it wrap the macros in $$s
before writing out to html with mathjax enabled?

It's not so simple, because of the reader/writer separation.
Pandoc only recognizes macros in the reader; once we get to
the HTML writer, it's just represented as raw LaTeX (which
pandoc passes through if --mathjax is selected). The
reader doesn't know you're going to be writing mathjax.

[petrelharp]

Hm, I see. Same thing for the header-includes: mechanism, I guess, and the problem comes down to mathjax diverging from latex on this point. Since mathjax and latex deal with macros differently, doing something that will work unconditionally with both of them requires having a way to keep macros separate, so that the writer knows what to do with them?

One option, and a tidier one than putting macros at the top of documents: I could store my macros in a separate .tex file, and modify the html and latex templates to include it in the right way, maybe specified in the YAML. For instance:

---
macros: "macros.tex"
---

and then, in the right place:

$if(macros)$
\input{$macros$}
$endif$

... the right way to get them into the html looks less obvious.

[jgm]

+++ Peter Ralph [Nov 24 15 23:10 ]:

Hm, I see. Same thing for the header-includes: mechanism, I guess, and
the problem comes down to mathjax diverging from latex on this point.
Since mathjax and latex deal with macros differently, doing something
that will work unconditionally with both of them requires having a way
to keep macros separate, so that the writer knows what to do with them?

One option, and a tidier one than putting macros at the top of
documents: I could store my macros in a separate .tex file, and modify
the html and latex templates to include it in the right way, maybe

specified in the YAML. For instance:

macros: "macros.tex"

and then, in the right place:
$if(macros)$
\input{$macros$}
$endif$

... the right way to get them into the html looks less obvious.

You can use a trick like

-V macros=$(cat macros.tex)

to inject the string content directly into the template.
This should work in HTML too.

[petrelharp]

Or, without modifying the template, for html:

pandoc test.md -H <(echo '\['; cat macros.tex; echo '\]') --mathjax --standalone -o test.html

and for pdf:

pandoc test.md -H macros.tex -o test.pdf

both work if this is test.md:

$$\begin{aligned}
    e^{i\pi} = -1 \in \R
\end{aligned}$$
but
\begin{align}
    e^{i\pi/2} = \notin \R
\end{align}

and this is macros.tex:

\newcommand{\R}{\mathbb{R}}

[petrelharp]

I have a reasonable way to do this now, so I'm closing this issue, but a good way to make it smoother for newcomers would still be nice. I don't know if this warrants a special command-line flag, but that'd be one way: like --latex-macros would include raw in the header if the output isn't mathjax; and wraps it in a math environment if it is mathjax.