locales: how to mix RTL with LTR

[pkra]

A question on TWN.

I'm not even sure how we handle RTL right now let alone the actual question on how to mix RTL and LTR.

The example is Missing \left or extra \right where \left and \right would be LTR in an RTL context.

[fred-wang]

I think we have a "dir" parameter for each language to add some CSS style but I'm not exactly sure where it is used.

Here, "left" and "right" are names of TeX commands, so I don't think they depend on the language direction?

[pkra]

Thanks, Fred, glad to hear we got the global direction covered.

That leaves the question how to preserve LTR content in the string (\left, \right). The user pointed out that left-to-right marks are one way that's now considered outdated in HTML.

I guess we could use ltrm since we convert things anyway. Or we could use markdown syntax (e.g. escaping by ` ). Or we could allow HTML strings.

[dpvc]

As Fred mentioned, there is a direction attribute in the language data that is applied to the CSS for any messages that appear in HTML-based dialogs (there is also a font parameter that can be used to suggest appropriate fonts for the language). But some messages appear in browser pop-up dialogs where there is no such control. Others appear in <merror> tags within the MathML (though the noErrors TeX extension prevents that for TeX input).

The error message in question is of the latter type. We do not yet have support for HTML within token elements, so at the moment that is not a solution. It would be possible, I suppose, to break up the error message into separate <merror> sections with different direction attributes. But since MathJax doesn't yet support the direction attribute, that wouldn't really help yet, either. And anything going to a browser popup has to be a plain text string, so I don't think we can do anything about it in that case.

So I don't think there is a simple global solution.

[pkra]

Good point.

What should we do in the mean time? One way or another, we need to handle mixed RTL and LTR content. Would markdown escaping via ` be enough?

[ebraminio]

To clarify the problem:
"Missing \left or extra \right" on Persian for example will become

"/left یا /right گم شده"

Hebrew translation is using this approach for such translations:
‎/left یا ‎/right گم شده that will become:

"‎/left یا ‎/right گم شده"

But that is not cool (http://www.w3.org/TR/its/#directionality http://www.w3.org/International/questions/qa-bidi-controls.en.php http://www.w3.org/TR/html-bidi/), so I suggested using <span dir="ltr" style="unicode-bidi: embed;">/left</span> یا <span dir="ltr" style="unicode-bidi: embed;">/right</span> گم شده (using dir="..." is not enough on inline elements)
but there is also another solution, using <bdi> tag that will autodetect the direction but it is a new HTML tag (and will use unicode-bidi: isolate; that is not supported on all browsers). <bdi>/left</bdi> یا <bdi>/right</bdi> گم شده

[pkra]

Thanks, @ebraminio, for the additional information, especially the fact that the Hebrew translation on TWN already uses ‎.

Just to clarify: the strings we get from TWN are processed later. So we can create correct HTML strings as long as we decide on how to represent them on TWN.

[dpvc]

Just to reiterate, not all the message strings are used in contexts where HTML is available, so it is not only an issue of how to represent them on TWN, but also how they are used. Some are in HTML, some are in MathML, and some are in browser dialogs. So not all of those can include tags for controlling the direction.

[pkra]

Ah, sorry! Yes, I should have mentioned that part as well.

[amire80]

... So yes, as @ebraminio says, when I translated MathJax to Hebrew in TWN, I used rlm in most places. It's the explicit rlm character, not an ampersand entity. It works, but it's not optimal, because it makes life hard for other translators, who will want to make new translations. I'm starting to think that the best way to include the commands in the message is to use parameters everywhere. That is, no explicit \commands in the messages at all. Wherever HTML can be used, they will be replaced by something like <code dir="ltr" class="tex-command">\command</code>, with the accompanying CSS saying
.tex-command {
unicode-bidi: -moz-isolate;
unicode-bidi: -webkit-isolate;
unicode-bidi: -ms-isolate;
unicode-bidi: -o-isolate;
unicode-bidi: isolate;
}

(dir="ltr" is specified in HTML rather than CSS according to W3C's recommendation; <code> is preferred to <bdi> for semantics.)

And where HTML cannot be used, lrm characters will be inserted. This way the translators won't have to insert the lrm characters manually.

[fred-wang]

Trying

      egrep "u200(e|f)" *.json

on the Hebrew translation, it seems that there are 33 messages using these marks, one is the MathML XML error and all the others seem to be TeX error messages. Most of them, but not all, are for TeX commands or environment names passed as parameters but not all.

So all these cases would be error messages (MathML merror) and so explicit marks must be used. Perhaps MathJax or the MathJax-i18n converter can do something to explicitly insert this mark in front of "\begin{...}", "\command" etc and the few other cases can be handled explicitly by the translators. Or perhaps we want a more general mechanism.

@pkra , @dpvc : I don't think this should be considered for v2.3?

[dpvc]

I agree that it can wait until after v2.3, but will need to be addressed then. We can probably do something like what you suggest for the macro names. It may be that we should be inserting those via %1, %2, etc, and can have a flag in the language data that indicates if the markers should be included (or just use the direction parameter for that).

[fred-wang]

So at the moment, I think the only RTL languages we have for v2.3 is Hebrew (with the explicit markers) and Farsi.

[fred-wang]

OK, so for MathJax 2.3 let's just add the markers by hand. I don't think we will have more translations for RTL languages before the release. I tried to semi-automatically add the markers for Farsi, based on what is done for Hebrew, but I'm not sure the method was really reliable as there are several exceptions. Another thing I'm note sure: why does Farsi use slashes in some cases and backslashes in other cases?

@ebraminio: can you please try http://devel.mathjax.org/testing/mathjax/MathJax-issue530/test/localization.html?locale=fa and tell me if that looks good to you?

@amire80 The Hebrew translation can be tested here: http://devel.mathjax.org/testing/mathjax/MathJax-issue530/test/localization.html?locale=he

[amire80]

Thanks for the links! I see a lot of issues there. I'll report them separately.

[ebraminio]

I've completed Persian translation of MathJax right now, I like to see a complete Persian translation will be shipped with v2.3, so at first please make the complete Persian translation available on that release (cc:@siebrand) then applying this enhancements would be nice.

[fred-wang]

OK, I think we will integrate Persian in v2.3, but MathJax-i18n is not up-to-date. Actually, I don't know from which TranslateWiki repository I have to fetch the data, so I'll let @siebrand do so.

Did you add the marker on the Persian translation?

[ebraminio]

I didn't, please reapply your patch on the updated translations again. Thanks

[amire80]

Reported #627. That's a separate issue, and it's more fundamental than this one.

[fred-wang]

@ebraminio @amire80 While we are on RTL, we wonder one thing during the development: would it be preferable to put the MathJax message box (at the bottom, that indicates Loading, Processing x% etc) on the right hand side for RTL languages? (If so, please open an issue for that)

[fred-wang]

@ebraminio OK, I've exported your translation from the TranslateWiki interface. Can you try http://devel.mathjax.org/testing/mathjax/MathJax-bidi/test/localization.html?locale=fa and check if there are not any issues (see #627 for the work-in-progress)

[ebraminio]

Looks good. Thanks :)

[fred-wang]

Thank you for the feedback :-)

[pkra]

Since #627 is fixed, do we consider this fixed?

[amire80]

Can I test it anywhere online?

Are TeX commands still hard-coded in the messages? In translatewiki.net I still see them in the messages.

[pkra]

#627 is part of MathJax v2.3, so just grab a copy of 2.3-latest or visit http://cdn.mathjax.org/mathjax/2.3-latest/test/localization.html?locale=he.

But I didn't read the thread carefully enough -- see this comment indicating we've postponed this past v2.3.