[Linux] Inconsistently parsed fontconfig causes font selection problems

[13rac1]

The Fontconfig Edit Mode: prepend_first is, probably, causing a number of font selection issues #35, #37, and #39. The existing/provided configuration is technically correct according to all tests I've tried, but it is still causing problems in programs which parse the fontconfig themselves. Issue: prepend_first does not work as expected in fontconfig.

Actual 56-emojione-color.conf

  <match target="pattern">
    <!-- If the requested font is sans-serif -->
    <test qual="any" name="family">
        <string>sans-serif</string>
    </test>
    <!-- Make Bitstream Vera Sans the first result -->
    <edit name="family" mode="prepend_first"> 
      <string>Bitstream Vera Sans</string>
    </edit>
    <!-- Followed by EmojiOne Color -->
    <edit name="family" mode="prepend_first">     <!-- THIS! -->
      <string>EmojiOne Color</string>
    </edit>
  </match>

Actual Result:

$ fc-match -s sans
Vera.ttf: "Bitstream Vera Sans" "Roman"
EmojiOneColor-SVGinOT.ttf: "EmojiOne Color" "Regular"
DejaVuSans.ttf: "DejaVu Sans" "Book"

Expected, Result:

$ fc-match -s sans
EmojiOneColor-SVGinOT.ttf: "EmojiOne Color" "Regular"
Vera.ttf: "Bitstream Vera Sans" "Roman"
DejaVuSans.ttf: "DejaVu Sans" "Book"

Why the "Expected Result"? prepend_first seems like it would be applied in order, so the first font listed would become the second font in the list when a second prepend_first is encountered. It doesn't work that way though. The first font stays the first font.

Need to find a configuration that works for everything.

[13rac1]

Possible "solution": Use the DejaVu Sans number glyphs.

(quotes intentional)
This doesn't actually fix the problem, but it would solve, aka mask, the issue for the vast majority of people.

[13rac1]

A good example of what the disabled glyphs cause by @kgizdov, further details in #46 :

[kgizdov]

Following your comment in #46 about ligatures. As far as I understand how these work, you need two characters to form one. In this case you need a number followed by the emoji glyph. So what I still don't understand is why would this make a number appear as the emoji glyph even if the application is loading the wrong font. If the font has the number and the emoji glyph, it should be also able to provide only the number when requested, no? I'm not an expert, but couldn't this be caused by the font always returning the glyph, even when only the number is requested?

[13rac1]

Ah, I'm not explaining it well enough. :)

Terminology (Rough):

• Glyph = The vector shape used to draw a character on screen. There are b&w and SVG color glyphs in this font.
• Code point = The Unicode ID number of a specific character or symbol
• Character = Generally, the representation of a code point drawn using the data referenced in a glyph.
• Ligature = A combination of code points replaced by a single glyph.

If the font has the number and the emoji glyph, it should be also able to provide only the number when requested, no?

Originally, I had to put something there at the 0x30 through 0x39 unicode code points, so I made the keypad numbers into "emoji-style" numbers. The new version of EmojiOne now includes "emoji-style" numbers at those code points so, my custom SVGs are no longer needed. Here's one (haha) from the removal commit listed above: https://github.com/eosrei/emojione-color-font/blob/e1a34da33493ee2966012cc6467ad519dfc518d3/assets/emojione-svg/disabled/0031.svg

There must be a glyph at 0x30 and 0x20e3 in the font providing the ligature for the 0x30-0x20e3 ligature to work. When EmojiOne Color is selected incorrectly as default by MPV, Steam, Evolution, etc then they load the EmojiOne 0x30 instead of the DejaVu 0x30. These, AFAIK, are bugs in all three programs.

Solutions:

• Create a Fontconfig which works everywhere. Current fontconfig: https://github.com/eosrei/emojione-color-font/blob/e1a34da33493ee2966012cc6467ad519dfc518d3/linux/fontconfig/56-emojione-color.conf How to fix?

Workarounds:

• Export the DejaVu Sans glyphs and use those for 0x23, 0x2a, and 0x30-0x39. I'd rather not do this.

[kgizdov]

I think you've explained it very well, but I also think you're not doing what the UTF standard asks. UTF implements ASCII and 0x30 through 0x39 (and other ASCII codes) should be represented the same in the wider 0x0030 - 0x0039 by any font following the standard. So having the glyphs for emoji numbers in the places for normal numbers is wrong. The glyph (as the standard requires http://www.fileformat.info/info/emoji/zero/index.htm), for emoji number 0 for example, should only be returned if codes 0x0030 and 0x20e3 are requested together in succession. Otherwise 0x0030 should be just the normal number 0 glyph. Why not draw your own numbers? You don't have to import DejaVu glyphs. But having the number glyphs is obviously needed to actually be following the spec.

[13rac1]

I was using 0x30 as the short form of 0x0030 since they are the same number and the extra zeros are redundant. Nothing about ASCII intended.

for emoji number 0 for example, should only be returned if codes 0x0030 and 0x20e3 are requested together in succession

That's exactly how the font works.

So having the glyphs for emoji numbers in the places for normal numbers is wrong.

Yes, the font doesn't do that.

Why not draw your own numbers?

EmojiOne already has it's own representation of regular numbers: https://github.com/Ranks/emojione/blob/8e4691229ed434834fb2fa97888d596986d3c636/assets/svg/0030.svg and I've disabled them: https://github.com/eosrei/emojione-color-font/tree/master/assets/emojione-svg/disabled which is why we are discussing this.

But having the number glyphs is obviously needed to actually be following the spec.

Yes, but I cannot, because then I have to re-open all the issues about MPV, Steam, and Evolution. Plus, I expect I'd get many more. I've described the ideal solution which I don't know how to solve and potential workaround which I'd rather avoid. Any help on a Fontconfig which will work everywhere is greatly appreciated. Thanks.

[13rac1]

I also think you're not doing what the UTF standard asks.

Please dissemble V1.2 of the font and check. As far as I know, I am following both the UTF and OpenType spec, except I have disabled the emoji numbers aka keycap numbers due to all listed issues.

[kgizdov]

OK, I went and opened up the fonts to see what's going on.
No issue with 0x20e3:

But as suspected 0x0030 is strange in v1.1:

In fact many of the symbols which are plain text ASCII have emoji glyphs. For contrast here are some screenshots from Google Noto Emoji, Bitstream Vera, DejaVu

Noto:

Bitstream Vera:

DejaVu:

Having started on this, I also noticed the following issue in Chrome:

Google's Noto answer to this is like that:

Now I understand that the issue arises from the fact that the UTF codes for these correspond to the fallback outline emoji and in Firefox and other apps that use the proper colour emoji this is not a problem. But I believe, this is exactly the point. Now, I do not want to tell you how to make your font. You seem to have done a pretty good job as it is and I like it very much. However, I still think that the standard requires the ASCII codes to stand for ASCII type characters with ASCII type glyphs.

Thus I am of the opinion the proper way to handle this is the way Google's Noto does it. The outline Noto Emoji font has the ASCII glyphs for the characters in question and the Noto Color Emoji has the emoji type once. To add to this, I am not sure how the Noto Color emoji font works exactly, as I could not open it neither in FontForge nor in BirdFont. This is probably where your expertise fills in the gaps to have the outline and color emoji fonts work as the user expected.

However you decide to solve this, I just want to say - thank you very much for the font, it is the default for me and I am overall very satisfied with it. 😜

[13rac1]

😄 These are not emoji:

They are the regular glyphs for numbers in EmojiOne (the graphic set). A stylistic representation of a number does not make it "emoji". Here are some examples of other fonts with far more interesting number representations.

Extra Sprinkles: http://www.dafont.com/extrasprinkles.font

Racing Numbers: http://www.dafont.com/racingnumbers.font

Solarium: http://www.dafont.com/solarium.font

There are a hundred thousand more examples just on on dafont.com. The number glyphs used in V1.1 of this font are correct.

Thus I am of the opinion the proper way to handle this is the way Google's Noto does it.

I argue it's a workaround, not a proper solution, but that's why I have it listed above.

Noto Color emoji font ... I could not open it neither in FontForge nor in BirdFont.

It doesn't follow the OpenType spec. It only has the CBLC-CBDT color glyphs. It doesn't contain any regular b&w glyphs, so Fontforge can't do anything with it.

[kgizdov]

Recently, I have noticed that if I have the font installed, Evince will not show greek characters (for math equations) correctly. The weird thing is, if I regenerate the PDFs from LaTeX source on my machine, it all works fine, but if it is generated elsewhere the characters are then missing when viewed on my machine. I have tested with PDF generated in LaTeX on Ubuntu 14.04 without EmojiOne installed and on my Arch Linux with EmojiOne installed. Not sure if that's Evince's fault or some font-config issue. If I uninstall EmojiOne, then I can see the characters but they look wierd - as if they are rendered with the wrong font face.

EDIT: It's definitely not Evince. I confirmed the same issue with other viewers.

[13rac1]

Can you check Character Map? If you right click on a symbol it will tell you which font the gylph is coming from.

[kgizdov]

When in Character Map and having selected EmojiOne Color as font, the characters in question appear as DejaVu Sans.

[13rac1]

And otherwise? That's what it should be.

[13rac1]

It's definitely not Evince. I confirmed the same issue with other viewers.

Are they using a different backend library or the same? What about PDF.js in Firefox?

[kgizdov]

I think the proper font should be Arial - Italic to be exact. I'll have to double check PDF.js a bit later and come back to you.

[13rac1]

If I uninstall EmojiOne, then I can see the characters but they look wierd - as if they are rendered with the wrong font face.

Arial isn't installed on my machine for example. You need to embed the fonts in the PDF.

[kgizdov]

So I tried both Chrome and Firefox. I also inspected the fonts in the document. Here's what I see.

• None of the fonts are embedded. They are listed as - Times New Roman, Helvetica, Courier New, ZapfDingbats, Symbols (not sure which this is exactly).
• They are substituted by NumbusRoman from NimbusRoman.otf and Bitstream Vera from Vera.ttf.
• Chrome and Firefox render the chars correctly
• Firefox uses Bitstream Vera for the whole document
• Chrome uses Roboto and Noto

And here's the important bit:

• if I remove EmojiOne the substitution fonts in Evince become - Numbus for Times New Roman, Liberation Mono, DejaVu Sans for ZapfDingbats and Symbols.
• The change is Courier from Bitstream Vera to Liberation Mono and Symbol and ZapfDingbats from Bitstream Vera to DejaVu Sans

I hope that helps.

[13rac1]

This is yet-another example of why this overall issue needs to exist.

The solution in your case is to embed the fonts in the PDF. It's the only way to make PDFs look consistent everywhere. Just like web development, you cannot trust the client! Here is Adobe's description of how/why: https://helpx.adobe.com/acrobat/using/pdf-fonts.html

I have no idea about a solution for the overall issue, but that is why I have this issue labeled "Help Wanted". Font configuration issues on all three OS have now take far more time than the actual font itself. I had no idea what I was getting into when I started this project. Haha! So many bugs in everything.

[kgizdov]

You will probably also like the fact that if I install the xorg-fonts, which some people say will help with this, the substitution font becomes EmojiOne. This of course is even worse... I'm not sure what to do. Is there a way to specify order of font fallback per application or something like that. The Arch Wiki seems to be kind of unclear on that front.

[13rac1]

Is there a way to specify order of font fallback per application or something like that.

Yes, you can, but you'd be adding more configuration to work around these bugs which could cause additional issues. The fontconfig docs are tough: https://www.freedesktop.org/software/fontconfig/fontconfig-user.html

[kgizdov]

As a workaround for some issues, I have managed to follow this hint to install tex-gyre-fonts and resolve several issues. Almost all of the wrongly substituted fonts are now correctly substituted with TexGyre (or the appropriate otherwise font). However, the outstanding issue for me is that ZapfDingbats, Symbol are still wrongly substituted by Bitstream Vera and the only way to resolve this is to actually install the Adobe fonts. Not sure, if TexGyre will give you any extra clues.

[13rac1]

No workaround is needed. Adobe's recommended solution is to embed the fonts. Your link says that also:

You may also experience similar problem when you open a PDF which requires Helvetica but does not have it embedded for viewing.

[kgizdov]

The workaround is needed when I want to read someone elses PDF that doesn't embed the fonts.

[13rac1]

Ah ha! Haha, good point :D