Add a legacy encoding for Azeri

[hsivonen]

• At least two implementers are interested (and none opposed):
  • Gecko: @hsivonen
  • …
• Tests are written and can be reviewed and commented upon at:
  • …
• Implementation bugs are filed:
  • Chrome: …
  • Firefox: Already handled.
  • Safari: …

(See WHATWG Working Mode: Changes for more details.)

Search for "Azerbaijan" in https://hsivonen.fi/chardetng/ for the story.

---

/parsing.html ( diff )

[hsivonen]

CC @whatwg/i18n

[hsivonen]

ced doesn't have windows-1254 higher than windows-1252 for the .az TLD, but it's unclear how self-referential those statistics are. That is, could the windows-1252 result arise from ced not having been well trained on Azeri?

macOS doesn't support setting Azeri as the primary language, so what Safari does at this time is moot.

[annevk]

@mfreed7 is this something you can look into for Chromium? Or find someone? 😊

[hsivonen]

Safari looks at the UI language as the spec currently suggests. (But Azeri cannot be set as the kind of language that Safari looks at.)

Firefox looks at the TLD. (The UI locale has no effect even for file: URLs.) In #7205, I'm handwavingly connecting ccTLDs to this language list.

If you put

46.226.111.119 hsivonen.tr
46.226.111.119 hsivonen.az

in /etc/hosts, swap the TLD in the URL https://hsivonen.com/test/moz/fallback-encoding.htm , and click through the certificate error, you can try the effect of the TLD in Firefox.

Chrome appears not to be sensitive to the TLD in the ASCII-only case. However, Chrome is sensitive to the TLD as a non-ASCII tie breaker: see https://hsivonen.tr/test/moz/fallback-encoding-non-ascii.htm with the above /etc/hosts mapping. Also, in the case of file: URLs, Chrome is sensitive to the UI locale (save https://hsivonen.com/test/moz/fallback-encoding-non-ascii.htm locally and try the Turkish locale).

I guess the practical implementation question for Chrome is whether Chrome is interested in tweaking the the ced TLD entry for .az and the ced language entry for Azeri to give more weight to windows-1254 than windows-1252. Answering that probably depends on how the values for those entries were chosen in the first place.

[hsivonen]

The relevant code for Firefox is within chardetng:
https://github.com/hsivonen/chardetng/blob/eae62e4c29c0f39f11811bfc0dc836209dd8773d/src/tld.rs#L70

[annevk]

If @mfreed7 and @hober can take a look at this before the next meeting agenda+ could be removed again.

[mfreed7]

So I generally support improving Azeri support. As for the particulars of what it would take to change ced to accomplish this, I'm not the expert. I'll reach out to our ced experts and ask them to take a look here.

Search for "Azerbaijan" in https://hsivonen.fi/chardetng/ for the story.

Or, if you're on a browser that supports scroll-to-text-fragment, just click here.

[mfreed7]

So I generally support improving Azeri support. As for the particulars of what it would take to change ced to accomplish this, I'm not the expert. I'll reach out to our ced experts and ask them to take a look here.

I finally reached some folks who are familiar with ced. This does indeed seem to be a relatively trivial change, as expected. I'm not sure why I didn't say this originally, but: I support this change to the spec. It makes sense and should be straightforward to implement. If you wouldn't mind, can you please file a Chromium bug for this?

[domenic]

Let's merge this!

@hsivonen, I assume this isn't really web-platform-testable? (Maybe it would be if they added fake .az domains, but that's probably a big can of worms.)

@mfreed7 also requested filing a Chromium bug, so if you could do that and link from the OP that'd be appreciated. Probably a WebKit one wouldn't hurt either.

[domenic]

Probably a WebKit one wouldn't hurt either.

Upon re-reading your earlier comment, I guess WebKit doesn't use TLD and only uses UI language, and doesn't let you set the UI language to Azeri. So a bug would not be useful to them, IIUC. Nevermind.