auto convert certain Unicode Latin characters to the ASCII equivalents

[SuperStormer]

For example, all of the bold, italics, etc, letters here https://www.compart.com/en/unicode/block/U+1D400 which are sometimes used instead of ASCII letters in YT titles to make it look fancy or something. I have personally never seen any of them used for any legitimate purpose in a title.

Current list of applicable characters:

• "Mathematical Alphanumeric Symbols"
• fullwidth characters
• letter emojis
• small caps

[ajayyy]

I have a feeling chat gpt could be useful to get a list of character maps

[mchangrh]

should this also be used for replacing 🅱️, α Α (Alpha) and similar glyphs that aren't transformation of a letter but are commonly used as replacements?

[ajayyy]

ooh, interesting. The new emoji cleaning probably broke a lot of 🅱️ titles

[ajayyy]

Yea...

[ajayyy]

I think we should exclude letter emojis until this issue is complete

[mchangrh]

We might have to add exclusions for B, A and AB since they are in the pictograph range iirc

[ajayyy]

Added

[SuperStormer]

A partial solution would be NFKC normalization (String#normalize in JS).

list of character replacements: out.txt

import sys
import unicodedata
from pprint import pprint

chrs = [chr(c) for c in range(sys.maxunicode+1)]
pprint([(c,unicodedata.normalize("NFKC",c),unicodedata.name(c,None),unicodedata.name(unicodedata.normalize('NFKC',c)[0],None)) for c in chrs if unicodedata.normalize('NFKC', c)!=c])

Note that this doesn't fix letter emojis, nor small caps.

[mchangrh]

I've used NFKD and NFKC normalization in other projects before, It creates a mess out of emojis. I prefer https://github.com/gc/confusables for normalization using a map

[ajayyy]

There's a JS function to normalize strings and one of the modes should work well for this ("𝗠𝘆 𝗖𝗼𝗼𝗹 𝗧𝗶𝘁𝗹𝗲".normalize("NFKD") results in "My Cool Title")

#283