Fix a bug introduced in Synapse v1.74.0 where searching with colons when using ICU for search term tokenisation would fail with an error.

[reivilibre]

Fixes: #14815

Introduced by: #14464 (as far as I can tell: it changed the tokenisation rules to leave :s in when you have ICU installed.)

Base: develop

This pull request is commit-by-commit review friendly.

Original commit schedule, with full messages:

1. Add failing test

2. Small rename of variables

3. Add minitests to characterise how we do tokenisation today

4. Add escaping to our tsquery builder

5. Test that we can find a user by prefix of MXID
   (I wasn't super confident that the ICU tokeniser wouldn't break this, but a bit of poking later,

   I think I'm happy. But thought it should be tested anyway!)

[babolivier]

I'm quite surprised this is an issue with libicu. From testing on my local machine libicu seems to correctly tokenise the string foo:bar into foo, :, bar (and then the colon is removed from the set of results by _parse_words_with_icu's check against the ([\w\-]+) regex):

>>> s = "foo:bar"
>>> import icu
>>> breaker = icu.BreakIterator.createWordInstance(icu.Locale.getDefault())
>>> breaker.setText(s)
>>> b = breaker.nextBoundary()
>>> result = s[0:b]
>>> result
'foo'
>>> b2 = breaker.nextBoundary()
>>> s[b:b2]
':'
>>> import re
>>> re.findall(r"([\w\-]+)", s[b:b2], re.UNICODE)
[]
>>> 

So with this logic, _parse_words should return [foo, bar] when ICU support is available (and also when it's not fwiw).

(though I'll admit I just tinkered with pyicu in the python interpreter and not with an actual instance Synapse so it's fairly likely I've missed something)

[reivilibre]

I'm quite surprised this is an issue with libicu. From testing on my local machine libicu seems to correctly tokenise the string foo:bar into foo, :, bar (and then the colon is removed from the set of results by _parse_words_with_icu's check against the ([\w\-]+) regex):

I didn't realise this to start with of course, but on CI and on my local machine, ICU is giving two different tokenisations. Possibly the underlying version of libICU itself is different?

edit: I'm not even sure my homeserver runs with ICU though, so maybe there's more to it than meets the eye.

edit: actually it does run with ICU. I don't remember installing it.

[babolivier]

Possibly the underlying version of libICU itself is different?

Possibly yeah, though I'd be a bit surprised if the tokenisation differs so much between versions. Not sure if it's worth investigating. Agreed it's weird that it gives you different results between CI and your local machine though.

edit: actually it does run with ICU. I don't remember installing it.

If you install Synapse with the all extra (or use either Docker or the debs), pyicu will be installed in Synapse's venv. iirc libicu itself is installed by default on a number of systems and distros.

[reivilibre]

If you install Synapse with the all extra (or use either Docker or the debs), pyicu will be installed in Synapse's venv. iirc libicu itself is installed by default on a number of systems and distros.

Yeah, indeed, but I don't use [all] in my installation tool. But it's possible I did it manually once when trying a git branch for something experimental and ICU has stuck around since.

[reivilibre]

I'd be a bit surprised if the tokenisation differs so much between versions. Not sure if it's worth investigating. Agreed it's weird that it gives you different results between CI and your local machine though.

Yes and no. My distro has 2 different versions of ICU packaged which is unusual unless they're known to be substantially different (smells like some software is only able to cope with one of the two). As for CI vs local machine, CI is running Ubuntu 20.04 and I'm on 22.04 (the major version of ICU is also different by a few versions). So I'm not entirely shocked.

Running your example:

Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "foo:bar"
>>> import icu
>>> breaker = icu.BreakIterator.createWordInstance(icu.Locale.getDefault())
>>> breaker.setText(s)
>>> b = breaker.nextBoundary()
>>> result = s[0:b]
>>> result
'foo:bar'
>>> b2 = breaker.nextBoundary()
>>> s[b:b2]
''
>>> import re
>>> re.findall(r"([\w\-]+)", s[b:b2], re.UNICODE)
[]
>>> 

plainly different. Maybe it makes sense? I think I've seen some languages use : in words to denote multiple variants, or something along those lines. I guess tokenisation is hard.

edit: or maybe using the 'default' locale is making a difference

[babolivier]

edit: or maybe using the 'default' locale is making a difference

Yeah I was going to comment that - maybe the default locale is the differentiating factor here.

[DMRobertson]

Looks sane, but one point to check:

[DMRobertson]

Is this test (and any others) running under with-ICU and without-ICU? (Should it?)

[reivilibre]

now it is :)