How to Split words on Hyphen ?

[jfgirard]

I noticed a difference in the way words are tokenized compared to Postgresql.

select to_tsvector('Pseudo-Mercator');
"'mercat':3 'pseudo':2 'pseudo-merc':1"

Basically, PG index both the "Pseudo-Mercator" and the sub words "Pseudo" and "Mercator".

Searching for "mercator" gives me a result with PG.

But, because Lunr only tokenize on white char, a search for "mercator" won't work.

I could create a afterTokenizer function to split each token and add them to the list.

function afterTokenizer(tokens) {
     var split;
     tokens.forEach(function(token){
     split = token.split(/-/g);
     if(split.length > 1){
         tokens = tokens.concat(split);
      }
     });
     return tokens;
}

So, index.pipeline.run(lunr.tokenizer(text)); would be index.pipeline.run(afterTokenizer(lunr.tokenizer(text)));

Is this the best way to acheive the same behavior ?

[nolanlawson]

Yes, that seems like a good solution.

For what it's worth, I'm pretty shocked that Lunr doesn't split on anything but whitespace. (Commas? Semicolons?) We should file a bug on Lunr.

[nolanlawson]

OK, I see now how it works. It's not just whitespace; it's also trailing and leading non-letters. (Source)[https://github.com/olivernn/lunr.js/blob/master/lib/trimmer.js#L22-L23].

So yeah, it just doesn't work for punctuation within a word, such as "Pseudo-Mercator". Depending on your use case that may or may not be okay.

[jfgirard]

Yes, the trim works correctly. But the tokenizer only splits on white char (\s).
https://github.com/olivernn/lunr.js/blob/master/lib/tokenizer.js#L28

It looks like its standard to split on hypen, both PG and Couchdb-Lucene do it.

For C-L, I did a simple test indexing the name property and query it like that:
localhost:5985/local/epsg/_design/cl/by_name?q=mercator&include_docs=true finds the doc with "pseudo-mercator".

[nolanlawson]

Opened a PR on Lunr: olivernn/lunr.js#98.

[jfgirard]

Great! Better fixing it upstream.

[nolanlawson]

5d0207a