better remove_ordinals regex pattern #96
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -148,8 +148,8 @@ function generate(){ | |||
| }, | |||
| "remove_ordinals" : { | |||
| "type" : "pattern_replace", | |||
| "pattern": "(([0-9])(st|nd|rd|th))", | |||
| "replacement": "$2" | |||
| "pattern": "(([0-9]*1)st?|([0-9]*2)nd?|([0-9]*3)rd?|([0-9]*[456789])th?|([0-9]+0)th?|([0-9]*1[0-9])th?)", | |||
There was a problem hiding this comment.
This is quite the regex. Looks like it does what you want though
|
This looks like an improvement over the old regex for sure. I'd be very interested to do some testing of this on dev. Somewhat related: have we tested not removing ordinals? There's definitely a big signal difference between 5th and 5. Just like in #93 it's not possible to find places named "whole food market" with kstemming because places named "whole foods" evaluate to the same tokens, having 5 and 5th indistinguishable seems less than ideal. |
Merged
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
see #94 for more info.
@orangejulius could you review this plz?
closes #94
[edit] so after much yak shaving I have come up with the perfect megaregex
replace( "(?i)((^| )((1)st?|(2)nd?|(3)rd?|([4-9])th?)|(([0-9]*)(1[0-9])th?)|(([0-9]*[02-9])((1)st?|(2)nd?|(3)rd?|([04-9])th?))($| ))", "$2$4$5$6$7$9$10$12$14$15$16$17$18"...which makes my eyes hurt but makes all the tests pass.
@trescube suggested we should maybe leave it a bit more relaxed so that users who are not perfect at english can still find what they need.
so there are 2 options, either we go with the super complex one above or we modify the existing one to support autocomplete with the simpler:
replace( "(([0-9])(st?|nd?|rd?|th?))", "$2" )or similar