Collapse overlapped highlighted ranges #1473

akr4 · 2022-08-24T10:22:12Z

Hi. This fixes a problem that occurs when calling Snippet.to_html() and the snippet.highlighted are overlapped ranges like this:

[0..2, 1..3]

This type of ranges is generated by NgramTokenizer, for example.

Error

In that case, there will be an error on calling Snippet.to_html(), which is caused by slicing String with invalid indices.

begin <= end (2 <= 1) when slicing `abc`
thread 'snippet::tests::test_snippet_with_overlapped_highlighted_ranges' panicked at 'begin <= end (2 <= 1) when slicing `abc`', library/core/src/str/mod.rs:111:5
stack backtrace:
...
   9: tantivy::snippet::Snippet::to_html
             at ./src/snippet/mod.rs:84:44

Fix

This PR collapses overlapped ranges into non-overlapped ones to fix the problem like this:

[0..2, 1..3] -> [0..3]

adamreichold · 2022-08-24T10:36:24Z

src/snippet/mod.rs

+    }
+
+    let mut result = Vec::new();
+    let mut current = ranges[0].clone();


I am not sure but maybe it would be nicer to iterate only once, e.g.

let mut result = Vec::new(); let mut ranges = ranges.iter(); let mut current = match ranges.next() { Some(range) => range.clone(), None => return result, }; for range in ranges { ...

Sounds good. I changed so. Thanks.

fulmicoton · 2022-08-24T11:03:20Z

src/snippet/mod.rs

@@ -186,6 +186,29 @@ fn select_best_fragment_combination(fragments: &[FragmentCandidate], text: &str)
    }
 }

+/// Returns ranges that are collapsed into non-overlapped ranges.


Please add your example in the comment.

fulmicoton · 2022-08-24T11:06:58Z

src/snippet/mod.rs

@@ -186,6 +186,29 @@ fn select_best_fragment_combination(fragments: &[FragmentCandidate], text: &str)
    }
 }

+/// Returns ranges that are collapsed into non-overlapped ranges.
+fn collapse_overlapped_ranges(ranges: &Vec<Range<usize>>) -> Vec<Range<usize>> {


Suggested change

fn collapse_overlapped_ranges(ranges: &Vec<Range<usize>>) -> Vec<Range<usize>> {

fn collapse_overlapped_ranges(ranges: &[Range<usize>]) -> Vec<Range<usize>> {

&Vec is almost never a good idea.

Another solution would be for you to pass an iterator directly.

impl Iterator<Item=Range<usize>>

src/snippet/mod.rs

fulmicoton · 2022-08-24T11:09:07Z

src/snippet/mod.rs

+
+    for range in ranges {
+        if current.end > range.start {
+            current = current.start..range.end;


nitpick... I feel the following is easier but that's your call.

Suggested change

current = current.start..range.end;

current.end = range.end;

fulmicoton · 2022-08-24T11:11:53Z

src/snippet/mod.rs

+/// Returns ranges that are collapsed into non-overlapped ranges.
+fn collapse_overlapped_ranges(ranges: &Vec<Range<usize>>) -> Vec<Range<usize>> {
+    let mut result = Vec::new();
+    let mut ranges = ranges.iter();


nitpick... I'm not fond of shadowing.
It is sometime useful to prevent the usage of a variable.

Here I would either call it ranges_it, or pass an Iterator as argument.

fulmicoton · 2022-08-24T11:13:32Z

@akr4 Awesome job spotting the job and fixing it! I had a bunch of rather minor comments.

codecov-commenter · 2022-08-24T11:30:10Z

Codecov Report

Merging #1473 (b6d5aa6) into main (82209c5) will increase coverage by 0.01%.
The diff coverage is 98.18%.

@@            Coverage Diff             @@
##             main    #1473      +/-   ##
==========================================
+ Coverage   94.07%   94.09%   +0.01%     
==========================================
  Files         239      239              
  Lines       44627    44833     +206     
==========================================
+ Hits        41983    42185     +202     
- Misses       2644     2648       +4

Impacted Files	Coverage Δ
src/snippet/mod.rs	`91.08% <98.18%> (+1.05%)`	⬆️
src/fastfield/serializer/mod.rs	`88.94% <0.00%> (-1.54%)`	⬇️
bitpacker/src/bitpacker.rs	`96.03% <0.00%> (-1.00%)`	⬇️
src/reader/mod.rs	`92.94% <0.00%> (-0.05%)`	⬇️
src/fastfield/mod.rs	`94.76% <0.00%> (-0.01%)`	⬇️
src/fastfield/gcd.rs	`100.00% <0.00%> (ø)`
fastfield_codecs/src/bitpacked.rs	`100.00% <0.00%> (ø)`
fastfield_codecs/src/multilinearinterpol.rs
fastfield_codecs/src/linearinterpol.rs
fastfield_codecs/src/blockwise_linear.rs	`100.00% <0.00%> (ø)`
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

NgramTokenizer with different min and max generates such ranges.

akr4 · 2022-08-24T11:40:49Z

@fulmicoton Thank you very much for your review. I will make changes tomorrow.

…ould be sorted, and debug_assert! that.

akr4 · 2022-08-25T08:03:01Z

Committed some changes. The main ones are changing the argument from &Vec to &[ ] and adding a comment. What do you think of this?

fulmicoton

Approved! Thank you @akr4

fulmicoton · 2022-08-25T15:17:39Z

@akr4 Can you run make fmt? (It requires rust nightly)

akr4 · 2022-08-25T23:22:49Z

@fulmicoton Done.

Collapse overlapped highlighted ranges

23f0057

adamreichold reviewed Aug 24, 2022

View reviewed changes

akr4 added 2 commits August 24, 2022 19:44

Iterate once

3c7b22d

Delete unused import and debug print

3907f5a

fulmicoton reviewed Aug 24, 2022

View reviewed changes

src/snippet/mod.rs Show resolved Hide resolved

fulmicoton reviewed Aug 24, 2022

View reviewed changes

Fix the problem in the case the next range's end is less than previous.

9c5e75a

NgramTokenizer with different min and max generates such ranges.

akr4 added 2 commits August 25, 2022 16:31

Use ref of slice for the parameter instead of &Vec

990c24b

Add doc comments on collapse_overlapped_ranges about the parameter sh…

b6d5aa6

…ould be sorted, and debug_assert! that.

fulmicoton approved these changes Aug 25, 2022

View reviewed changes

make fmt

c813c86

fulmicoton merged commit 17093e8 into quickwit-oss:main Aug 26, 2022

This was referenced Jan 13, 2023

truncation comment PSeitz/tantivy#30

Closed

use stats PSeitz/tantivy#31

Closed

PSeitz mentioned this pull request Jan 31, 2023

update lz4 flex PSeitz/tantivy#33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Collapse overlapped highlighted ranges #1473

Collapse overlapped highlighted ranges #1473

akr4 commented Aug 24, 2022 •

edited

Loading

adamreichold Aug 24, 2022

akr4 Aug 24, 2022

fulmicoton Aug 24, 2022 •

edited

Loading

fulmicoton Aug 24, 2022

fulmicoton Aug 24, 2022

fulmicoton Aug 24, 2022

fulmicoton Aug 24, 2022

fulmicoton commented Aug 24, 2022

codecov-commenter commented Aug 24, 2022 •

edited

Loading

akr4 commented Aug 24, 2022

akr4 commented Aug 25, 2022

fulmicoton left a comment

fulmicoton commented Aug 25, 2022

akr4 commented Aug 25, 2022

	fn collapse_overlapped_ranges(ranges: &Vec<Range<usize>>) -> Vec<Range<usize>> {
	fn collapse_overlapped_ranges(ranges: &[Range<usize>]) -> Vec<Range<usize>> {

Collapse overlapped highlighted ranges #1473

Collapse overlapped highlighted ranges #1473

Conversation

akr4 commented Aug 24, 2022 • edited Loading

Error

Fix

adamreichold Aug 24, 2022

Choose a reason for hiding this comment

akr4 Aug 24, 2022

Choose a reason for hiding this comment

fulmicoton Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

fulmicoton Aug 24, 2022

Choose a reason for hiding this comment

fulmicoton Aug 24, 2022

Choose a reason for hiding this comment

fulmicoton Aug 24, 2022

Choose a reason for hiding this comment

fulmicoton Aug 24, 2022

Choose a reason for hiding this comment

fulmicoton commented Aug 24, 2022

codecov-commenter commented Aug 24, 2022 • edited Loading

Codecov Report

akr4 commented Aug 24, 2022

akr4 commented Aug 25, 2022

fulmicoton left a comment

Choose a reason for hiding this comment

fulmicoton commented Aug 25, 2022

akr4 commented Aug 25, 2022

akr4 commented Aug 24, 2022 •

edited

Loading

fulmicoton Aug 24, 2022 •

edited

Loading

codecov-commenter commented Aug 24, 2022 •

edited

Loading