add sparse codec #1723

PSeitz · 2022-12-13T10:22:12Z

test null_index::dense::bench::bench_dense_codec_translate_dense_to_orig_90percent_filled_full_scan                ... bench:  50,590,455 ns/iter (+/- 24,920,729)
test null_index::dense::bench::bench_dense_codec_translate_orig_to_dense_10percent_filled_random_stride            ... bench:     542,766 ns/iter (+/- 8,079)
test null_index::dense::bench::bench_dense_codec_translate_orig_to_dense_full_scan_10percent                       ... bench:  10,259,153 ns/iter (+/- 6,611,325)
test null_index::dense::bench::bench_dense_codec_translate_orig_to_dense_full_scan_90percent                       ... bench:  10,515,926 ns/iter (+/- 7,147,110)
test null_index::sparse::bench::bench_sparse_codec_translate_orig_to_sparse_10percent_filled_random_stride         ... bench:   5,232,288 ns/iter (+/- 3,810,411)
test null_index::sparse::bench::bench_sparse_codec_translate_orig_to_sparse_1percent_filled_random_stride          ... bench:   1,166,540 ns/iter (+/- 2,005,600)
test null_index::sparse::bench::bench_sparse_codec_translate_orig_to_sparse_5percent_filled_random_stride          ... bench:   1,318,146 ns/iter (+/- 1,663,206)
test null_index::sparse::bench::bench_sparse_codec_translate_orig_to_sparse_full_scan_10percent                    ... bench:  32,506,578 ns/iter (+/- 66,071,241)
test null_index::sparse::bench::bench_sparse_codec_translate_orig_to_sparse_full_scan_1percent                     ... bench:  15,320,323 ns/iter (+/- 479,625)
test null_index::sparse::bench::bench_sparse_codec_translate_sparse_to_orig_1percent_filled_full_scan              ... bench:      13,582 ns/iter (+/- 446)
test null_index::sparse::bench::bench_sparse_codec_translate_sparse_to_orig_1percent_filled_random_stride          ... bench:       1,031 ns/iter (+/- 7)
test null_index::sparse::bench::bench_sparse_codec_translate_sparse_to_orig_1percent_filled_random_stride_big_step ... bench:         109 ns/iter (+/- 0)

fastfield_codecs/src/null_index/sparse.rs

codecov-commenter · 2022-12-14T09:22:05Z

Codecov Report

Merging #1723 (0d1e901) into main (f9171a3) will increase coverage by 0.00%.
The diff coverage is 96.66%.

@@           Coverage Diff            @@
##             main    #1723    +/-   ##
========================================
  Coverage   94.06%   94.07%            
========================================
  Files         262      263     +1     
  Lines       49994    50518   +524     
========================================
+ Hits        47028    47525   +497     
- Misses       2966     2993    +27

Impacted Files	Coverage Δ
fastfield_codecs/src/null_index/mod.rs	`100.00% <ø> (ø)`
fastfield_codecs/src/null_index/dense.rs	`99.06% <93.33%> (-0.32%)`	⬇️
fastfield_codecs/src/null_index/sparse.rs	`96.74% <96.74%> (ø)`
fastfield_codecs/src/serialize.rs	`86.56% <100.00%> (ø)`
src/indexer/segment_updater.rs	`94.40% <0.00%> (-1.05%)`	⬇️
src/store/index/mod.rs	`97.83% <0.00%> (-0.55%)`	⬇️
src/schema/schema.rs	`98.78% <0.00%> (-0.14%)`	⬇️
sstable/src/block_reader.rs	`82.14% <0.00%> (+1.78%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

fastfield_codecs/src/null_index/sparse.rs

fulmicoton · 2022-12-15T02:39:35Z

fastfield_codecs/src/null_index/sparse.rs

+    // The number of vals so far
+    let mut offset = 0;
+    let mut sparse_codec_blocks = Vec::new();
+    let num_blocks = u16::from_le_bytes([data[data.len() - 2], data[data.len() - 1]]);


It would be nice to introduce

#[inline(always)] pub fn read_u16_at_offset(data: &[u8], offset: usize) -> u16 { assert!(offset+1 < data.len()); unsafe { read_u16_at_offset_unsafe(data, offset) } } #[inline(always)] pub unsafe fn read_u16_at_offset_unsafe(data: &[u8], offset: usize) -> u16 { unsafe { let u16_ptr: *const [u8; 2] = data.as_ptr().offset(offset as isize) as *const [u8; 2]; let u16_bytes: [u8; 2] = u16_ptr.read_unaligned(); u16::from_le_bytes(u16_bytes) } }

And use it everywhere

I would only use unsafe when we can see the difference in the benchmark

Let's forget the unsafe part then, we already have a different implementation of this in one place...
Can we extract it as a function mark it as inline and use it everywhere in this file.

fastfield_codecs/src/null_index/sparse.rs

fulmicoton · 2022-12-15T03:02:11Z

fastfield_codecs/src/null_index/sparse.rs

+
+/// Splits a idx into block index and value in the block
+fn split_in_block_idx_and_val_in_block(idx: u32) -> (u16, u16) {
+    let val_in_block = (idx % u16::MAX as u32) as u16;


Suggested change

let val_in_block = (idx % u16::MAX as u32) as u16;

let val_in_block = (idx % u16::MAX as u32) as u16;

That's a bug isn't it? u16::MAX is incorrect. We want 1 << 16

discussed offline, that was a measured decision to deal with the fact that
a len in a block defined that way takes values in [0..u16::MAX], so that it cannot be represented as a u16.

fulmicoton · 2022-12-15T03:04:33Z

fastfield_codecs/src/null_index/sparse.rs

+        let (block_idx, val_in_block) = split_in_block_idx_and_val_in_block(idx);
+        // There may be trailing nulls without data, those are not stored as blocks. It would be
+        // possible to create empty blocks, but for that we would need to serialize the number of
+        // values or pass them when opening


I'd rather do that actually.

It would add some fixed bytes to every sparse codec field

fulmicoton · 2022-12-15T03:06:23Z

fastfield_codecs/src/null_index/sparse.rs

+
+    /// Return the number of non-null values in an index
+    pub fn num_non_null_vals(&self) -> u32 {
+        self.blocks.last().map(|block| block.offset).unwrap_or(0)


We seem to use num_null_vals or num_non_null_vals in different places. (Index traits)

I'd rather settle to num_non_null_vals everywhere... What do you think?

yes, let's stick to that

can you make the modification in the ValueIndex trait(s?)

fastfield_codecs/src/null_index/sparse.rs

fulmicoton

See comments

fulmicoton · 2022-12-20T05:28:53Z

fastfield_codecs/src/null_index/sparse.rs

+fn value_addr(idx: u32) -> ValueAddr {
+    /// Static assert number elements per block this method expects
+    #[allow(clippy::assertions_on_constants)]
+    const _: () = assert!(ELEMENTS_PER_BLOCK == (1 << 16));


That's the first time I see this trick. Interesting.

fastfield_codecs/src/null_index/sparse.rs

fulmicoton

Good work. Please have a look at clippy comments before merging.

Co-authored-by: Paul Masurel <paul@quickwit.io>

* add sparse codec * Apply suggestions from code review Co-authored-by: Paul Masurel <paul@quickwit.io> * Apply suggestions from code review Co-authored-by: Paul Masurel <paul@quickwit.io> * Apply suggestions from code review Co-authored-by: Paul Masurel <paul@quickwit.io> * add the -1 u16 fix for metadata num_vals * add dense block encoding to sparse codec * add comment, refactor u16 reading Co-authored-by: Paul Masurel <paul@quickwit.io>

PSeitz requested a review from fulmicoton December 13, 2022 10:23

PSeitz force-pushed the sparse_dense_index branch 2 times, most recently from fbf4e86 to f7e5efa Compare December 14, 2022 08:03

fulmicoton reviewed Dec 14, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 14, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 14, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 14, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

PSeitz force-pushed the sparse_dense_index branch 3 times, most recently from 615c0c2 to 7bac90f Compare December 14, 2022 09:09

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Show resolved Hide resolved

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Show resolved Hide resolved

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Outdated Show resolved Hide resolved

fulmicoton reviewed Dec 15, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Show resolved Hide resolved

fulmicoton requested changes Dec 15, 2022

View reviewed changes

PSeitz force-pushed the sparse_dense_index branch from db4764d to f0bec47 Compare December 20, 2022 04:37

fulmicoton reviewed Dec 20, 2022

View reviewed changes

fastfield_codecs/src/null_index/sparse.rs Show resolved Hide resolved

fulmicoton approved these changes Dec 20, 2022

View reviewed changes

PSeitz and others added 3 commits December 20, 2022 17:16

add sparse codec

ac45a45

Apply suggestions from code review

2d4fe50

Co-authored-by: Paul Masurel <paul@quickwit.io>

Apply suggestions from code review

6ed11bc

Co-authored-by: Paul Masurel <paul@quickwit.io>

PSeitz and others added 3 commits December 20, 2022 17:16

Apply suggestions from code review

6d143c7

Co-authored-by: Paul Masurel <paul@quickwit.io>

add the -1 u16 fix for metadata num_vals

7b19fd2

add dense block encoding to sparse codec

c38bf95

PSeitz force-pushed the sparse_dense_index branch from f0bec47 to 7f12317 Compare December 20, 2022 09:16

add comment, refactor u16 reading

0d1e901

PSeitz force-pushed the sparse_dense_index branch from 7f12317 to 0d1e901 Compare December 20, 2022 12:01

PSeitz merged commit 2ac1cc2 into main Dec 20, 2022

PSeitz deleted the sparse_dense_index branch December 20, 2022 14:30

This was referenced Jan 13, 2023

truncation comment PSeitz/tantivy#30

Closed

use stats PSeitz/tantivy#31

Closed

PSeitz mentioned this pull request Jan 31, 2023

update lz4 flex PSeitz/tantivy#33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add sparse codec #1723

add sparse codec #1723

PSeitz commented Dec 13, 2022 •

edited

Loading

codecov-commenter commented Dec 14, 2022 •

edited

Loading

fulmicoton Dec 15, 2022 •

edited

Loading

PSeitz Dec 19, 2022

fulmicoton Dec 20, 2022

fulmicoton Dec 15, 2022

fulmicoton Dec 15, 2022

fulmicoton Dec 15, 2022

PSeitz Dec 15, 2022

fulmicoton Dec 15, 2022

PSeitz Dec 15, 2022

fulmicoton Dec 15, 2022

fulmicoton left a comment

fulmicoton Dec 20, 2022

fulmicoton left a comment

	let val_in_block = (idx % u16::MAX as u32) as u16;
	let val_in_block = (idx % u16::MAX as u32) as u16;

add sparse codec #1723

add sparse codec #1723

Conversation

PSeitz commented Dec 13, 2022 • edited Loading

codecov-commenter commented Dec 14, 2022 • edited Loading

Codecov Report

fulmicoton Dec 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fulmicoton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fulmicoton left a comment

Choose a reason for hiding this comment

PSeitz commented Dec 13, 2022 •

edited

Loading

codecov-commenter commented Dec 14, 2022 •

edited

Loading

fulmicoton Dec 15, 2022 •

edited

Loading