Implement BloomFilter skipping index building logic #242
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
dai-chen
requested review from
rupal-bq,
vmmusings,
penghuo,
anirudha,
kaituo and
YANG-DB
as code owners
vmmusings
reviewed
Feb 7, 2024
| * Bloom filter interface inspired by [[org.apache.spark.util.sketch.BloomFilter]] but adapts to | ||
| * Flint index use and remove unnecessary API. | ||
| */ | ||
| public interface BloomFilter { |
There was a problem hiding this comment.
Curious, Aren't there any opensource library implementations we can leverage.
is this because we need custom serialization to write to opensearch?
There was a problem hiding this comment.
Yes, indeed as the Javadoc, we're porting Spark's built-in BloomFilter, BitArray, Murmur3_x86_32 to our flint-core library. We only maintain the minimal API as needed and implement them within flint-core, considering the future possibility below:
- Integration with other query engine: We can implement bloom filter index in other query engine with the flint-core library
- User creates Flint index using our library in ingestion pipeline: We can add BloomFilter field type and user can generate Flint index at ingestion time
vmmusings
reviewed
Feb 7, 2024
...re/src/main/java/org/opensearch/flint/core/field/bloomfilter/classic/ClassicBloomFilter.java
Show resolved
Hide resolved
vmmusings
approved these changes
Feb 7, 2024
This was referenced Feb 8, 2024
5 tasks
This was referenced Mar 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is the first PR for BloomFilter skipping index support. This PR is focus on bloom filter building side and introduced core classes as below. Please read #206 for big picture, including final user experience, design decision, proof of concept and benchmark.
PR Planned
Documentation
Updated user manual: https://github.com/dai-chen/opensearch-spark/blob/add-bloom-filter-building-logic/docs/index.md#feature-highlights
Class Diagram
Issues Resolved
#206
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.