Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Star tree] Star tree merge changes #14626

Closed
bharath-techie opened this issue Jul 2, 2024 · 0 comments · Fixed by #14652
Closed

[Star tree] Star tree merge changes #14626

bharath-techie opened this issue Jul 2, 2024 · 0 comments · Fixed by #14652
Assignees
Labels
enhancement Enhancement or improvement to existing feature or request Indexing:Performance v2.17.0

Comments

@bharath-techie
Copy link
Contributor

bharath-techie commented Jul 2, 2024

Is your feature request related to a problem? Please describe

To support star tree index in OpenSearch, we need changes in the new composite codec [https://github.com//issues/14387] to build star tree index during merge flows.

Describe the solution you'd like

We will enhance the custom DocValuesFormat to support star tree creation during merge. We will override the merge method and merge the star tree indices across the segments.

The idea is to build the star tree from the existing star tree doc values from each segment instead of again building everything from scratch using segment documents.
We will basically merge all the documents which are not star documents [ essentially documents with values for all dimensions ] and we will build star tree and then we will derive star documents again from the tree.

This method optimizes the performance and resource utilization greatly and also ensures that newly created star tree documents during merge still adheres to 'maxLeafDocs' configuration.

Related component

Indexing:Performance

Describe alternatives you've considered

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing:Performance v2.17.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants