Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Concurrent Segment Search] Change ConcurrentSearchTasksIT to use custom slice count #9317

Open
jed326 opened this issue Aug 14, 2023 · 8 comments
Assignees
Labels
enhancement Enhancement or improvement to existing feature or request good first issue Good for newcomers :test Adding or fixing a test

Comments

@jed326
Copy link
Collaborator

jed326 commented Aug 14, 2023

Now that #9107 and #8884 have been merged, we should fix ConcurrentSearchTasksIT to not hard code the segment count and instead validate against the customer slice count.

Ref: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/internalClusterTest/java/org/opensearch/action/admin/cluster/node/tasks/ConcurrentSearchTasksIT.java#L111

@jed326 jed326 added enhancement Enhancement or improvement to existing feature or request untriaged :test Adding or fixing a test and removed untriaged labels Aug 14, 2023
@jed326 jed326 self-assigned this Aug 14, 2023
@jed326 jed326 removed their assignment Sep 26, 2023
@jed326 jed326 added the good first issue Good for newcomers label Nov 29, 2023
@dipeshsingh253
Copy link

Hi @jed326 , I would like to work on this issue. can you provide me with some context here? I visited this file but it looked like a simple test only.

@jed326
Copy link
Collaborator Author

jed326 commented Jan 8, 2024

Hey @dipeshsingh253 , thanks for the interest! You can refer to https://opensearch.org/docs/latest/search-plugins/concurrent-segment-search/#slicing-mechanisms for an explanation of how "slicing" works for concurrent search.

In this issue specifically we would like to convert the test from using the hard coded slice calculation here (the left side of the assertion statement):

// Concurrent search forks each slice of 5 segments to different thread (see please
// https://github.com/apache/lucene/issues/12498)
assertEquals((int) Math.ceil(getSegmentCount(INDEX_NAME) / 5.0) + 1, threadStats.size());

And instead set the slice count via the cluster settings and then use that set value in the test assertion.

@dipeshsingh253
Copy link

Hi @jed326 , thanks for your assistance and this opportunity. Can you provide some resource on how exactly we are going to get slice count via the cluster settings ?

@jed326
Copy link
Collaborator Author

jed326 commented Jan 11, 2024

Hi @dipeshsingh253, the cluster settings API can be used to get and set cluster settings: https://opensearch.org/docs/latest/api-reference/cluster-api/cluster-settings/

The other link I provided above has details on the specific cluster setting to be used.

@dipeshsingh253
Copy link

dipeshsingh253 commented Jan 11, 2024

Hi @jed326 , this was helpful but it still did not demonstrate how to use it in this project, for example :

assertEquals((int) Math.ceil(getSegmentCount(INDEX_NAME) / 5.0) + 1, threadStats.size()); 
//in above line we have to remove the static part and get something like below,
assertEquals((int) Math.ceil(getSliceCount(), threadStats.size()); 

so my doubt is how do we use Java code to set or get values in/from cluster setting? If you have any reference to such an example or resource, please point them here. Also if there is any existing file that uses the same sort of logic, please mention that as well.

Thanks in advance !!!

@rampreeth
Copy link

rampreeth commented Jul 11, 2024

Hi @jed326 , can I pick this up?

@rampreeth
Copy link

Hi @jed326 , please let me know if this can be picked up. Thanks!

@jed326
Copy link
Collaborator Author

jed326 commented Jul 15, 2024

Hey @rampreeth please go ahead!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request good first issue Good for newcomers :test Adding or fixing a test
Projects
Status: No status
Development

No branches or pull requests

3 participants