Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing composite aggregations with correct parameters #967

Merged

Conversation

VachaShah
Copy link
Collaborator

@VachaShah VachaShah commented May 2, 2024

Description

Introduced classes for composite aggregations for terms, datehistogram, histogram and geotilegrid. I looked in the server code from https://github.com/opensearch-project/OpenSearch/tree/main/server/src/main/java/org/opensearch/search/aggregations/bucket/composite.

Issues Resolved

Fixes #957

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vacha Shah <vachshah@amazon.com>
@VachaShah
Copy link
Collaborator Author

I am working on fixing/improving the test I added in this PR.

Signed-off-by: Vacha Shah <vachshah@amazon.com>
@VachaShah
Copy link
Collaborator Author

I am wondering if I missed something in the code or I am using it incorrectly in tests 🤔 Looking through server code, it looks like the code is matching but could use another pair of eyes. @dblock @reta

@dblock
Copy link
Member

dblock commented May 2, 2024

I am wondering if I missed something in the code or I am using it incorrectly in tests 🤔 Looking through server code, it looks like the code is matching but could use another pair of eyes. @dblock @reta

Add a working sample to samples, maybe start an aggregations guide if we don't have anything n that, this way you know whether the code works?

Also do check what's available in https://github.com/opensearch-project/opensearch-api-specification for this, good opportunity to fix/add as we have an incoming code generator in #366.

@reta
Copy link
Collaborator

reta commented May 2, 2024

Thanks @VachaShah , I think it looks good model wise

@VachaShah
Copy link
Collaborator Author

I am wondering if I missed something in the code or I am using it incorrectly in tests 🤔 Looking through server code, it looks like the code is matching but could use another pair of eyes. @dblock @reta

Add a working sample to samples, maybe start an aggregations guide if we don't have anything n that, this way you know whether the code works?

Also do check what's available in https://github.com/opensearch-project/opensearch-api-specification for this, good opportunity to fix/add as we have an incoming code generator in #366.

I will make sure to add samples and guides and will look through the specs as well!

@VachaShah
Copy link
Collaborator Author

Thanks @VachaShah , I think it looks good model wise

Thanks @reta! Its weird why the test that I added doesn't fail here, it fails for me on my local 🤔

@reta
Copy link
Collaborator

reta commented May 2, 2024

Thanks @reta! Its weird why the test that I added doesn't fail here, it fails for me on my local 🤔

Hm ... I am not sure, could look later on if you need a hand!

@VachaShah
Copy link
Collaborator Author

Thanks @reta! Its weird why the test that I added doesn't fail here, it fails for me on my local 🤔

Hm ... I am not sure, could look later on if you need a hand!

Thank you @reta! Yeah I still have test failures on my local but the workflows here are green, my failure is:

org.opensearch.client.opensearch.integTest.restclient.SearchRequestIT > shouldReturnSearchResultsWithCompositeAgg FAILED
    org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [x_content_parse_exception] [1:95] [composite] failed to parse field [sources]
        at __randomizedtesting.SeedInfo.seed([8D5592150CC0C93:3514B0F383EF841C]:0)
        at app//org.opensearch.client.transport.Endpoint.exceptionConverter(Endpoint.java:102)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:312)
        at app//org.opensearch.client.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:148)
        at app//org.opensearch.client.opensearch.OpenSearchClient.search(OpenSearchClient.java:1381)
        at app//org.opensearch.client.opensearch.integTest.AbstractSearchRequestIT.shouldReturnSearchResultsWithCompositeAgg(AbstractSearchRequestIT.java:88)
        at java.base@11.0.13/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base@11.0.13/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base@11.0.13/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base@11.0.13/java.lang.reflect.Method.invoke(Method.java:566)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)

This is against OpenSearch 2.11.0. I will continue debugging into this.

@dblock
Copy link
Member

dblock commented May 3, 2024

@VachaShah One possible explanation would be that we're not actually running this integration test in CI/CD. Will you please try to make a change that shows which/how many tests are run? I see Task :java-client:integrationTest in the logs, but it doesn't show more detail.

@VachaShah
Copy link
Collaborator Author

VachaShah commented May 3, 2024

@VachaShah One possible explanation would be that we're not actually running this integration test in CI/CD. Will you please try to make a change that shows which/how many tests are run? I see Task :java-client:integrationTest in the logs, but it doesn't show more detail.

Yeah thats what is happening I think. Running ./gradlew :java-client:integrationTest --info:

> Task :java-client:integrationTest
Caching disabled for task ':java-client:integrationTest' because:
  Build cache is disabled
Task ':java-client:integrationTest' is not up-to-date because:
  Value of input property 'systemProperties' has changed for task ':java-client:integrationTest'
Starting process 'Gradle Test Executor 1'. Working directory: /home/ubuntu/opensearch-java/java-client Command: /usr/lib/jvm/jdk-11.0.13+8/bin/java -Dhttps=true -Dorg.gradle.internal.worker.tmpdir=/home/ubuntu/opensearch-java/java-client/build/tmp/integrationTest/work -Dorg.gradle.native=false -Dpassword=admin -Dtests.awsSdk2support.domainHost -Dtests.awsSdk2support.domainRegion=us-east-1 -Dtests.awsSdk2support.serviceName=es -Dtests.security.manager=false -Duser=admin @/home/ubuntu/.gradle/.tmp/gradle-worker-classpath1009533041375880968txt -Xmx512m -Dfile.encoding=UTF-8 -Duser.country -Duser.language=en -Duser.variant -ea worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 1'

org.opensearch.client.opensearch.integTest.aws.AwsSdk2BulkRequestIT > testBulkRequest SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testAsyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testSyncAsyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testAsyncAsyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testSyncClient SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SearchIT > testDoubleWrappedException SKIPPED

org.opensearch.client.opensearch.integTest.aws.AwsSdk2SecurityIT > testUnAuthorizedException SKIPPED
Finished generating test XML results (0.011 secs) into: /home/ubuntu/opensearch-java/java-client/build/test-results/integrationTest
Generating HTML test report...
Finished generating test html results (0.016 secs) into: /home/ubuntu/opensearch-java/java-client/build/reports/tests/integrationTest

Ah I think the tests are only running from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java/org/opensearch/client and not from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java11/org/opensearch/client/opensearch/integTest

@reta
Copy link
Collaborator

reta commented May 6, 2024

Ah I think the tests are only running from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java/org/opensearch/client and not from https://github.com/opensearch-project/opensearch-java/tree/main/java-client/src/test/java11/org/opensearch/client/opensearch/integTest

Thanks @VachaShah , I used to always run ./gradlew check locally, which runs all tests, but indeed GA workflows use separate tasks, should be fixed by #968

VachaShah added 2 commits May 6, 2024 20:45
Signed-off-by: Vacha Shah <vachshah@amazon.com>
Signed-off-by: Vacha Shah <vachshah@amazon.com>
@VachaShah VachaShah changed the title [WIP] Fixing composite aggregations with correct parameters Fixing composite aggregations with correct parameters May 6, 2024
@VachaShah
Copy link
Collaborator Author

@dblock @reta This is ready for review. I added samples and guides as well.

Copy link
Member

@dblock dblock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@dblock dblock merged commit cc6c273 into opensearch-project:main May 7, 2024
55 checks passed
@dblock dblock added the backport 2.x Backport to 2.x branch label May 7, 2024
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-967-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 cc6c273ded052be68810bc2907bceabda6572eb1
# Push it to GitHub
git push --set-upstream origin backport/backport-967-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-967-to-2.x.

@dblock
Copy link
Member

dblock commented May 7, 2024

will need a manual backport pls

@VachaShah VachaShah deleted the fix-composite-aggregations branch May 7, 2024 17:04
VachaShah added a commit to VachaShah/opensearch-java that referenced this pull request May 7, 2024
…ject#967)

* Fixing composite aggregations with correct parameters

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Adding tests

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Adding guides, samples and fixing tests

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Addressing comments

Signed-off-by: Vacha Shah <vachshah@amazon.com>

---------

Signed-off-by: Vacha Shah <vachshah@amazon.com>
reta pushed a commit that referenced this pull request May 7, 2024
…974)

* Fixing composite aggregations with correct parameters (#967)

* Fixing composite aggregations with correct parameters

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Adding tests

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Adding guides, samples and fixing tests

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Addressing comments

Signed-off-by: Vacha Shah <vachshah@amazon.com>

---------

Signed-off-by: Vacha Shah <vachshah@amazon.com>

* Fix: test. (#972)

Signed-off-by: dblock <dblock@amazon.com>
Signed-off-by: Vacha Shah <vachshah@amazon.com>

---------

Signed-off-by: Vacha Shah <vachshah@amazon.com>
Signed-off-by: dblock <dblock@amazon.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] The .order() method of aggregation does not work with composite aggregations
3 participants