Bump to Lucene99 #11421

mch2 · 2023-11-30T22:55:04Z

Description

Bump Lucene to Version 9.9.1

Related Issues

Resolves #11414

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-11-30T23:00:53Z

Compatibility status:

Checks if related components are compatible with change 9024bdb

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/k-nn.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

github-actions · 2023-11-30T23:03:20Z

❌ Gradle check result for 26f9fc6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/apache/lucene/search/grouping/CollapseTopFieldDocs.java

msfroh

@gashutos -- I suspect you know more about the pruning logic than any of us. Can you confirm that we're doing this properly?

Thanks!

...c/main/java/org/opensearch/index/fielddata/fieldcomparator/DoubleValuesComparatorSource.java

server/src/main/java/org/opensearch/index/fielddata/IndexFieldData.java

...r/src/main/java/org/opensearch/search/aggregations/bucket/composite/CompositeAggregator.java

gashutos · 2023-12-01T06:06:21Z

@gashutos -- I suspect you know more about the pruning logic than any of us. Can you confirm that we're doing this properly?

Thanks!

@msfroh @mch2 Changes look correct to me. We need to address filterPruning helper in comparatorSource as @msfroh suggested.
Very surprising that small change in Lucene is introducing API breaks (compile time). Could have been better

github-actions · 2023-12-04T20:56:57Z

❌ Gradle check result for 98b3098: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mch2 · 2023-12-04T21:39:30Z

@gashutos @msfroh Thanks for the review here, latest revision handles the comparatorSource changes .

Still working through some unit failures, theres a break in some concurrent search related tests that mocked a now final IndexSearcher#getSlices.

github-actions · 2023-12-04T21:50:21Z

❌ Gradle check result for 6a97ffa: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

gashutos

Looks good !

github-actions · 2023-12-06T00:48:05Z

❌ Gradle check result for 1c3d186: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-06T01:45:14Z

❌ Gradle check result for 63b75f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-06T20:16:23Z

❌ Gradle check result for 286514d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Updates to support interface change to FieldComparatorSource to use Pruning enum instead of enableSkipping boolean. Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

…TestCase method of the same name Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

server/src/main/java/org/opensearch/index/fielddata/IndexFieldData.java

reta · 2024-01-03T18:31:36Z

Thanks a lot for doing that @mch2 !

reta · 2024-01-03T20:54:57Z

@msfroh could you please look at this change as well? (Lucene 9.9.1 update) thank you

...rc/main/java/org/opensearch/plugin/correlation/core/index/codec/CorrelationCodecVersion.java

...r/src/main/java/org/opensearch/search/aggregations/bucket/BestBucketsDeferringCollector.java

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

github-actions · 2024-01-09T22:45:03Z

❌ Gradle check result for 0185041: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Marc Handalian <handalm@amazon.com>

github-actions · 2024-01-09T23:34:23Z

❌ Gradle check result for 0185041:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-09T23:34:28Z

❌ Gradle check result for 9024bdb:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

mch2 · 2024-01-09T23:56:18Z

check seems to be having issues - latest run finished with Finished: UNSTABLE but did not report as such. re-firing

github-actions · 2024-01-10T00:43:43Z

✅ Gradle check result for 9024bdb: SUCCESS

opensearch-trigger-bot · 2024-01-10T02:19:46Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11421-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4a42150ba0cbf10022797c038f35faaea2f548f6
# Push it to GitHub
git push --set-upstream origin backport/backport-11421-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11421-to-2.x.

Upgraded Lucene dependency to 9.9.1. --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Signed-off-by: Marc Handalian <handalm@amazon.com> (cherry picked from commit 4a42150) Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

* Bump to Lucene99 (#11421) Upgraded Lucene dependency to 9.9.1. --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Signed-off-by: Marc Handalian <handalm@amazon.com> (cherry picked from commit 4a42150) Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * update deprecated CodecService constructor only in 2.x Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Remove unused emptyReader method in Lucene.java This method has a conflict inside leafReader's getMetaData as the constructor to LeafMetaData has changed. However this method is never invoked so removing it. Signed-off-by: Marc Handalian <marc.handalian@gmail.com> * Fix Version for 2.12.0 to correctly point to Lucene 9.9.1 Signed-off-by: Marc Handalian <marc.handalian@gmail.com> --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Signed-off-by: Marc Handalian <handalm@amazon.com>

Upgraded Lucene dependency to 9.9.1. --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Signed-off-by: Marc Handalian <handalm@amazon.com>

Upgraded Lucene dependency to 9.9.1. --------- Signed-off-by: Marc Handalian <marc.handalian@gmail.com> Signed-off-by: Marc Handalian <handalm@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added dependencies Pull requests that update a dependency file enhancement Enhancement or improvement to existing feature or request v2.13.0 Issues and PRs related to version 2.13.0 v3.0.0 Issues and PRs related to version 3.0.0 labels Nov 30, 2023

andrross reviewed Nov 30, 2023

View reviewed changes

server/src/main/java/org/apache/lucene/search/grouping/CollapseTopFieldDocs.java Outdated Show resolved Hide resolved

msfroh reviewed Dec 1, 2023

View reviewed changes

...c/main/java/org/opensearch/index/fielddata/fieldcomparator/DoubleValuesComparatorSource.java Outdated Show resolved Hide resolved

gashutos reviewed Dec 1, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/fielddata/IndexFieldData.java Outdated Show resolved Hide resolved

gashutos reviewed Dec 1, 2023

View reviewed changes

...r/src/main/java/org/opensearch/search/aggregations/bucket/composite/CompositeAggregator.java Outdated Show resolved Hide resolved

gashutos approved these changes Dec 5, 2023

View reviewed changes

mch2 added 12 commits December 6, 2023 15:01

Bump version to 9.9.0-snapshot

35f7b80

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Support change apache/lucene#12405

08a9ae7

Updates to support interface change to FieldComparatorSource to use Pruning enum instead of enableSkipping boolean. Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Codec bump

455f999

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Update to honor enableSkipping in XFieldComparatorSource

0e355fc

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Use released 9.9 version

13f5b3b

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

spotless

4cc1ce6

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Fix javadoc in SortedWiderNumericSortField

ed62ad1

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Fix tests mocking now final IndexSearcher#getSlices method

771f40c

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Bump CorrelationCodec to V_9_9_0

5a773ce

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

update javadoc linking to old codec

b2546b2

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

update SHAs

1e04b1b

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

rename shutdownExecutorService in tests that now conflict with Lucene…

7794507

…TestCase method of the same name Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

reta reviewed Jan 3, 2024

View reviewed changes

server/src/main/java/org/opensearch/index/fielddata/IndexFieldData.java Show resolved Hide resolved

reta approved these changes Jan 3, 2024

View reviewed changes

reta mentioned this pull request Jan 4, 2024

[BUG] Flaky org.opensearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT test #10154

Closed

msfroh reviewed Jan 5, 2024

View reviewed changes

...rc/main/java/org/opensearch/plugin/correlation/core/index/codec/CorrelationCodecVersion.java Show resolved Hide resolved

msfroh reviewed Jan 5, 2024

View reviewed changes

...r/src/main/java/org/opensearch/search/aggregations/bucket/BestBucketsDeferringCollector.java Show resolved Hide resolved

mch2 added 2 commits January 9, 2024 13:49

rename correlation950 package to correlation990

0688dfa

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

add comment to BestBucketsDeferringCollector

0185041

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

Merge branch 'main' into Lucene99

9024bdb

Signed-off-by: Marc Handalian <handalm@amazon.com>

msfroh approved these changes Jan 10, 2024

View reviewed changes

msfroh merged commit 4a42150 into opensearch-project:main Jan 10, 2024
28 checks passed

msfroh added the backport 2.x Backport to 2.x branch label Jan 10, 2024

opensearch-trigger-bot bot added the backport-failed label Jan 10, 2024

msfroh mentioned this pull request Jan 10, 2024

[Proposal] Move the OpenSearch 2.12.0 release to Feb 20 2024 opensearch-project/opensearch-build#4290

Closed

reta added v2.12.0 Issues and PRs related to version 2.12.0 and removed v2.13.0 Issues and PRs related to version 2.13.0 labels Jan 10, 2024

mch2 mentioned this pull request Jan 10, 2024

[Backport 2.x] Bump to Lucene99 #11832

Merged

8 tasks

mch2 deleted the Lucene99 branch January 10, 2024 06:25

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump to Lucene99 #11421

Bump to Lucene99 #11421

mch2 commented Nov 30, 2023 •

edited

Loading

github-actions bot commented Nov 30, 2023 •

edited

Loading

github-actions bot commented Nov 30, 2023

msfroh left a comment

gashutos commented Dec 1, 2023

github-actions bot commented Dec 4, 2023

mch2 commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

gashutos left a comment

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

reta commented Jan 3, 2024

reta commented Jan 3, 2024

github-actions bot commented Jan 9, 2024

github-actions bot commented Jan 9, 2024

github-actions bot commented Jan 9, 2024

mch2 commented Jan 9, 2024

github-actions bot commented Jan 10, 2024

opensearch-trigger-bot bot commented Jan 10, 2024

Bump to Lucene99 #11421

Bump to Lucene99 #11421

Conversation

mch2 commented Nov 30, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 30, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Nov 30, 2023

msfroh left a comment

Choose a reason for hiding this comment

gashutos commented Dec 1, 2023

github-actions bot commented Dec 4, 2023

mch2 commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

gashutos left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

github-actions bot commented Dec 6, 2023

reta commented Jan 3, 2024

reta commented Jan 3, 2024

github-actions bot commented Jan 9, 2024

github-actions bot commented Jan 9, 2024

github-actions bot commented Jan 9, 2024

mch2 commented Jan 9, 2024

github-actions bot commented Jan 10, 2024

opensearch-trigger-bot bot commented Jan 10, 2024

mch2 commented Nov 30, 2023 •

edited

Loading

github-actions bot commented Nov 30, 2023 •

edited

Loading