Upgrade Lucene to 9.8.0 #10276

msfroh · 2023-09-29T18:03:45Z

Description

9.8.0 was officially released this morning:
https://lucene.apache.org/core/9_8_0/changes/Changes.html

Related Issues

N/A

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-29T18:34:29Z

Compatibility status:

Checks if related components are compatible with change 65d73e0

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-29T18:44:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26482/
CommitID: 753d4f1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-29T19:07:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26483/
CommitID: e5b556f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-29T19:21:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: e5b556f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-29T21:41:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26490/
CommitID: e5b556f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-29T23:25:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26495/
CommitID: e5b556f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

zhichao-aws · 2023-09-30T00:00:48Z

Hi, want to have a double check. This won't be backport to 2.x before 2.11 release, right?

9.8.0 was officially released this morning: https://lucene.apache.org/core/9_8_0/changes/Changes.html Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-09-30T05:58:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26516/
CommitID: 65d73e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-30T07:19:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26521/
CommitID: 65d73e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-30T07:51:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: 65d73e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-30T14:36:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: 65d73e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-30T14:58:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: 65d73e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-30T16:45:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.CloneSnapshotIT.testCloneShallowSnapshotIndex

URL: https://build.ci.opensearch.org/job/gradle-check/26524/
CommitID: 65d73e0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-30T16:45:51Z

Codecov Report

Merging #10276 (65d73e0) into main (d656e3d) will decrease coverage by 0.08%.
Report is 4 commits behind head on main.
The diff coverage is 67.64%.

@@             Coverage Diff              @@
##               main   #10276      +/-   ##
============================================
- Coverage     71.24%   71.17%   -0.08%     
+ Complexity    58260    58233      -27     
============================================
  Files          4830     4830              
  Lines        274575   274586      +11     
  Branches      40015    40020       +5     
============================================
- Hits         195629   195436     -193     
- Misses        62570    62759     +189     
- Partials      16376    16391      +15

Files	Coverage Δ
...org/opensearch/index/search/stats/SearchStats.java	`82.02% <0.00%> (ø)`
...java/org/opensearch/index/shard/IndexingStats.java	`86.61% <0.00%> (ø)`
...rg/opensearch/repositories/s3/S3BlobContainer.java	`76.61% <76.66%> (-3.62%)`	⬇️

... and 476 files with indirect coverage changes

msfroh · 2023-09-30T16:57:35Z

This won't be backport to 2.x before 2.11 release, right?

I don't know. For now, I'm just focused on applying the change to main.

We can theoretically backport to 2.x before the start of the release window.

reta · 2023-10-01T01:47:29Z

@CEHENKLE please let us know if you are OK making 2.11 release with Apache Lucene 9.8.0, AFAIK there are no regressions we are aware of, the changes we know about were integrated (#8668, #10031)

heemin32 · 2023-10-02T17:11:03Z

+1 with backporting to 2.11.
With Lucene 9.8.0, #10246, and opensearch-project/k-NN#1182 together, a user can have parent join feature in nested query for vector search in OpenSearch 2.11.

9.8.0 was officially released this morning: https://lucene.apache.org/core/9_8_0/changes/Changes.html Signed-off-by: Michael Froh <froh@amazon.com>

dblock · 2023-10-03T22:16:55Z

It’s cutting it a little short for 2.11 but it looks like the performance improvements are compelling.

dblock · 2023-10-03T22:17:55Z

@CEHENKLE please let us know if you are OK making 2.11 release with Apache Lucene 9.8.0, AFAIK there are no regressions we are aware of, the changes we know about were integrated (#8668, #10031)

Is there a backport PR ready? What are the reasons other than “it’s a little short before a code freeze” not to take it?

reta · 2023-10-03T23:16:30Z

Is there a backport PR ready? What are the reasons other than “it’s a little short before a code freeze” not to take it?

I could make one tomorrow morning (essentially bundling 2 PRs).

msfroh · 2023-10-03T23:44:35Z

I've had some folks approach me around the office about this with similar concerns to those voiced by @model-collapse in #8668:

Before OpenSearch 2.11, the main branch of ml-commons is behind 2.x on some commits. Although this has fixed at the moment, we still suspect that some other repos may have the same kind of problems. The main branch should catch up with 2.x with the latest features & implementations before we back-port Lucene 9.8 to 2.x, otherwise, there will be endless conflicts to deal with.

We believe that all repos have done their functional tests as well as some benchmarkings to their main branch with Lucene 9.8 snapshot and so far there was no negative feedback. But as a rigorous procedure, we suggest all the repo to reproduce their tests with the formal release artifacts of Luence 9.8 (not snapshot version) so as to validate whether there's some negative impact from the Lucene upgrade to OpenSearch. The neural search plugin is absorbing the performance improvement, we also want to see others' feedback.

We'd like to see whether the change of APIs or runtime behavior will have impact to AWS hosted search service which will subscribe this upgrade in the feature. That is another topic. We shall be aware but not discuss it on github.

Based on our "release window" model where we have two weeks to produce a working distribution build from tomorrow's code freeze and the fact that we nearly missed it for 2.10, I'm starting to appreciate the risk that we could end up "just not releasing" 2.11. (Frankly, it is a time of year when people at AWS are particularly concerned about missing releases.)

macohen · 2023-10-04T15:25:37Z

Would it make sense to do a fast-follow 2.12 release with only Lucene 9.8?

9.8.0 was officially released this morning: https://lucene.apache.org/core/9_8_0/changes/Changes.html Signed-off-by: Michael Froh <froh@amazon.com>

LiuCao0614 · 2023-10-24T23:27:16Z

Hi @msfroh , do we have any plan on releasing a new version with this fix any time soon?

reta · 2023-10-25T03:42:11Z

Hi @msfroh , do we have any plan on releasing a new version with this fix any time soon?

@LiuCao0614 2.12 as per https://github.com/orgs/opensearch-project/projects/1

LiuCao0614 · 2023-10-25T17:57:20Z

Cool thanks @reta .

9.8.0 was officially released this morning: https://lucene.apache.org/core/9_8_0/changes/Changes.html Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

msfroh force-pushed the lucene_9_8_0 branch from 753d4f1 to e5b556f Compare September 29, 2023 18:06

Upgrade Lucene to 9.8.0

65d73e0

9.8.0 was officially released this morning: https://lucene.apache.org/core/9_8_0/changes/Changes.html Signed-off-by: Michael Froh <froh@amazon.com>

msfroh force-pushed the lucene_9_8_0 branch from e5b556f to 65d73e0 Compare September 30, 2023 05:26

msfroh marked this pull request as ready for review September 30, 2023 16:54

msfroh requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2 and nknize as code owners September 30, 2023 16:54

msfroh requested review from Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, VachaShah, dbwiddis, sachinpkale and sohami as code owners September 30, 2023 16:54

msfroh mentioned this pull request Sep 30, 2023

Update Apache Lucene to 9.8.0-snapshot-4373c3b #8668

Merged

6 tasks

reta approved these changes Oct 1, 2023

View reviewed changes

reta merged commit 797def6 into opensearch-project:main Oct 1, 2023
60 of 61 checks passed

msfroh deleted the lucene_9_8_0 branch October 3, 2023 00:17

reta mentioned this pull request Oct 4, 2023

[Backport] [2.x] Upgrade Lucene to 9.8.0 #10360

Merged

7 tasks

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Lucene to 9.8.0 #10276

Upgrade Lucene to 9.8.0 #10276

msfroh commented Sep 29, 2023

github-actions bot commented Sep 29, 2023 •

edited

Loading

github-actions bot commented Sep 29, 2023

github-actions bot commented Sep 29, 2023

github-actions bot commented Sep 29, 2023

github-actions bot commented Sep 29, 2023

github-actions bot commented Sep 29, 2023

zhichao-aws commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

codecov bot commented Sep 30, 2023

msfroh commented Sep 30, 2023

reta commented Oct 1, 2023

heemin32 commented Oct 2, 2023 •

edited

Loading

dblock commented Oct 3, 2023

dblock commented Oct 3, 2023

reta commented Oct 3, 2023

msfroh commented Oct 3, 2023

macohen commented Oct 4, 2023

LiuCao0614 commented Oct 24, 2023

reta commented Oct 25, 2023

LiuCao0614 commented Oct 25, 2023

Upgrade Lucene to 9.8.0 #10276

Upgrade Lucene to 9.8.0 #10276

Conversation

msfroh commented Sep 29, 2023

Description

Related Issues

Check List

github-actions bot commented Sep 29, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 29, 2023

Gradle Check (Jenkins) Run Completed with:

zhichao-aws commented Sep 30, 2023

github-actions bot commented Sep 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 30, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 30, 2023

Codecov Report

msfroh commented Sep 30, 2023

reta commented Oct 1, 2023

heemin32 commented Oct 2, 2023 • edited Loading

dblock commented Oct 3, 2023

dblock commented Oct 3, 2023

reta commented Oct 3, 2023

msfroh commented Oct 3, 2023

macohen commented Oct 4, 2023

LiuCao0614 commented Oct 24, 2023

reta commented Oct 25, 2023

LiuCao0614 commented Oct 25, 2023

github-actions bot commented Sep 29, 2023 •

edited

Loading

heemin32 commented Oct 2, 2023 •

edited

Loading