Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

Closed
wants to merge 1,482 commits into from
Closed

[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

wants to merge 1,482 commits into from

Conversation

msfroh
Copy link
Collaborator

@msfroh msfroh commented Oct 12, 2023

Description

Manual backport of #10036

There are latency-sensitive users for whom the default 20% allowable accumulation of deletes is too high, so they force merge with "only_expunge_deletes".

Unfortunately, this has ignored the max segment size, so despite the word "only" in there, they get the terrible side-effect of producing segments larger than the max segment size. These oversized segments are unlikely to participate in future merges, until they have so many deletes that they could drop back below the max segment size, likely making the whole "too many deletes" problem even worse.

This commit fixes that so that "only_expunge_deletes" ONLY EXPUNGES DELETES without creating oversized segments.

Related Issues

N/A

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

reta and others added 30 commits August 8, 2023 10:15
@reta @dependabot
Bump io.projectreactor.netty:reactor-netty-http from 1.1.8 to 1.1.9 i…
cc9d907 
…n /plugins/repository-azure (opensearch-project#9147) (opensearch-project#9169)

* Bump io.projectreactor.netty:reactor-netty-http

Bumps [io.projectreactor.netty:reactor-netty-http](https://github.com/reactor/reactor-netty) from 1.1.8 to 1.1.9.
- [Release notes](https://github.com/reactor/reactor-netty/releases)
- [Commits](reactor/reactor-netty@v1.1.8...v1.1.9)

---
updated-dependencies:
- dependency-name: io.projectreactor.netty:reactor-netty-http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit e1eade1)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
added opensearch.internal and opensearch.api to additional tags in Ja…
2c4035b 
…vadocDeclaration (Intellij Project_Default.xml) (opensearch-project#9121) (opensearch-project#9152)

(cherry picked from commit 2a5e92a)

Signed-off-by: LSchultebraucks <L.Schultebraucks@t-online.de>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@reta @dependabot
Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 in /buildSrc (o…
20be307 
…pensearch-project#9148) (opensearch-project#9174)

* Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 in /buildSrc

Bumps [org.apache.maven:maven-model](https://github.com/apache/maven) from 3.9.3 to 3.9.4.
- [Release notes](https://github.com/apache/maven/releases)
- [Commits](apache/maven@maven-3.9.3...maven-3.9.4)

---
updated-dependencies:
- dependency-name: org.apache.maven:maven-model
  dependency-type: direct:production
  update-type: version-update:semver-patch
...



* Update changelog



---------




(cherry picked from commit 75f651b)

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
@indrajohn7 @cenation07
Backport/backport 8377 to 2.x (opensearch-project#9167)
ab8e306 
* Moving get snapshot requests to listener based async calls

Signed-off-by: Indrajit Banerjee <banerind@amazon.com>

* “Empty-Commit”

Signed-off-by: Indrajit Banerjee <banerind@amazon.com>

---------

Signed-off-by: Indrajit Banerjee <banerind@amazon.com>
Co-authored-by: Indrajit Banerjee <banerind@amazon.com>
@Gaganjuneja
Adds support for tracing runnable scenarios (opensearch-project#9187)
c3a0784 
Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>
@indrajohn7
Backport/backport 8377 to 2.x (opensearch-project#9192)
545bf9d 
* Fixing SnapshotResiliencyTests.testTransportGetSnapshotsAction with single listener object

Signed-off-by: Indrajit Banerjee <banerind@amazon.com>
@Shephalimittal
[Backport 2.x] Added Integration tests support for Telemetry span Val…
560e54e 
…idations (opensearch-project#8444) (opensearch-project#9188)

Signed-off-by: Shephali Mittal <shephalm@amazon.com>
@opensearch-trigger-bot @github-actions
Mute flaky test ShardMovementStrategyTests.testClusterGreenAfterParti…
2f30604 
…alRelocationNoPreferenceShardMovementPrimaryFirstEnabled (opensearch-project#9181) (opensearch-project#9202)

(cherry picked from commit ccf0b9a)

Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@reta @dependabot
Bump io.grpc:grpc-context from 1.56.1 to 1.57.1 in /plugins/repositor…
76b0298 
…y-gcs (opensearch-project#9145) (opensearch-project#9195)

* Bump io.grpc:grpc-context in /plugins/repository-gcs

Bumps [io.grpc:grpc-context](https://github.com/grpc/grpc-java) from 1.56.1 to 1.57.1.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.56.1...v1.57.1)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-context
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 7560afb)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Mute flaky test SearchWeightedRoutingIT.testStrictWeightedRoutingWith…
81059c0 
…CustomString (opensearch-project#9175) (opensearch-project#9205)

(cherry picked from commit b2af655)

Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions @dependabot
Bump org.apache.zookeeper:zookeeper from 3.8.2 to 3.9.0 in /test/fixt…
ed3cfce 
…ures/hdfs-fixture (opensearch-project#9146) (opensearch-project#9179)

* Bump org.apache.zookeeper:zookeeper in /test/fixtures/hdfs-fixture

Bumps org.apache.zookeeper:zookeeper from 3.8.2 to 3.9.0.

---
updated-dependencies:
- dependency-name: org.apache.zookeeper:zookeeper
  dependency-type: direct:production
  update-type: version-update:semver-minor
...



* Update changelog



---------




(cherry picked from commit d4f102b)

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
@jed326
Do not evaluate shard_size and shard_min_doc_count at segment slice l…
f28c80b 
…evel (opensearch-project#9085) (opensearch-project#9211)

* Use BucketCountThresholds in InternalTerms and InternalAggregations and do not apply shard level thresholds at slice level for Concurrent Segment Search



* Addressing comments



* Re-introduce shardSize member to InternalMultiTerms and InternalMappedTerms



* Introduce LocalBucketCountThresholds for local size and min_doc_count values



---------

Signed-off-by: Jay Deng <jayd0104@gmail.com>
@nknize
[Backport 2.x] [Refactor] From XContentType.JSON to MediaTypeRegistry…
240acb8 
….JSON (opensearch-project#9156) (opensearch-project#9210)

This commit rote refactors nearly all instances and usages of the
XContentType.JSON singleton instance defined in the
:libs:opensearch-x-content library to the MediaTypeRegistry.JSON
singleton instance defined in :libs:opensearch-core. This decouples the
server and test implementations from the x-content library such that
foundation classes can be further refactored from the :server module to
the :opensearch-core library without requiring a strict dependency on
the x-content library. This keeps opensearch-core library loosely
coupled with the remaining :libs:opensearch-* libraries such that
serverless and cloud-native implementations can selective import only
the libraries needed for implementation.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
(cherry picked from commit 7278f43)

---------

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@kotwanikunal
[2.x] Added bwc version 1.3.13. (opensearch-project#9240)
e752bc3 
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
@sarthakaggarwal97
introducing codecs as their algo name (opensearch-project#9123) (open…
32e8187 
…search-project#9170)

Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
(cherry picked from commit 3713bcb)
@mch2
[Remote Store] Fix couple of Remote Store flaky test and use bulk api…
eb5d8e1 
… for ingestion (opensearch-project#9190) (opensearch-project#9251)
@reta @dependabot
Bump com.azure:azure-storage-blob from 12.22.3 to 12.23.0 in /plugins…
c568386 
…/repository-azure (opensearch-project#9231) (opensearch-project#9260)

* Bump com.azure:azure-storage-blob in /plugins/repository-azure

Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0)

---
updated-dependencies:
- dependency-name: com.azure:azure-storage-blob
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 1162865)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@kotwanikunal @dependabot @Xtansia
[Backport 2.x] Backports missing dependency upgrades (opensearch-proj…
bc69838 
…ect#9271)

* Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (opensearch-project#9227)

* Bump com.diffplug.spotless from 6.19.0 to 6.20.0

Bumps com.diffplug.spotless from 6.19.0 to 6.20.0.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 48ec206)

* Force upgrade the vulnerable dependencies of hadoop-minicluster (opensearch-project#9252)

* Force upgrade the vulnerable dependencies of hadoop-minicluster

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

* Add changelog entry

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

---------

Signed-off-by: Thomas Farr <tsfarr@amazon.com>
(cherry picked from commit a1fc31c)

* Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 in /plugins/repository-gcs (opensearch-project#9230)

* Bump com.google.code.gson:gson in /plugins/repository-gcs

Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.9.0 to 2.10.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md)
- [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.10.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updating SHAs

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 5c283e0)

* Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (opensearch-project#9228)

* Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0

Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 1.2.0 to 1.8.0.
- [Release notes](https://github.com/lycheeverse/lychee-action/releases)
- [Commits](lycheeverse/lychee-action@v1.2.0...v1.8.0)

---
updated-dependencies:
- dependency-name: lycheeverse/lychee-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 4b21849)

---------

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Thomas Farr <tsfarr@amazon.com>
@mch2
[Backport 2.x] [Segment Replication] Refactor file cleanup logic and …
c8471a0 
…fix PIT/Scroll with remote store. (opensearch-project#9272)

* [Segment Replication] Refactor file cleanup logic and fix PIT/Scroll with remote store. (opensearch-project#9111)

* Remove divergent commit logic with segment replication.

This change removes divergent commit paths for segrep node-node and remote store.
All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file.
This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before
an engine is opened.
This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add more NRTReplicationEngineTests.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add more shard level tests.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add test ensuring commits are cleaned up on replicas.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Self review.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Use refresh level sync before recovery

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
(cherry picked from commit c301544)

* Fix test SegmentReplicationIndexShardTests.testPrimaryRestart.

This test is specific to remote store and should not be run for node-node replication.

Signed-off-by: Marc Handalian <handalm@amazon.com>
(cherry picked from commit a33f67e)
Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
@opensearch-trigger-bot @github-actions
Bump snakeyaml from 2.0 to 2.1 (opensearch-project#9269) (opensearch-…
3de6cc6 
…project#9274)

(cherry picked from commit 3ddff4c)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Fix flaky test SegmentReplicationIT.testDeleteOperations. (opensearch…
df6245f 
…-project#9244) (opensearch-project#9273)

(cherry picked from commit ab08d73)

Signed-off-by: Marc Handalian <handalm@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@dreamer-89
[Segment Replication] Unmute testIndexingWithSegRep rolling upgrade t…
6171cfc 
…est (opensearch-project#9079) (opensearch-project#9278)

* Unmute testIndexingWithSegRep rolling upgrade test



* Assert on row count before processing the result



* Filter rows with 0 doc count



* Retry assertHitCount



* Handle exception



* Add comment



---------

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@opensearch-trigger-bot @github-actions
docCountError must be 0 for shard level reduce (opensearch-project#9246…
2ee5c38 
…) (opensearch-project#9282)

(cherry picked from commit db768d1)

Signed-off-by: Jay Deng <jayd0104@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@linuxpi
[Backport 2.x][Remote Store] Introduce RemoteStoreRestoreService to h…
d30199f 
…andle restore … (opensearch-project#9261)

---------

Signed-off-by: bansvaru <bansvaru@amazon.com>
@ashking94
Fix local segments stats update in RemoteStoreRefreshListener (opense…
77df735 
…arch-project#8758) (opensearch-project#9268)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
@opensearch-trigger-bot @github-actions
Make SearchTemplateRequest implement IndicesRequest.Replaceable (open…
156118a 
…search-project#9122) (opensearch-project#9275)

* Make SearchTemplateRequest implement IndicesRequest.Replaceable



* Add to CHANGELOG



---------


(cherry picked from commit c73f727)

Signed-off-by: Craig Perkins <cwperx@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@Gaganjuneja
Adds attributes to startSpan (opensearch-project#9199) (opensearch-pr…
a3cfccc 
…oject#9292)

* Adds attributes to startSpan



* Update Changelog



* Adds attributes to startSpan



* Refactor code



* Add java doc



* Refactor code



* Refactor code



* Refactor code



* Removes dependency



---------

Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>
Co-authored-by: Gagan Juneja <gjjuneja@amazon.com>
@opensearch-trigger-bot @github-actions
[Remote Store] Remove stale code for remote restore from Restore Serv… (
ef66423 
opensearch-project#9293) (opensearch-project#9295)

---------


(cherry picked from commit 5fdebee)

Signed-off-by: bansvaru <bansvaru@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@reta
[Backport] [2.x] [BWC and API enforcement] Define the initial set of …
55ab42d 
…annotations, their meaning and relations between them (opensearch-project#9223) (opensearch-project#9296)

* [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (opensearch-project#9223)

* [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Signed-off-by: Peter Nied <petern@amazon.com>
Co-authored-by: Peter Nied <petern@amazon.com>
(cherry picked from commit 0b2c07a)

* [BWC and API enforcement] Extend the element types the initial set of annotations could be applied to

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

---------

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta @owaiskazi19
The spotless should take care of unified imports ordering (libs/ subp…
4110ab0 
…rojects) (opensearch-project#9309) (opensearch-project#9319)

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
(cherry picked from commit 301c766)

Co-authored-by: Owais Kazi <owaiskazi19@gmail.com>
dblock and others added 27 commits October 3, 2023 10:41
@dblock
Fix: register mulitple extensions. (opensearch-project#10256) (opense…
5f5f2f8 
…arch-project#10315)

* Fix: register mulitple extensions.



* Updated CHANGELOG.



* Added tests.



---------


(cherry picked from commit 1d66af3)

Signed-off-by: dblock <dblock@amazon.com>
@RS146BIJAY
Add unreferenced file cleanup count to merge stats (opensearch-projec…
f366dc3 
…t#10204) (opensearch-project#10333)

Signed-off-by: Rishav Sagar <rissag@amazon.com>
Co-authored-by: Rishav Sagar <rissag@amazon.com>
@opensearch-trigger-bot @github-actions
Replace deprecated command with environment file (opensearch-project#…
2f430ed 
…10289) (opensearch-project#10332)

(cherry picked from commit 2098975)

Signed-off-by: Jongwoo Han <jongwooo.han@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@reta
Bump netty from 4.1.97.Final to 4.1.99.Final (opensearch-project#10306)…
537e4fc 
… (opensearch-project#10319)

(cherry picked from commit beb25b1)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@reta @dependabot
Bump peter-evans/create-pull-request from 3 to 5 (opensearch-project#…
4bf4857 
…10301) (opensearch-project#10337)

* Bump peter-evans/create-pull-request from 3 to 5

Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3 to 5.
- [Release notes](https://github.com/peter-evans/create-pull-request/releases)
- [Commits](peter-evans/create-pull-request@v3...v5)

---
updated-dependencies:
- dependency-name: peter-evans/create-pull-request
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit fe80682)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Add TODO for updating public documentation (opensearch-project#10242) (…
3c1e5fb 
…opensearch-project#10335)

(cherry picked from commit e7909fc)

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Refactor async blob read to avoid blocking calls, support non multipa…
a0f9736 
…rt calls (opensearch-project#10192) (opensearch-project#10278)

(cherry picked from commit 61495bf)

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@rishabhmaurya
Use of LogByteSizeMergePolicy for data stream use cases (opensearch-p…
81cd1f2 
…roject#9992) (opensearch-project#10312)

* Configurable merge policy for index

* additional setting to configure merge policy for timestamp based index
* introduction of logbytesize merge policy as an option

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* remove the trace log not required anymore

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* Refactor the merge policy extraction logic

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* Rename constant DEFAULT to DEFAULT_POLICY

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* Simplify merge policy extraction and selection logic

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* missing javadoc error

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* Renaming log byte size policy setting with mb

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* Move validation exception to enum from setting defn

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

* rename time_index to time_series_index

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>

---------

Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com>
(cherry picked from commit fa66beb)
@opensearch-trigger-bot @github-actions
Disable concurrent search when terminate_after is used (opensearch-pr…
d48aa44 
…oject#10200) (opensearch-project#10329)

(cherry picked from commit 3a790c1)

Signed-off-by: Jay Deng <jayd0104@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions @dblock
Simplify initialization of Settings (opensearch-project#10194) (opens…
6170dd9 
…earch-project#10317)

There was a weird circular class-loading dependency between Settings and
Settings.Builder that could result in ClassLoader deadlock if multiple
threads trigger class-loading of Settings at the same time.


(cherry picked from commit d3bf230)

Signed-off-by: Michael Froh <froh@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
@opensearch-trigger-bot @github-actions
Fix the HttpChannel trace termination in case of exceptions (opensear…
912b60a 
…ch-project#10325) (opensearch-project#10344)

(cherry picked from commit bddf0d3)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@kotwanikunal
Add support for encrypted async blob read (opensearch-project#10131) (o…
689adc1 
…pensearch-project#10346)

* Add support for encrypted async blob read

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Add async blob read support for encrypted containers

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

---------

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
(cherry picked from commit c4c4ad8)
@opensearch-trigger-bot @github-actions
Add node id to segment and translog metadata (opensearch-project#10229)…
9c678ab 
… (opensearch-project#10353)

---------


(cherry picked from commit 7159e2e)

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Enable remote segment upload backpressure by default (opensearch-proj…
beec6b7 
…ect#10356) (opensearch-project#10366)

(cherry picked from commit e5024a8)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Fix shard failure due to translog tragic close on upload failure (ope…
a0dc2b9 
…nsearch-project#10363) (opensearch-project#10368)

(cherry picked from commit a0cb344)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@reta @dependabot
Bump org.xerial.snappy:snappy-java from 1.1.10.4 to 1.1.10.5 in /test…
407197d 
…/fixtures/hdfs-fixture (opensearch-project#10299) (opensearch-project#10369)

* Bump org.xerial.snappy:snappy-java in /test/fixtures/hdfs-fixture

Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.4 to 1.1.10.5.
- [Release notes](https://github.com/xerial/snappy-java/releases)
- [Commits](xerial/snappy-java@v1.1.10.4...v1.1.10.5)

---
updated-dependencies:
- dependency-name: org.xerial.snappy:snappy-java
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update changelog

Signed-off-by: dependabot[bot] <support@github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 3ed7cc4)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@psychbot
[Backport 2.x][Remote Store] Add support to restrict creation & delet…
38ca3e1 
…ion if system repository and mutation of immutable settings of system repository (opensearch-project#9839) (opensearch-project#10364)

---------


(cherry picked from commit 699d235)

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
@opensearch-trigger-bot @github-actions
Retry on failure to acquire lock on remote metadata (opensearch-proje…
07ece0c 
…ct#10341) (opensearch-project#10361)

(cherry picked from commit 677ff75)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@andrross
Refactor multipart download to a more async model (opensearch-project…
fc7dc20 
…#10349) (opensearch-project#10373)

* Refactor read context streams to async streams

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

* Refactor multipart download to a more async model

The previous approach of kicking off the stream requests for all parts
of a file did not work well for very large files. For example, a 20GiB
file uploaded in 16MiB parts will consist of 1200+ parts. When we
attempted to initiate streaming for all parts concurrently, some parts
would hit a client timeout after 2 minutes without being able to get a
connection due to the other parts not having been completed in that time
frame. This refactoring adds yet another layer of indirection in order
to allow the code that is actually writing the destination file to
control the rate at which streams are started. This should allow for
downloading files consisting of arbitrarily many parts at any connection
speed.

This commit also wires in the download rate limiter so that the
`indices.recovery.max_bytes_per_sec` is properly honored.

Signed-off-by: Andrew Ross <andrross@amazon.com>

---------

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
Signed-off-by: Andrew Ross <andrross@amazon.com>
Co-authored-by: Kunal Kotwani <kkotwani@amazon.com>
(cherry picked from commit 28f185b)
@andrross @mch2
Update multipart download path to first write to temp files (opensear…
78c55b7 
…ch-project#10347) (opensearch-project#10389)

* Update multipart download path to write to temp files.

This change updates ReadContextListener to first write parts to a temp location
until all parts have been received.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Suppress forbidden IOUtils.fsync

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove unnecessary logging format

Signed-off-by: Marc Handalian <handalm@amazon.com>

* sync directory after file rename

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Remove flaky threadpool terminate test

Signed-off-by: Marc Handalian <handalm@amazon.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
(cherry picked from commit ca0dae6)

Co-authored-by: Marc Handalian <handalm@amazon.com>
@Gaganjuneja
[Backport 2.x] - Add Telemetry metrics framework (opensearch-project#…
92a261b 
…10393)

Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>
@gbbafna
[Backport 2.x] [Remote Store] Add support to reload repository metada…
217ec16 
…ta inplace (opensearch-project#9569) (opensearch-project#10390)

* [Remote Store] Add support to reload repository metadata inplace (opensearch-project#9569)

---------

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

* Empty commit to retrigger build

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

* Empty commit to retrigger build

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

---------

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>
Signed-off-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com>
Co-authored-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
[Tracing Instrumentation] Add instrumentation in InboundHandler (open…
890663f 
…search-project#10143) (opensearch-project#10381)

* Add instrumentation in InboundHandler



* Add CHANGELOG



* Address review comment



* Address review comment



* Address review comment



* Empty-Commit



* Address review comment



* Address review comment



* Address review comment



---------




(cherry picked from commit aac0f09)

Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>
Signed-off-by: Gagan Juneja <gagandeepjuneja@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gagan Juneja <gjjuneja@amazon.com>
@opensearch-trigger-bot @github-actions
fix stale remote cluster uuid state not purged from remote (opensearc…
16d12e1 
…h-project#10016) (opensearch-project#10403)

---------
(cherry picked from commit 1eec0f1)

Signed-off-by: bansvaru <bansvaru@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot @github-actions
Adding cancellation start time in client side TaskInfo (opensearch-pr…
1926449 
…oject#10326) (opensearch-project#10384)

(cherry picked from commit 71b6948)

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@msfroh
[Backport 2.x] Bump org.apache.commons:commons-compress from 1.23.0 t…
fea095d 
…o 1.24.0 in /buildSrc (opensearch-project#10386)

Manual backport of
opensearch-project#10205

Signed-off-by: Michael Froh <froh@amazon.com>
@msfroh
Honor max segment size during only_expunge_deletes (opensearch-projec…
abb99b5 
…t#10036)

(cherry picked from commit 1dde018)
Signed-off-by: Michael Froh <froh@amazon.com>
@msfroh msfroh closed this Oct 12, 2023
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Compatibility status:

Checks if related components are compatible with change abb99b5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz will be requested when the pull request is marked ready for review anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross will be requested when the pull request is marked ready for review andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 will be requested when the pull request is marked ready for review ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar will be requested when the pull request is marked ready for review Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE will be requested when the pull request is marked ready for review CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock will be requested when the pull request is marked ready for review dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis will be requested when the pull request is marked ready for review dbwiddis is a code owner

@gaobinlong gaobinlong Awaiting requested review from gaobinlong gaobinlong will be requested when the pull request is marked ready for review gaobinlong is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna will be requested when the pull request is marked ready for review gbbafna is a code owner

@jed326 jed326 Awaiting requested review from jed326 jed326 will be requested when the pull request is marked ready for review jed326 is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal will be requested when the pull request is marked ready for review kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 will be requested when the pull request is marked ready for review mch2 is a code owner

@nknize nknize Awaiting requested review from nknize nknize will be requested when the pull request is marked ready for review nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 will be requested when the pull request is marked ready for review owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied peternied will be requested when the pull request is marked ready for review peternied is a code owner

@reta reta Awaiting requested review from reta reta will be requested when the pull request is marked ready for review reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 will be requested when the pull request is marked ready for review Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale will be requested when the pull request is marked ready for review sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli will be requested when the pull request is marked ready for review saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja will be requested when the pull request is marked ready for review shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami will be requested when the pull request is marked ready for review sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah will be requested when the pull request is marked ready for review VachaShah is a code owner

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk will be requested when the pull request is marked ready for review jainankitk is a code owner

@linuxpi linuxpi Awaiting requested review from linuxpi linuxpi will be requested when the pull request is marked ready for review linuxpi is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.