Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make number of segment metadata files in remote segment store configurable #11329

Merged
merged 7 commits into from
Nov 27, 2023
Merged

Make number of segment metadata files in remote segment store configurable #11329

merged 7 commits into from
Nov 27, 2023

Conversation

sachinpkale
Copy link
Member

@sachinpkale sachinpkale commented Nov 24, 2023 •
edited
Loading

Description

Current Behavior

  • Remote segment store uploads a metadata file with each successful upload of segments post refresh.
  • To read any segment data from remote segment store, we need to read metadata file first.
  • All the flows that involves copying segments between 2 nodes, like replication, peer recovery are integrated with remote store.
  • Following are the steps to cleanup stale segments (that are not live on local) from remote segment store:
    • On flush, we fetch all but latest 10 metadata files, let's call them stale metadata files.
    • Create a list of segments that are referenced by the stale metadata files and not currently live, let's call them stale segment files.
    • Delete stale segment files followed by deletion of stale metadata files.

Issue

  • Replication and peer recovery flows are async. When replica (or new primary in case of primary relocation) starts downloading data from remote segment store, it reads the latest metadata file and starts downloading segment files referred by the metadata file.
  • To avoid race condition where slow replica is taking a lot of time to download and the segment files became stale at primary (due to segment merges) and deleted from remote segment store, we kept a hard-coded value of 10 to keep last 10 metadata files.
  • If the amount of data to be downloaded is high and if the flush (and merge) happens at primary during that time, it is possible that replica will not find a segment file in the remote store and recovery fails with following exception.
IndexShardRecoveryException[Exception while copying segment files from remote segment store]; 
nested: IOException[Error occurred when downloading segments from remote store]; 
nested: NoSuchFileException[Blob object [<Segment Path>/_6yt.cfs__KegSxIsBOrEU3DYZHraf] not found: 
The specified key does not exist]

Solution

  • In this PR, we are mitigating the above mentioned issue by making last N metadata files to keep configurable.
  • This is not a long term solution to the problem and we need to come up with a lock based solution (used by snapshot interop) but until then this change provides a way to avoid NoSuchFileException.
  • We also made a change to peer recovery flow where, if downloading from remote segment store fails, recovery flow will continue with node-node segment copy. This is something that exists even today with remote store integration of peer recovery flow.

Related Issues

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Make number of segment metadata files to keep configurable
d5ac8ac 
Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 24, 2023 •
edited
Loading

Compatibility status:

Checks if related components are compatible with change 1a6cfad

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for d5ac8ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ashking94
ashking94 reviewed Nov 25, 2023
View reviewed changes
Copy link
Member

@ashking94 ashking94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but interested in plan for this setting as it will be user facing. I can review once UTs and ITs are added.

Sachin Kale added 2 commits November 25, 2023 18:37
Add UTs and ITs, fix existing UTs
fb10d90 
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Add changlog entry
71e133c 
Signed-off-by: Sachin Kale <kalsac@amazon.com>
@sachinpkale sachinpkale marked this pull request as ready for review November 25, 2023 13:28
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 71e133c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Address PR comments
cb81e86 
Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for cb81e86: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Fix compile errors in test
98884c1 
Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 98884c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Fix failing test
250fdcc 
Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions GitHub Actions
Copy link
Contributor

❕ Gradle check result for 250fdcc: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@codecov Codecov
Copy link

codecov bot commented Nov 26, 2023 •
edited
Loading

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (5bb6cae) 71.21% compared to head (250fdcc) 71.29%.
Report is 3 commits behind head on main.

❗ Current head 250fdcc differs from pull request most recent head 1a6cfad. Consider uploading reports for the commit 1a6cfad to get more accurate results

Files Patch % Lines
...ch/indices/recovery/PeerRecoveryTargetService.java 0.00% 4 Missing ⚠️
...earch/index/store/RemoteSegmentStoreDirectory.java 0.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #11329      +/-   ##
============================================
+ Coverage     71.21%   71.29%   +0.08%     
- Complexity    58926    58984      +58     
============================================
  Files          4890     4890              
  Lines        277434   277451      +17     
  Branches      40313    40313              
============================================
+ Hits         197567   197812     +245     
+ Misses        63464    63237     -227     
+ Partials      16403    16402       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Remove max limit from cluster.remote_store.index.segment_metadata.ret…
1a6cfad 
…ention.max_count setting

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@github-actions GitHub Actions
Copy link
Contributor

❕ Gradle check result for 1a6cfad: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@gbbafna gbbafna merged commit ec5a0f9 into opensearch-project:main Nov 27, 2023
33 of 36 checks passed
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Nov 27, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11329-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ec5a0f99f89b82bbbd14cc0ae06ea8ea6ba08667
# Push it to GitHub
git push --set-upstream origin backport/backport-11329-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11329-to-2.x.

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request Nov 28, 2023
@sachinpkale
Make number of segment metadata files in remote segment store configu…
5e3dd85 
…rable (opensearch-project#11329)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@sachinpkale sachinpkale mentioned this pull request Nov 28, 2023
Merged
gbbafna pushed a commit that referenced this pull request Nov 28, 2023
@sachinpkale
Make number of segment metadata files in remote segment store configu…
cf7ba36 
…rable (#11329) (#11357)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
fahadshamiinsta pushed a commit to fahadshamiinsta/OpenSearch270 that referenced this pull request Dec 4, 2023
@sachinpkale @fahadshamiinsta
Make number of segment metadata files in remote segment store configu…
628b29a 
…rable (opensearch-project#11329)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request Dec 11, 2023
@sachinpkale @deshsidd
Make number of segment metadata files in remote segment store configu…
2009e07 
…rable (opensearch-project#11329)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@BrewTestBot BrewTestBot mentioned this pull request Feb 21, 2024
Merged
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024
@sachinpkale @rayshrey
Make number of segment metadata files in remote segment store configu…
b973096 
…rable (opensearch-project#11329)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
@sachinpkale @shiv0408
Make number of segment metadata files in remote segment store configu…
cb17870 
…rable (opensearch-project#11329)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashking94 ashking94 ashking94 left review comments

@gbbafna gbbafna gbbafna approved these changes

@Bukhtawar Bukhtawar Bukhtawar approved these changes

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@peternied peternied Awaiting requested review from peternied

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@setiah setiah Awaiting requested review from setiah

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

Assignees
No one assigned
Labels
backport 2.x Backport to 2.x branch backport-failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sachinpkale @ashking94 @Bukhtawar @gbbafna