Cleanup file cache on deleting index/shard directory #11443

bugmakerrrrrr · 2023-12-04T12:53:00Z

Description

Currently, we rely on the index event listener to manage file cache cleanup. This approach is effective when we simply delete the index. However, in certain situations, such as shard relocation, the shard data may be deleted asynchronously, causing the preset index event listener to not be triggered (as noted in the comments for org.opensearch.indices.cluster.IndicesClusterStateService#removeIndices/removeShards). To address this issue, I suggest that we implement file cache cleanup (including cache entry eviction and file deletion) during the deletion of the index/shard directory in NodeEnvironment.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-12-04T13:00:18Z

❌ Gradle check result for 50c3b60: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2023-12-04T13:15:17Z

Compatibility status:

Checks if related components are compatible with change e1e1472

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git]

bugmakerrrrrr · 2023-12-11T16:03:01Z

@andrross @adnapibar Would you mind taking a look?

opensearch-trigger-bot · 2024-01-11T15:20:54Z

This PR is stalled because it has been open for 30 days with no activity.

bugmakerrrrrr · 2024-01-11T15:41:50Z

@andrross Sorry to bother you again. I believe this is a bug that need to fix, can you have a look

andrross · 2024-01-11T18:24:26Z

@bugmakerrrrrr Apologies for the delay, I'll look into this today, thanks!

server/src/main/java/org/opensearch/env/NodeEnvironment.java

test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java

Accidentally clicked approve...

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr · 2024-02-02T17:12:58Z

@andrross I add a new listener IndexStoreListener to NodeEnvironment, can you take a look?

andrross · 2024-02-02T17:30:50Z

@bugmakerrrrrr I'll take a look today, thanks!

github-actions · 2024-02-02T17:53:49Z

❕ Gradle check result for 36a6bb1: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-02-02T17:55:24Z

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (57cc0dd) 71.34% compared to head (e1e1472) 71.41%.
Report is 3 commits behind head on main.

Files	Patch %	Lines
...index/store/remote/filecache/FileCacheCleaner.java	60.86%	5 Missing and 4 partials ⚠️
.../main/java/org/opensearch/env/NodeEnvironment.java	57.89%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11443      +/-   ##
============================================
+ Coverage     71.34%   71.41%   +0.07%     
- Complexity    59486    59511      +25     
============================================
  Files          4927     4927              
  Lines        279663   279682      +19     
  Branches      40663    40666       +3     
============================================
+ Hits         199513   199745     +232     
+ Misses        63533    63335     -198     
+ Partials      16617    16602      -15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-02-02T17:58:52Z

❕ Gradle check result for e1e1472: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

* cleanup file cache on deleting index/shard directory Signed-off-by: panguixin <panguixin@bytedance.com> * add index store listener Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> (cherry picked from commit c564ee3) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* cleanup file cache on deleting index/shard directory * add index store listener --------- (cherry picked from commit c564ee3) Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ect#11443) * cleanup file cache on deleting index/shard directory Signed-off-by: panguixin <panguixin@bytedance.com> * add index store listener Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com>

…ect#11443) * cleanup file cache on deleting index/shard directory Signed-off-by: panguixin <panguixin@bytedance.com> * add index store listener Signed-off-by: panguixin <panguixin@bytedance.com> --------- Signed-off-by: panguixin <panguixin@bytedance.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jan 11, 2024

andrross previously approved these changes Jan 11, 2024

View reviewed changes

server/src/main/java/org/opensearch/env/NodeEnvironment.java Outdated Show resolved Hide resolved

test/framework/src/main/java/org/opensearch/test/InternalTestCluster.java Outdated Show resolved Hide resolved

andrross added backport 2.x Backport to 2.x branch bug Something isn't working Search:Remote Search and removed stalled Issues that have stalled labels Jan 11, 2024

cleanup file cache on deleting index/shard directory

bb58e2b

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr force-pushed the clean_file_cache branch from 50c3b60 to 36a6bb1 Compare February 2, 2024 17:01

add index store listener

e1e1472

Signed-off-by: panguixin <panguixin@bytedance.com>

bugmakerrrrrr force-pushed the clean_file_cache branch from 36a6bb1 to e1e1472 Compare February 2, 2024 17:09

andrross added the skip-changelog label Feb 3, 2024

andrross approved these changes Feb 3, 2024

View reviewed changes

andrross merged commit c564ee3 into opensearch-project:main Feb 3, 2024
32 of 34 checks passed

opensearch-trigger-bot bot mentioned this pull request Feb 3, 2024

[Backport 2.x] Cleanup file cache on deleting index/shard directory #12154

Merged

jed326 mentioned this pull request Nov 6, 2024

Make IndexStoreListener a pluggable interface #16583

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup file cache on deleting index/shard directory #11443

Cleanup file cache on deleting index/shard directory #11443

bugmakerrrrrr commented Dec 4, 2023

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023 •

edited

Loading

bugmakerrrrrr commented Dec 11, 2023

opensearch-trigger-bot bot commented Jan 11, 2024

bugmakerrrrrr commented Jan 11, 2024

andrross commented Jan 11, 2024

bugmakerrrrrr commented Feb 2, 2024

andrross commented Feb 2, 2024

github-actions bot commented Feb 2, 2024

codecov bot commented Feb 2, 2024 •

edited

Loading

github-actions bot commented Feb 2, 2024

Cleanup file cache on deleting index/shard directory #11443

Cleanup file cache on deleting index/shard directory #11443

Conversation

bugmakerrrrrr commented Dec 4, 2023

Description

Related Issues

Check List

github-actions bot commented Dec 4, 2023

github-actions bot commented Dec 4, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

bugmakerrrrrr commented Dec 11, 2023

opensearch-trigger-bot bot commented Jan 11, 2024

bugmakerrrrrr commented Jan 11, 2024

andrross commented Jan 11, 2024

bugmakerrrrrr commented Feb 2, 2024

andrross commented Feb 2, 2024

github-actions bot commented Feb 2, 2024

codecov bot commented Feb 2, 2024 • edited Loading

Codecov Report

github-actions bot commented Feb 2, 2024

github-actions bot commented Dec 4, 2023 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading