[Tiered Caching] [Bug Fix] Use concurrentMap instead of HashMap to fix Concurrent Modification Exception

[kiranprakash154]

Description

#14032 reported msearches failing with a Concurrent Modification Exception, though they were not able to repro it consistently they were able to attach a debugger and find out the line throwing the exception, which was

OpenSearch/server/src/main/java/org/opensearch/indices/IndicesRequestCache.java

Line 571 in 5b93f2e

cleanupKeyToCountMap.computeIfAbsent(shardId, k -> new HashMap<>()).merge(cleanupKey.readerCacheKeyId, 1, Integer::sum);

We are using a HashMap (not thread safe) for the inner map of the cleanupKeyToCountMap and hence it throws a Concurrent Modification Exception when the map is getting updated by multiple threads concurrently.

The fix is to use a thread safe Concurrent Map instead.

Related Issues

Resolves #14032

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 5a4484d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[andrross]

Approving as we can follow up on my comments later if there's a rush to merge this. Letting @sohami have a final look.

[sohami]

@kiranprakash154 Discussed with @andrross. Merging the change for 2.15, but we will need follow-up to address the comments here

[kiranprakash154]

@kiranprakash154 Discussed with @andrross. Merging the change for 2.15, but we will need follow-up to address the comments here

Yep, I'd love that. Thanks for helping this get through