[BUG] [Remote Store] RemoteStoreRefreshListenerTests.testAfterCommit flaky test failure

[dreamer-89]

Coming from meta issue tracking Segment Replication flaky test failures

3 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterCommit (19629,19844,19937)

[sachinpkale]

Not able to reproduce in local (ran successfully for 200+ times). Resolving.

[sachinpkale]

Occurred again: https://build.ci.opensearch.org/job/gradle-check/24227/testReport/junit/org.opensearch.index.shard/RemoteStoreRefreshListenerTests/testAfterCommit/

[mch2]

Another - https://build.ci.opensearch.org/job/gradle-check/27720/

[ashking94]

When the test is run in isolation, it did not fail even after 1K iterations. But when this runs as part of the entire class, seeing failures with below exception -

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterCommit" -Dtests.seed=81C8CA41597DE51F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=uk -Dtests.timezone=America/Belize -Druntime.java=21

org.opensearch.index.shard.RemoteStoreRefreshListenerTests > testAfterCommit FAILED
    java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
        at __randomizedtesting.SeedInfo.seed([81C8CA41597DE51F:CCBB4D0438019FC1]:0)
        at org.apache.lucene.tests.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:876)
        at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
        at org.apache.lucene.store.FilterDirectory.close(FilterDirectory.java:111)
        at org.opensearch.index.store.Store$StoreDirectory.innerClose(Store.java:952)
        at org.opensearch.index.store.Store.closeInternal(Store.java:571)
        at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
        at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
        at org.opensearch.index.store.Store.decRef(Store.java:546)
        at org.opensearch.index.store.Store.close(Store.java:553)
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
        at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)
        at org.opensearch.index.shard.IndexShardTestCase.closeShard(IndexShardTestCase.java:965)
        at org.opensearch.index.shard.IndexShardTestCase.closeShards(IndexShardTestCase.java:972)
        at org.opensearch.index.shard.IndexShardTestCase.closeShards(IndexShardTestCase.java:952)
        at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.tearDown(RemoteStoreRefreshListenerTests.java:105)

        Caused by:
        java.lang.RuntimeException: unclosed IndexInput: _0.cfs
            at org.apache.lucene.tests.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:783)
            at org.apache.lucene.tests.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:835)
            at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
            at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
            at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:78)
            at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:87)
            at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
            at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:96)
            at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178)
            at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220)
            at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542)
            at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138)
            at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:604)
            at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
            at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355)
            at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345)
            at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
            at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
            at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72)
            at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52)
            at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
            at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
            at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:433)
            at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:413)
            at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
            at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
            at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1771)
            at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1748)
            at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1341)
            at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.setup(RemoteStoreRefreshListenerTests.java:81)
            at org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testAfterCommit(RemoteStoreRefreshListenerTests.java:196)```