[Remote Store] Move blocking refreshes on writes off write thread pool

[Bukhtawar]

Is your feature request related to a problem? Please describe

With wait for refresh semantics write threadpool would be holding up a single lock for refresh till all upload has completed.

"opensearch[1841231c5bfe4d1c5502c71ede14821e][write][T#28]" #8250 daemon prio=5 os_prio=0 cpu=204398.20ms elapsed=25040.27s tid=0x0000fffc28109480 nid=0xba3f waiting on condition  [0x0000ffd9bb6fc000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@17.0.6/Native Method)
        - parking to wait for  <0x000000176ccd4138> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(java.base@17.0.6/LockSupport.java:211)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@17.0.6/AbstractQueuedSynchronizer.java:715)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@17.0.6/AbstractQueuedSynchronizer.java:1047)
        at java.util.concurrent.CountDownLatch.await(java.base@17.0.6/CountDownLatch.java:230)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.syncSegments(RemoteStoreRefreshListener.java:270)
        at org.opensearch.index.shard.RemoteStoreRefreshListener.performAfterRefreshWithPermit(RemoteStoreRefreshListener.java:152)
        at org.opensearch.index.shard.ReleasableRetryableRefreshListener.runAfterRefreshWithPermit(ReleasableRetryableRefreshListener.java:160)
        - locked <0x00000010ab8cc400> (a org.opensearch.index.shard.RemoteStoreRefreshListener)
        at org.opensearch.index.shard.ReleasableRetryableRefreshListener.afterRefresh(ReleasableRetryableRefreshListener.java:66)
        at org.apache.lucene.search.ReferenceManager.notifyRefreshListenersRefreshed(ReferenceManager.java:275)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:182)
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:433)
        at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:413)
        at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
        at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
        at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1860)
        at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1837)
        at org.opensearch.index.shard.IndexShard.refresh(IndexShard.java:1397)
        at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.<init>(TransportWriteAction.java:429)

Describe the solution you'd like

We can offload the blocking part to a queue such that write threads don't get blocked while upload is underway. Once upload is complete the blocked requests in the queue can be notified for completion.

Related component

Storage:Remote

Describe alternatives you've considered

No response

Additional context

No response

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@Bukhtawar Thanks for filing, we'd review a pull request to address this improvement.