You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Upon indexing documents to an index with segment replication enabled, sometimes search requests sent to that index hang indefinitely. This behavior is noticed both before and after refreshing the index.
To Reproduce
Steps to reproduce the behavior:
Run gradle with 2 nodes
Create an index with segment replication enabled curl -X PUT "localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d '{"settings" : { "index" : { "replication.type" : "SEGMENT" }}}'
Can use the cat shards api to see that documents are being added and deleted as expected. Occasionally, may need to refresh to see the correct updated value.
curl -X GET "localhost:9200/_cat/shards?pretty" curl -X POST "localhost:9200/my-index-000001/_refresh?pretty"
Run any search request to the index to see if it hangs indefinitely instead of providing an output. May take a few tries to see this behavior.
On digging into this issue, I found that the search requests hang whenever they are routed to the replica instead of the primary. This happens on all occasions even when we can see the files are copied over in the directory and through _cat/shards calls.
Can follow the below steps to confirm:
Use these two commands to determine which is the primary and replica and their specific node IDs.
curl -X GET "localhost:9200/_cat/nodes?v=true&h=id,ip,port,v,m,n&pretty"
curl -X GET "localhost:9200/_cat/shards?pretty"
Send the search requests with preference set to specific node IDs to see when the search request hangs. This query is of type -
curl -X GET "localhost:9200/my-index-000001/_search?preference=&pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"match_all" : {}
}
}
'
You can see that all the search requests routed to the primary provide an output and all routed to replica hang indefinitely. I think this is a good starting point to dig into what could be wrong.
This is related to external refresh listeners not getting wired in NRTReplicationEngine, so awaitShardSearchActive. Never resolves. Working on a PR to fix.
Describe the bug
Upon indexing documents to an index with segment replication enabled, sometimes search requests sent to that index hang indefinitely. This behavior is noticed both before and after refreshing the index.
To Reproduce
Steps to reproduce the behavior:
Run gradle with 2 nodes
Create an index with segment replication enabled
curl -X PUT "localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d '{"settings" : { "index" : { "replication.type" : "SEGMENT" }}}'
Add some documents to the index
curl -X GET "localhost:9200/_cat/shards?pretty"
curl -X POST "localhost:9200/my-index-000001/_refresh?pretty"
Expected behavior
All search requests should be able to resolve with the correct output of documents in the index quickly.
The text was updated successfully, but these errors were encountered: