Delete might returns false isFound() while primary is relocated

[s1monw]

here is a link to a relevant failure: http://build-us-00.elasticsearch.org/job/es_core_master_metal/5665/

the test is indexing into a single shard with 0 replicas and this only shard is moving from node 0 to node 1. During the time we index the documents we mark the primary as started but until the clusterstate is published on node 0 it depends on which node you hit for indexing if the doc goes into both shards or not. Now if you hit node 1 the doc only goes into the new started (relocated) primary. Yet if you are fast enough and hit node 0 for the delete we can't find the doc in the currently relocating (clusterstate is not yet updated) - the logic knows it needs to push the delete into the new primary on node 1 so it really get's deleted but we return the response from node 0 which has isFound = false

[s1monw]

FYI - I disabled the check for now since it might bring up other failures here and there if timing allows. We should reenable the check once we have a fix - I added a log line instead

[bleskes]

a short update, now that we disabled the hard failure, we see that the tests fail to delete the doc: http://build-us-00.elasticsearch.org/job/es_bwc_1x/5883/CHECK_BRANCH=origin%2F1.4,jdk=JDK7,label=bwc/testReport/junit/org.elasticsearch.bwcompat/BasicBackwardsCompatibilityTest/testRecoverFromPreviousVersion/