Added `_shards` header to all write responses. #7994

martijnvg · 2014-10-06T13:29:56Z

The header indicates to how many shard copies (primary and replicas shards) a write was supposed to go to, to how many shard copies to write succeeded and potentially captures shard failures if writing into a replica shard fails.

For async writes it also includes the number of shards a write is still pending.

bleskes · 2014-10-08T12:03:02Z

docs/reference/docs/index_.asciidoc

+* `successful`- Indicates the number of shard copies to index operation succeeded on.
+* `pending` - Indicates to how many shard copies this index operation still needs to go to at the time index operation
+ succeeded on the primary shard. This field is only returned if `async` replication is used.
+* `failures` - An array that contains replication related errors in the case an index operation failed on a replica shard.


I think we need failed: INT , just like in search.

+1 I'll add that.

martijnvg · 2014-10-09T21:44:56Z

@javanna @bleskes I updated the PR and the provided feedback has been applied.

The other most noticeable change is that the replication logic in TransportShardReplicationOperationAction has been changed to keep better track of the state of replication while it is being performed. This helped to simplify the replication logic.

javanna · 2014-10-10T08:30:50Z

docs/reference/docs/delete.asciidoc

@@ -16,6 +16,10 @@ The result of the above delete operation is:
 [source,js]
 --------------------------------------------------
 {
+    "_shards" : {
+        "total" : 10,
+        "successful" : 10


missing failed?

I added it to the code, I'll add it to the examples as well!

bleskes · 2014-12-08T20:54:14Z

...a/org/elasticsearch/action/support/replication/TransportIndexReplicationOperationAction.java

+                                // The failure doesn't include the node id, maybe add it to ShardOperationFailedException...
+                                ShardOperationFailedException sf = shardActionResult.shardFailure;
+
+                                ShardIterator thisShardIterator = null;


why not capture the shard info during the failure handling? Then we don't need to search for the iterator..

good call, let me change that.

bleskes · 2014-12-08T20:56:29Z

Thx @martijnvg .

Not sure if we should repeat the primary exception or just tell we never executed on replica shards.

yeah, I see what you mean. I was thinking wrapping the primary failure with a new exception which has the right info in the message? (ala "Failed to execute on shard [](primary + [] replicas): [MESSAGE]")

left some other minor comments.

martijnvg · 2014-12-09T12:24:34Z

Thanks for the feedback @bleskes. I updated the PR. Since we don't have exceptions (the Failure class has no failure wrapping) for the replica shards, I instead just wrap the original primary shard failure on the message of the replica shard failure. I think this is sufficient?

martijnvg · 2015-01-06T09:40:07Z

@bleskes @javanna I applied the feedback and Boaz's commits (bleskes@a9b659d and bleskes@6180417), rebased to master and squashed everything to a single commit for clarity purposes.

I think this is getting close to get merged now, would be great if someone else can take a look at the PR before it gets merged in.

imotov · 2015-01-06T16:59:26Z

src/test/java/org/elasticsearch/deleteByQuery/DeleteByQueryTests.java

+
+    private void assertSyncShardInfo(ActionWriteResponse.ShardInfo shardInfo, NumShards numShards) {
+        assertThat(shardInfo.getTotal(), equalTo(numShards.totalNumShards));
+        assertThat(shardInfo.getSuccessful(), greaterThanOrEqualTo(1));


Why not greaterThanOrEqualTo(numShards.numPrimaries) or even `equalTo(numShards.totalNumShards)?

I don't know why this wasn't using numShards.numPrimaries, let me change that!

imotov · 2015-01-06T17:36:38Z

I left a couple of small comments. In general looks good to me.

martijnvg · 2015-01-07T06:47:27Z

@imotov I updated the PR and applied your comments.

imotov · 2015-01-07T15:23:13Z

LGTM

The header indicates to how many shard copies (primary and replicas shards) a write was supposed to go to, to how many shard copies to write succeeded and potentially captures shard failures if writing into a replica shard fails. For async writes it also includes the number of shards a write is still pending. Closes elastic#7994

martijnvg added v1.5.0 v2.0.0-beta1 >enhancement resiliency and removed resiliency labels Oct 6, 2014

bleskes reviewed Oct 8, 2014
View reviewed changes

martijnvg force-pushed the feature/shard_info_write_responses branch from f728490 to 9af8a87 Compare October 9, 2014 21:08

javanna reviewed Oct 10, 2014
View reviewed changes

bleskes reviewed Dec 8, 2014
View reviewed changes

martijnvg force-pushed the feature/shard_info_write_responses branch from c99bb0f to f6ecd57 Compare January 6, 2015 09:35

imotov reviewed Jan 6, 2015
View reviewed changes

martijnvg changed the title ~~Added _shards header to all write responses.~~ Core: added _shards header to all write responses. Jan 8, 2015

martijnvg added the :Core/Infra/Core Core issues without another label label Jan 8, 2015

martijnvg force-pushed the feature/shard_info_write_responses branch from 9f21553 to 6fc0992 Compare January 8, 2015 16:24

martijnvg force-pushed the feature/shard_info_write_responses branch from 6fc0992 to 33f4241 Compare January 8, 2015 16:35

martijnvg force-pushed the feature/shard_info_write_responses branch from 33f4241 to 1951ba3 Compare January 8, 2015 17:00

martijnvg force-pushed the feature/shard_info_write_responses branch from 1951ba3 to 9f10744 Compare January 8, 2015 17:09

martijnvg force-pushed the feature/shard_info_write_responses branch from 9f10744 to ca4f27f Compare January 8, 2015 17:10

martijnvg merged commit ca4f27f into elastic:master Jan 8, 2015

martijnvg added the resiliency label Mar 2, 2015

martijnvg mentioned this pull request Mar 2, 2015

Docs: updated resilience page #9949

Merged

martijnvg deleted the feature/shard_info_write_responses branch May 18, 2015 23:27

clintongormley added :Core/Infra/REST API REST infrastructure and utilities and removed :Core/Infra/Core Core issues without another label labels Jun 6, 2015

clintongormley changed the title ~~Core: added _shards header to all write responses.~~ Added _shards header to all write responses. Jun 6, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added `_shards` header to all write responses. #7994

Added `_shards` header to all write responses. #7994

martijnvg commented Oct 6, 2014

bleskes Oct 8, 2014

martijnvg Oct 8, 2014

martijnvg commented Oct 9, 2014

javanna Oct 10, 2014

martijnvg Oct 10, 2014

bleskes Dec 8, 2014

martijnvg Dec 9, 2014

bleskes commented Dec 8, 2014

martijnvg commented Dec 9, 2014

martijnvg commented Jan 6, 2015

imotov Jan 6, 2015

martijnvg Jan 7, 2015

imotov commented Jan 6, 2015

martijnvg commented Jan 7, 2015

imotov commented Jan 7, 2015

Added _shards header to all write responses. #7994

Added _shards header to all write responses. #7994

Conversation

martijnvg commented Oct 6, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martijnvg commented Oct 9, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes commented Dec 8, 2014

martijnvg commented Dec 9, 2014

martijnvg commented Jan 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imotov commented Jan 6, 2015

martijnvg commented Jan 7, 2015

imotov commented Jan 7, 2015

Added `_shards` header to all write responses. #7994

Added `_shards` header to all write responses. #7994