Document Seq No powered optimistic concurrency control #37284

bleskes · 2019-01-10T01:16:13Z

Add documentation to describe the new sequence number powered optimistic concurrency control

Relates #36148
Relates #10708

elasticmachine · 2019-01-10T01:16:15Z

Pinging @elastic/es-distributed

zuketo

Looks good, just a few minor suggestions

zuketo · 2019-01-10T17:18:45Z

docs/reference/docs/bulk.asciidoc

+=== Optimistic Concurrency Control
+
+Each index and delete bulk item can include the `if_seq_no` and `if_primary_term`
+parameters in their respective action and meta data lines. These parameters


Each index and delete action within a bulk API call may include the if_seq_no and if_primary_term parameters in their respective action and meta data lines.

thx. adapted

zuketo · 2019-01-10T17:23:06Z

docs/reference/docs/bulk.asciidoc

+Each index and delete bulk item can include the `if_seq_no` and `if_primary_term`
+parameters in their respective action and meta data lines. These parameters
+allow controlling how these operations will be performed based on the last
+modification to existing documents. See <<optimistic-concurrency-control>> for more details.


The if_seq_no and if_primary_term parameters control how operations are executed, based on the most recently modified version of an existing document.

Adopted the first part. I left out the "modified version" suggestion because it's not about the document version but rather the seq# and primary term of the modification.

zuketo · 2019-01-10T17:24:09Z

docs/reference/docs/bulk.asciidoc

+allow controlling how these operations will be performed based on the last
+modification to existing documents. See <<optimistic-concurrency-control>> for more details.
+
+


Can we include a very short example for passing the if_seq_no and if_primary_term parameters to the bulk API?

good question. we don't do it for other parameters, like routing etc. I think we should consistent and adding it to all may be very verbose.

+1, good point

zuketo · 2019-01-10T17:26:38Z

docs/reference/docs/concurrency-control.asciidoc

+Elasticsearch needs a way of ensuring that an older version of a document never 
+overwrites a newer version.
+
+To ensure this, every operation performed to a document is assigned a sequence number


To ensure an older version of a document doesn't overwrite a newer version, every operation performed is assigned a sequence number ...

zuketo · 2019-01-10T17:27:17Z

docs/reference/docs/concurrency-control.asciidoc

+To ensure this, every operation performed to a document is assigned a sequence number
+by the primary shard that coordinates that change. The sequence number is increased
+with each operation and thus newer operations are guaranteed to have a higher sequence
+number. Elasticsearch can then use the sequence number of operations to make sure they


than older operations

zuketo · 2019-01-10T17:28:01Z

docs/reference/docs/concurrency-control.asciidoc

+never override a newer document version is never overridden by a change that has a 
+smaller sequence number assigned to it. 
+
+For example, the following indexing command will create a document and assign it its 


it its = an

zuketo · 2019-01-10T17:33:26Z

docs/reference/docs/concurrency-control.asciidoc

+
+
+Elasticsearch keeps tracks of the sequence number and primary of the last
+operation to have changed each of the document it stores. These are returned


Elasticsearch keeps tracks of the sequence number and primary term for every document it stores, this value will change based on the last operation to modify the document. The _seq_no and _primary_term fields are returned

I will adopt the second part of the suggestion, but I feel the first part is not accurate - there's no sequence number of a document but rather a seq of the last operation to have changed the doc.

zuketo · 2019-01-10T17:34:19Z

docs/reference/docs/concurrency-control.asciidoc

+for each search hit by requesting the `_seq_no` and `_primary_term` <<search-request-docvalue-fields,Doc Value Fields>>.
+
+The sequence number and the primary term uniquely identify a change. By noting down 
+the sequence number and primary term return, you can make sure to only change the 


zuketo · 2019-01-10T17:35:04Z

docs/reference/docs/concurrency-control.asciidoc

+is done by setting the `if_seq_no` and `if_primary_term` parameters of either the
+<<docs-index_,Index API>> or the <<docs-delete,Delete API>>. 
+
+For example, the following indexing call, will make sure to add a tag to the


no comma needed:

For example, the following indexing call will make sure to add a tag to the

bleskes · 2019-01-11T15:09:36Z

Thanks @zuketo . I addressed your comments. Can you please take another look?

zuketo

lgtm

zuketo · 2019-01-11T15:14:57Z

docs/reference/docs/bulk.asciidoc

+allow controlling how these operations will be performed based on the last
+modification to existing documents. See <<optimistic-concurrency-control>> for more details.
+
+


+1, good point

ywelsch · 2019-01-11T15:49:16Z

docs/reference/docs/concurrency-control.asciidoc

+operation and thus newer operations are guaranteed to have a higher sequence
+number than older operations. Elasticsearch can then use the sequence number of
+operations to make sure they never override a newer document version is never
+overridden by a change that has a smaller sequence number assigned to it.


something is wrong with this sentence.

ywelsch · 2019-01-11T15:52:10Z

docs/reference/docs/index_.asciidoc

+
+The above will succeed since the the supplied version of 2 is higher than
+the current document version of 1. If the document was already updated
+and it's version was set to 2 or higher, the indexing command will fail


ywelsch · 2019-01-11T15:52:45Z

docs/reference/docs/index_.asciidoc

+WARNING: External versioning supports the value 0 as a valid version number.
+This allows the version to be in sync with an external versioning system
+where version numbers start from zero instead of one. It has the side effect
+that documents with version number equal to zero cannot neither be updated


can neither

ywelsch · 2019-01-11T15:53:47Z

docs/reference/docs/index_.asciidoc

+database, as long as version numbers from the source database are used.
+Even the simple case of updating the Elasticsearch index using data from
+a database is simplified if external versioning is used, as only the
+latest version will be used if the index operations are out of order for


arrive out of order

ywelsch · 2019-01-11T15:55:28Z

docs/reference/docs/index_.asciidoc

+the different version types and their semantics.
+
+`internal`:: only index the document if the given version is identical to the version
+of the stored document.


should we point to optimistic concurrency control here and make it clear that using if_seq_no is the preferred method for this?

I decided to only do this once internal versioning is deprecated. I can add a note if you prefer

I decided to only do this once internal versioning is deprecated

ok, if the deprecation is being added to 6.6, I'm fine with that.

I don't think we'll make the 6.6 time frame as we need to first remove all usages of it (only reindex is left)

bleskes · 2019-01-11T16:00:58Z

@ywelsch sorry, I didn't see your comments on the page I've hit the merge button on. I'll address them in a follow up commit.

Add documentation to describe the new sequence number powered optimistic concurrency control Relates #36148 Relates #10708

bleskes added 4 commits January 9, 2019 06:57

initial docs

6e9cec7

tabs begone

51ddfa7

fix tests

83fd8ad

more fixes

5d6071c

bleskes added :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. v7.0.0 v6.6.0 labels Jan 10, 2019

bleskes requested a review from zuketo January 10, 2019 01:16

zuketo reviewed Jan 10, 2019

View reviewed changes

feedback

7c19495

bleskes requested a review from zuketo January 11, 2019 15:09

zuketo approved these changes Jan 11, 2019

View reviewed changes

ywelsch approved these changes Jan 11, 2019

View reviewed changes

bleskes added the v6.7.0 label Jan 11, 2019

bleskes merged commit cae71cd into elastic:master Jan 11, 2019

bleskes deleted the cas_doc branch January 11, 2019 16:08

bleskes added a commit that referenced this pull request Jan 11, 2019

followup to #37284 with additional feedback

093d4a3

bleskes added a commit that referenced this pull request Jan 11, 2019

Document Seq No powered optimistic concurrency control (#37284)

2ee2c29

Add documentation to describe the new sequence number powered optimistic concurrency control Relates #36148 Relates #10708

bleskes added a commit that referenced this pull request Jan 11, 2019

followup to #37284 with additional feedback

439944f

bleskes added a commit that referenced this pull request Jan 11, 2019

Document Seq No powered optimistic concurrency control (#37284)

28069db

Add documentation to describe the new sequence number powered optimistic concurrency control Relates #36148 Relates #10708

bleskes added a commit that referenced this pull request Jan 11, 2019

followup to #37284 with additional feedback

3e98069

pcsanwald added the >enhancement label Jan 17, 2019

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document Seq No powered optimistic concurrency control #37284

Document Seq No powered optimistic concurrency control #37284

bleskes commented Jan 10, 2019

elasticmachine commented Jan 10, 2019

zuketo left a comment

zuketo Jan 10, 2019

bleskes Jan 11, 2019 •

edited

Loading

zuketo Jan 10, 2019

bleskes Jan 11, 2019

zuketo Jan 10, 2019

bleskes Jan 11, 2019

zuketo Jan 11, 2019

zuketo Jan 10, 2019

zuketo Jan 10, 2019

bleskes Jan 11, 2019

zuketo Jan 10, 2019

zuketo Jan 10, 2019

bleskes Jan 11, 2019

zuketo Jan 10, 2019

bleskes Jan 11, 2019

zuketo Jan 10, 2019

bleskes commented Jan 11, 2019

zuketo left a comment

zuketo Jan 11, 2019

ywelsch Jan 11, 2019

ywelsch Jan 11, 2019

ywelsch Jan 11, 2019

ywelsch Jan 11, 2019

ywelsch Jan 11, 2019

bleskes Jan 11, 2019

ywelsch Jan 11, 2019

bleskes Jan 15, 2019

bleskes commented Jan 11, 2019

		allow controlling how these operations will be performed based on the last
		modification to existing documents. See <<optimistic-concurrency-control>> for more details.



		Elasticsearch keeps tracks of the sequence number and primary of the last
		operation to have changed each of the document it stores. These are returned

Document Seq No powered optimistic concurrency control #37284

Document Seq No powered optimistic concurrency control #37284

Conversation

bleskes commented Jan 10, 2019

elasticmachine commented Jan 10, 2019

zuketo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes Jan 11, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes commented Jan 11, 2019

zuketo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes commented Jan 11, 2019

bleskes Jan 11, 2019 •

edited

Loading