[CI] Failure in org.elasticsearch.index.mapper.DynamicMappingTests#testMappingVersionAfterDynamicMappingUpdate #38428
Assignees
Labels
:Search Foundations/Mapping
Index mappings, including merging and defining field types
Team:Search Foundations
Meta label for the Search Foundations team in Elasticsearch
>test-failure
Triaged test failures from CI
v7.2.0
Comments
original-brownbear
added
:Search Foundations/Mapping
|
Pinging @elastic/es-search |
|
I am able to reproduce this after running the tests thousands of times at once. I'm still looking into the cause. |
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this issue
Feb 7, 2019
This assertBusy is necessary. When a mapping update is needed as a document is indexed, the document is tried, rejected (due to mapping conflict), then a mapping update sent off, the document is then *immediately* retried to see if the mapping change has occurred quickly enough, and if it has, indexing does not wait for the next cluster state to occur before moving ahead. In very rare cases this immediate retry succeeds, which causes the indexing request to complete (because it was successful) but the new cluster state to not be propagated entirely yet. In that case, we need to wait because the mapping version will eventually be updated, it just hasn't been updated *yet*. The first mapping-update-necessary check: https://github.com/elastic/elasticsearch/blob/622a7f1e207a552af56fec993045286abc3839e9/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java#L487 Followed immediately by the mapping update: https://github.com/elastic/elasticsearch/blob/622a7f1e207a552af56fec993045286abc3839e9/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java#L490 And then the *immediate* retry: https://github.com/elastic/elasticsearch/blob/622a7f1e207a552af56fec993045286abc3839e9/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java#L499 In the event the immediate retry fails (99.9999% of the time), the context is marked as needing to wait for a new cluster state before proceeding: https://github.com/elastic/elasticsearch/blob/622a7f1e207a552af56fec993045286abc3839e9/server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAction.java#L501-L504 In the 0.0001% case, the immediate retry succeeds, causing the test to fail. I was able to reproduce this bug about once every 10,000 tests. With the awaitsFix I ran this 100,000 times with no failures. Resolves elastic#38428
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this issue
Feb 13, 2019
Prior to this commit, when an indexing operation resulted in an `Engine.Result.Type.MAPPING_UPDATE_REQUIRED`, TransportShardBulkAction immediately retries the indexing operation to see if it succeeds. In the event that it succeeds the context does not wait until the mapping update has propagated through the cluster state before finishing the indexing. In some of our tests we rely on mappings being available as soon as they've been introduced in a document that indexed correctly. By removing the immediate retry we always wait for this to be the case. Resolves elastic#38428 Supercedes elastic#38579 Relates to elastic#38711
dakrone
added a commit
that referenced
this issue
Feb 14, 2019
Prior to this commit, when an indexing operation resulted in an `Engine.Result.Type.MAPPING_UPDATE_REQUIRED`, TransportShardBulkAction immediately retries the indexing operation to see if it succeeds. In the event that it succeeds the context does not wait until the mapping update has propagated through the cluster state before finishing the indexing. In some of our tests we rely on mappings being available as soon as they've been introduced in a document that indexed correctly. By removing the immediate retry we always wait for this to be the case. Resolves #38428 Supercedes #38579 Relates to #38711
dakrone
added a commit
that referenced
this issue
Feb 14, 2019
Prior to this commit, when an indexing operation resulted in an `Engine.Result.Type.MAPPING_UPDATE_REQUIRED`, TransportShardBulkAction immediately retries the indexing operation to see if it succeeds. In the event that it succeeds the context does not wait until the mapping update has propagated through the cluster state before finishing the indexing. In some of our tests we rely on mappings being available as soon as they've been introduced in a document that indexed correctly. By removing the immediate retry we always wait for this to be the case. Resolves #38428 Supercedes #38579 Relates to #38711
dakrone
added a commit
that referenced
this issue
Feb 14, 2019
Prior to this commit, when an indexing operation resulted in an `Engine.Result.Type.MAPPING_UPDATE_REQUIRED`, TransportShardBulkAction immediately retries the indexing operation to see if it succeeds. In the event that it succeeds the context does not wait until the mapping update has propagated through the cluster state before finishing the indexing. In some of our tests we rely on mappings being available as soon as they've been introduced in a document that indexed correctly. By removing the immediate retry we always wait for this to be the case. Resolves #38428 Supercedes #38579 Relates to #38711
dakrone
added a commit
that referenced
this issue
Feb 14, 2019
Prior to this commit, when an indexing operation resulted in an `Engine.Result.Type.MAPPING_UPDATE_REQUIRED`, TransportShardBulkAction immediately retries the indexing operation to see if it succeeds. In the event that it succeeds the context does not wait until the mapping update has propagated through the cluster state before finishing the indexing. In some of our tests we rely on mappings being available as soon as they've been introduced in a document that indexed correctly. By removing the immediate retry we always wait for this to be the case. Resolves #38428 Supercedes #38579 Relates to #38711
javanna
added
the
Team:Search Foundations
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+periodic/670/console
Failed with:
The text was updated successfully, but these errors were encountered: