Added CCR rolling upgrade tests #36648

martijnvg · 2018-12-14T16:33:13Z

Tests index following and auto following in a rolling upgrade scenario.

Test are written as Java tests using low level client instead of yaml tests, because CCR needs assertBusy(...) snippets to be tested correctly and that is not possible in yaml tests.

elasticmachine · 2018-12-14T16:33:15Z

Pinging @elastic/es-distributed

martijnvg · 2018-12-14T16:35:55Z

This test is currently marked as WIP as it depends on a bug fix: #36647 and
sometime it fails with serialization errors that occur when a 6.6.0-SNAPSHOT node receives a shard changes reponse from a 7.0.0-SNAPSHOT node:

Caused by: java.lang.IllegalArgumentException: No version type match [-1]
        at org.elasticsearch.index.VersionType.fromValue(VersionType.java:368) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.translog.Translog$Index.<init>(Translog.java:1151) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.translog.Translog$Index.<init>(Translog.java:1120) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.translog.Translog$Operation.readOperation(Translog.java:1072) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.io.stream.StreamInput.readArray(StreamInput.java:734) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ccr.action.ShardChangesAction$Response.readFrom(ShardChangesAction.java:282) ~[?:?]

I'm not yet sure how this can happen, because the serialization logic in Translog.Index looks okay to me in both master and 6.x branches.

martijnvg · 2018-12-14T19:09:10Z

I've also noticed this serialization error in bulk shards operations api:

[2018-12-14T19:52:59,864][WARN ][o.e.x.c.a.ShardFollowNodeTask] [upgraded-node-0] shard follow task encounter non-retryable error
org.elasticsearch.transport.RemoteTransportException: [node-2][127.0.0.1:34723][indices:data/write/bulk_shard_operations[s]]
Caused by: java.lang.IllegalArgumentException: No version type match [-1]
        at org.elasticsearch.index.VersionType.fromValue(VersionType.java:368) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.translog.Translog$Index.<init>(Translog.java:1151) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.translog.Translog$Index.<init>(Translog.java:1120) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.index.translog.Translog$Operation.readOperation(Translog.java:1072) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:959) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.common.io.stream.StreamInput.readList(StreamInput.java:941) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
        at org.elasticsearch.xpack.ccr.action.bulk.BulkShardOperationsRequest.readFrom(BulkShardOperationsRequest.java:54) ~[?:?]

dnhatn · 2018-12-19T17:18:02Z

@martijnvg I have merged #36676 which should fix the translog serialization issue.

martijnvg · 2018-12-20T14:49:47Z

run the gradle build tests 1

martijnvg · 2018-12-20T14:49:58Z

run the gradle build tests 2

that resets the auto follow stats

martijnvg · 2018-12-21T12:39:40Z

@dnhatn I've update this PR. Do you want to take another look?

dnhatn

Thanks @martijnvg. I like these tests - they are easy to follow. I left some minor comments.

dnhatn · 2019-01-02T05:23:46Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+
+    private static void putAutoFollowPattern(String patternName) throws IOException {
+        Request request = new Request("PUT", "/_ccr/auto_follow/" + patternName);
+        request.setJsonEntity("{\"leader_index_patterns\": [\"logs-*\"], \"remote_cluster\": \"local\"," +


It's easier to follow if we pass the pattern (that's logs-*) as a parameter.

dnhatn · 2019-01-02T05:30:13Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+    }
+
+    private void assertFollowerGlobalCheckpoint(String followerIndex, int expectedFollowerCheckpoint) throws IOException {
+        Request statsRequest = new Request("GET", "/" + followerIndex + "/_ccr/stats");


I think a seq_no stats of the following index would give us a stronger confidence than the ccr stats.

dnhatn · 2019-01-02T05:32:13Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+        }
+    }
+
+    private void assertNumberOfSuccessfulFollowedIndices(int expectedNumberOfSuccessfulFollowedIndices) throws IOException {


Maybe remove this method and assert the result of "getNumberOfSuccessfulFollowedIndices"?

dnhatn · 2019-01-02T05:34:33Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+
+                pauseFollow("copy-" + leaderIndex2);
+                closeIndex("copy-" + leaderIndex2);
+                unfollow("copy-" + leaderIndex2);


Maybe do all pause, close and unfollow in a single method?

dnhatn · 2019-01-02T05:38:02Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+            case MIXED:
+                if (SECOND_ROUND == false) {
+                    index(leaderIndex, "2");
+                    assertDocumentExists(leaderIndex, "2");


Maybe assert the existence of both doc1 and doc2? You can extend assertDocumentExists to accept varargs.

martijnvg · 2019-01-07T09:59:06Z

@dnhatn I've updated the PR.

dnhatn

LGTM. Thanks @martijnvg. I left minor nits but feel free to merge once CI is happy.

dnhatn · 2019-01-07T18:14:44Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+                assertDocumentExists(leaderIndex, "4");
+                assertBusy(() -> {
+                    assertFollowerGlobalCheckpoint(followerIndex, 3);
+                    assertDocumentExists(followerIndex, "4");


nit: can we also verify doc1, doc2, and doc3?

dnhatn · 2019-01-07T18:15:02Z

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/CCRIT.java

+                    assertDocumentExists(leaderIndex, "3");
+                    assertBusy(() -> {
+                        assertFollowerGlobalCheckpoint(followerIndex, 2);
+                        assertDocumentExists(followerIndex, "3");


nit: can we also verify doc1, and doc2

martijnvg · 2019-01-08T10:05:59Z

I will backport this later to 6.x, if I didn't encounter build failures related to this change.

Added CCR rolling upgrade tests.

These tests were added via #36648

Added CCR rolling upgrade tests

af67a13

martijnvg added >test Issues or PRs that are addressing/adding tests WIP v7.0.0 :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features v6.6.0 labels Dec 14, 2018

martijnvg added 2 commits December 14, 2018 19:47

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

3ba5d9a

re-ordered assertions

ff096c7

fix translog serialization

1f242fd

dnhatn mentioned this pull request Dec 15, 2018

Fix translog bwc serialization #36676

Merged

jasontedor added v6.7.0 and removed v6.6.0 labels Dec 19, 2018

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

52cb061

martijnvg removed the WIP label Dec 20, 2018

martijnvg added 3 commits December 20, 2018 09:29

adjusted version

fa4b45d

take into account that mixed cluster type is ran twice

f250c0f

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

0f9b9f4

martijnvg added 2 commits December 21, 2018 09:40

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

ca10738

take into account that elected master node also gets upgraded and

d186232

that resets the auto follow stats

martijnvg added 3 commits December 24, 2018 08:55

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

4145476

take into account that no shard follow stats can be returned

99d53a2

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

72d25d8

dnhatn reviewed Jan 2, 2019

View reviewed changes

martijnvg added 3 commits January 7, 2019 09:21

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

85371ed

iter

78754af

fixed test issue

c5b8071

dnhatn approved these changes Jan 7, 2019

View reviewed changes

martijnvg added 2 commits January 8, 2019 09:42

iter

62461ef

Merge remote-tracking branch 'es/master' into ccr_rolling_upgrade_tests

f554102

martijnvg merged commit c980cc1 into elastic:master Jan 8, 2019

martijnvg added the backport pending label Jan 8, 2019

martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request Jan 11, 2019

Added CCR rolling upgrade tests (elastic#36648)

4779525

Added CCR rolling upgrade tests.

martijnvg removed backport pending v6.7.0 labels Jan 11, 2019

martijnvg added a commit that referenced this pull request Jan 12, 2019

Backported CCR rolling upgrade tests (#37368)

fcecaee

These tests were added via #36648

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added CCR rolling upgrade tests #36648

Added CCR rolling upgrade tests #36648

martijnvg commented Dec 14, 2018

elasticmachine commented Dec 14, 2018

martijnvg commented Dec 14, 2018

martijnvg commented Dec 14, 2018

dnhatn commented Dec 19, 2018

martijnvg commented Dec 20, 2018

martijnvg commented Dec 20, 2018

martijnvg commented Dec 21, 2018

dnhatn left a comment

dnhatn Jan 2, 2019

dnhatn Jan 2, 2019

dnhatn Jan 2, 2019

dnhatn Jan 2, 2019

dnhatn Jan 2, 2019

martijnvg commented Jan 7, 2019

dnhatn left a comment

dnhatn Jan 7, 2019

dnhatn Jan 7, 2019

martijnvg commented Jan 8, 2019

Added CCR rolling upgrade tests #36648

Added CCR rolling upgrade tests #36648

Conversation

martijnvg commented Dec 14, 2018

elasticmachine commented Dec 14, 2018

martijnvg commented Dec 14, 2018

martijnvg commented Dec 14, 2018

dnhatn commented Dec 19, 2018

martijnvg commented Dec 20, 2018

martijnvg commented Dec 20, 2018

martijnvg commented Dec 21, 2018

dnhatn left a comment

Choose a reason for hiding this comment

dnhatn Jan 2, 2019

Choose a reason for hiding this comment

dnhatn Jan 2, 2019

Choose a reason for hiding this comment

dnhatn Jan 2, 2019

Choose a reason for hiding this comment

dnhatn Jan 2, 2019

Choose a reason for hiding this comment

dnhatn Jan 2, 2019

Choose a reason for hiding this comment

martijnvg commented Jan 7, 2019

dnhatn left a comment

Choose a reason for hiding this comment

dnhatn Jan 7, 2019

Choose a reason for hiding this comment

dnhatn Jan 7, 2019

Choose a reason for hiding this comment

martijnvg commented Jan 8, 2019