[CI] CCRIT testIndexFollowing Failure

[jdconrad]

I was unable to reproduce locally, but have seen this failure twice now in CI.

CONSOLE:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/491/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/496/console

REPRODUCE WITH:

./gradlew :client:rest-high-level:integTestRunner \
  -Dtests.seed=123CB2463E70E344 \
  -Dtests.class=org.elasticsearch.client.CCRIT \
  -Dtests.method="testIndexFollowing" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-ES \
  -Dtests.timezone=Africa/Niamey \
  -Dcompiler.java=11 \
  -Druntime.java=8

FAILURE:

13:59:41    > Throwable #1: java.lang.AssertionError: 
13:59:41    > Expected: <0L>
13:59:41    >      but: was <-1L>
13:59:41   2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+intake/client/rest-high-level/build/testrun/integTestRunner/J0/temp/org.elasticsearch.client.CCRIT_123CB2463E70E344-001
13:59:41    > 	at __randomizedtesting.SeedInfo.seed([123CB2463E70E344:9DE2A28CBEC49EE7]:0)
13:59:41    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
13:59:41   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=232, maxMBSortInHeap=6.833528072827601, sim=RandomSimilarity(queryNorm=false): {}, locale=es-ES, timezone=Africa/Niamey
13:59:41    > 	at org.elasticsearch.client.CCRIT.lambda$testIndexFollowing$1(CCRIT.java:121)
13:59:41    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
13:59:41    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
13:59:41    > 	at org.elasticsearch.client.CCRIT.testIndexFollowing(CCRIT.java:113)
13:59:41    > 	at java.lang.Thread.run(Thread.java:748)
13:59:41    > 	Suppressed: java.lang.AssertionError: 
13:59:41   2> NOTE: Linux 3.10.0-862.14.4.el7.x86_64 amd64/Oracle Corporation 1.8.0_192 (64-bit)/cpus=16,threads=1,free=429691976,total=517996544
13:59:41   2> NOTE: All tests run in this JVM: [StoredScriptsDocumentationIT, PingAndInfoIT, MlClientDocumentationIT, StoredScriptsIT, LicensingDocumentationIT, BulkRequestWithGlobalParametersIT, WatcherDocumentationIT, IngestClientDocumentationIT, SnapshotIT, BulkProcessorIT, CCRDocumentationIT, IndicesClientDocumentationIT, IngestClientIT, SearchDocumentationIT, RollupIT, LicenseIT, CrudIT, ClusterClientIT, GraphIT, CCRIT]```

[elasticmachine]

Pinging @elastic/es-distributed

[jdconrad]

@martijnvg I noted that you commented on the last failure involving CCRIT, so I assigned this to you. Apologies, if it should be someone else!

[jdconrad]

Another on master:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/574/console

[jdconrad]

Muted this test on master and 6.x

[martijnvg]

I couldn't figure out why it fails and could not reproduce. I re-enabled this test on master with extra logging: 38a1d4f
When it fails then there if more info and if it starts failing too often then I will mute the test.

[tlrx]

I might have opened a duplicate of this issue: #37231.

testIndexFollowing failed again today:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1127

[martijnvg]

@tlrx The test here is in high level rest client and the test that failed in the build is a rolling upgrade test. The test names happen to be the same.

[martijnvg]

Closing as the high level rest client ccr tests have not been failing for a while now.