[CI] LegacyDynamicMappingIT testMappingsPropagatedToMasterNodeImmediatelyMultiType fails in 6.x

[droberts195]

LegacyDynamicMappingIT testMappingsPropagatedToMasterNodeImmediatelyMultiType failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1150/console

The error was:

12:20:09 ERROR   12.9s J2 | LegacyDynamicMappingIT.testMappingsPropagatedToMasterNodeImmediatelyMultiType <<< FAILURES!
12:20:09    > Throwable #1: java.lang.NullPointerException
12:20:09    > 	at __randomizedtesting.SeedInfo.seed([ED2233D5C1DA94DD:114FBFC798B4280]:0)
12:20:09    > 	at org.elasticsearch.index.mapper.LegacyDynamicMappingIT.testMappingsPropagatedToMasterNodeImmediatelyMultiType(LegacyDynamicMappingIT.java:56)
12:20:09    > 	at java.lang.Thread.run(Thread.java:748)

I tried to reproduce this locally using:

./gradlew :server:integTest \
  -Dtests.seed=ED2233D5C1DA94DD \
  -Dtests.class=org.elasticsearch.index.mapper.LegacyDynamicMappingIT \
  -Dtests.method="testMappingsPropagatedToMasterNodeImmediatelyMultiType" \
  -Dtests.security.manager=true \
  -Dtests.locale=en \
  -Dtests.timezone=Europe/Istanbul \
  -Dcompiler.java=11 \
  -Druntime.java=8

This time the test failed, but with a test assertion rather than an NPE:

FAILURE 6.68s | LegacyDynamicMappingIT.testMappingsPropagatedToMasterNodeImmediatelyMultiType <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Could not find [bar] in {_all={enabled=true}, properties={foo={type=long}}}
   >    at __randomizedtesting.SeedInfo.seed([ED2233D5C1DA94DD:114FBFC798B4280]:0)
   >    at org.elasticsearch.index.mapper.DynamicMappingIT.assertMappingsHaveField(DynamicMappingIT.java:66)
   >    at org.elasticsearch.index.mapper.LegacyDynamicMappingIT.testMappingsPropagatedToMasterNodeImmediatelyMultiType(LegacyDynamicMappingIT.java:51)
   >    at java.lang.Thread.run(Thread.java:745)

[elasticmachine]

Pinging @elastic/es-search

[droberts195]

I muted this test for 6.x in bb5747c

[droberts195]

The same test failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1148/console

The exact failure is different to the two in the original issue description:

12:10:17 FAILURE 0.44s J2 | LegacyDynamicMappingIT.testMappingsPropagatedToMasterNodeImmediatelyMultiType <<< FAILURES!
12:10:17    > Throwable #1: java.lang.AssertionError
12:10:17    > 	at __randomizedtesting.SeedInfo.seed([C3ED4C523F85E2BF:2FDB847B87D434E2]:0)
12:10:17    > 	at org.elasticsearch.index.mapper.DynamicMappingIT.assertMappingsHaveField(DynamicMappingIT.java:61)
12:10:17    > 	at org.elasticsearch.index.mapper.LegacyDynamicMappingIT.testMappingsPropagatedToMasterNodeImmediatelyMultiType(LegacyDynamicMappingIT.java:46)
12:10:17    > 	at java.lang.Thread.run(Thread.java:748)

However, the sort of thing it's asserting is similar, so I've added it to this issue.

The repro line from this build was:

./gradlew :server:integTest \
  -Dtests.seed=C3ED4C523F85E2BF \
  -Dtests.class=org.elasticsearch.index.mapper.LegacyDynamicMappingIT \
  -Dtests.method="testMappingsPropagatedToMasterNodeImmediatelyMultiType" \
  -Dtests.security.manager=true \
  -Dtests.locale=id-ID \
  -Dtests.timezone=Asia/Saigon \
  -Dcompiler.java=11 \
  -Druntime.java=8

[cbuescher]

Also @droberts195 mentioned he saw some general problems with cluster state related IT test today, namely #37820.
Not saying those are related but maybe worth keeping in mind since this seems to be a race condition, thus more difficult to reproduce.

[dnhatn]

I pushed f925690 to busily check the mapping update. This requires since #31140 where an index request no long requires acking on its dynamic mapping update. /cc @ywelsch.