MlMigrationIT#testConfigMigration fails in 6.x

[DaveCTurner]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+intake/24/console fails as follows:

  2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v5.6.15#twoThirdsUpgradedTestRunner -Dtests.seed=679945B82D7ABBD4 -Dtests.class=org.elasticsearch.upgrades.MlMigrationIT -Dtests.method="testConfigMigration" -Dtests.security.manager=true -Dtests.locale=it -Dtests.timezone=Pacific/Ponape -Dcompiler.java=11 -Druntime.java=8
  1> [2018-12-21T13:28:59,644][INFO ][o.e.u.MlMigrationIT      ] [testConfigMigration] before test
  1> [2018-12-21T13:28:59,646][INFO ][o.e.u.MlMigrationIT      ] [testConfigMigration] initializing REST clients against [http://[::1]:43118]
  1> [2018-12-21T13:29:43,197][INFO ][o.e.u.MlMigrationIT      ] [testConfigMigration] There are still tasks running after this test that might break subsequent tests [xpack/ml/datafeed[c], xpack/ml/job[c]].
  1> [2018-12-21T13:29:43,198][INFO ][o.e.u.MlMigrationIT      ] [testConfigMigration] after test
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.6+intake/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v5.6.15#twoThirdsUpgradedTestRunner/J0/temp/org.elasticsearch.upgrades.MlMigrationIT_679945B82D7ABBD4-001
  2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=it, timezone=Pacific/Ponape
FAILURE 43.6s | MlMigrationIT.testConfigMigration <<< FAILURES!
  2> NOTE: Linux 3.10.0-957.1.3.el7.x86_64 amd64/Oracle Corporation 1.8.0_192 (64-bit)/cpus=16,threads=1,free=427703536,total=514850816
  2> NOTE: All tests run in this JVM: [WatchBackwardsCompatibilityIT, MlMigrationIT]
   > Throwable #1: org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
   > 	at __randomizedtesting.SeedInfo.seed([679945B82D7ABBD4:9157453DEE0A24D3]:0)
   > 	at org.elasticsearch.upgrades.MlMigrationIT.lambda$waitForDatafeedToBeAssigned$10(MlMigrationIT.java:432)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
   > 	at org.elasticsearch.upgrades.MlMigrationIT.waitForDatafeedToBeAssigned(MlMigrationIT.java:424)
   > 	at org.elasticsearch.upgrades.MlMigrationIT.mixedClusterTests(MlMigrationIT.java:197)
   > 	at org.elasticsearch.upgrades.MlMigrationIT.testConfigMigration(MlMigrationIT.java:113)
   > 	at java.lang.Thread.run(Thread.java:748)

The REPRODUCE WITH line sometimes fails for this reason, but more frequently fails because of a node being erroneously killed, causing the cluster to fall to pieces. We're investigating this separately. Sometimes also fails on the first upgraded node (i.e. replacing twoThirdsUpgradedTestRunner with oneThirdUpgradedTestCluster) avoiding the cluster formation issue.

[elasticmachine]

Pinging @elastic/ml-core

[dliappis]

To reduce CI noise I annotated the method as AwaitsFix: 9c5790a

[droberts195]

I think the aspect of this problem that relates to ML is the same as #36810

[davidkyle]

Closing this issue as it is a duplicate of #36810 and closed by #37349. The tests have been unmuted on 6.6 and 6.x