[CI] ClassificationIT.testTooLowConfiguredMemoryStillStarts & RunDataFrameAnalyticsIT.testModelMemoryLimitLowerThanEstimatedMemoryUsage

[ywangd]

Build scan:
https://gradle-enterprise.elastic.co/s/lqjgfg43cew4q
Note it has two failed tests

Repro line:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTest' --tests "org.elasticsearch.xpack.ml.integration.ClassificationIT.testTooLowConfiguredMemoryStillStarts" -Dtests.seed=B6982EE4D20C808E -Dtests.security.manager=true -Dtests.locale=es-CL -Dtests.timezone=Australia/West -Druntime.java=11

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTest' --tests "org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.testModelMemoryLimitLowerThanEstimatedMemoryUsage" -Dtests.seed=B6982EE4D20C808E -Dtests.security.manager=true -Dtests.locale=sr-RS -Dtests.timezone=America/Sitka -Druntime.java=11

Reproduces locally?:
No

Applicable branches:
master

Failure history:

The method testModelMemoryLimitLowerThanEstimatedMemoryUsage failed for the very first faliure in last 30 days according to build-stats. I was gonna let it pass, but then I noticed that testTooLowConfiguredMemoryStillStarts is a new method added in today's changes. Both methods are touched in the change set. So there might be something going on.

Failure excerpt:

java.lang.AssertionError: |  
-- | --
  | Expected: <0L> |  
  | but: was <1L> |  


at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
-- | --
  |   | •••
  |   | at org.elasticsearch.xpack.ml.integration.MlNativeDataFrameAnalyticsIntegTestCase.cleanUpAnalytics(MlNativeDataFrameAnalyticsIntegTestCase.java:96) |  
  |   | at org.elasticsearch.xpack.ml.integration.MlNativeDataFrameAnalyticsIntegTestCase.cleanUpResources(MlNativeDataFrameAnalyticsIntegTestCase.java:87) |  
  |   | at org.elasticsearch.xpack.ml.integration.MlNativeIntegTestCase.cleanUp(MlNativeIntegTestCase.java:181) |  
  |   | at org.elasticsearch.xpack.ml.integration.RunDataFrameAnalyticsIT.cleanup(RunDataFrameAnalyticsIT.java:62)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[nik9000]

@benwtrent this looks to have failed again, this time in 7.x: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic-fips/ES_RUNTIME_JAVA=openjdk15,nodes=general-purpose/196/

[benwtrent]

Yeah, more discussion here #61704

I think we have found the cause. Hopefully it will get fixed soon. It can be muted for the time being.