Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upgraded_cluster/30_ml_jobs_crud fails with "No known job with id 'old-cluster-job'" #43342

Closed
DaveCTurner opened this issue Jun 18, 2019 · 6 comments · Fixed by #43358
Closed
Labels
:ml Machine learning >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

The following build failed: https://scans.gradle.com/s/crky52pgtdybm/console-log?task=:x-pack:qa:rolling-upgrade:v7.3.0%23upgradedClusterTestRunner


1> [2019-06-18T19:12:12,539][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] [p0=upgraded_cluster/30_ml_jobs_crud/Test get old cluster job's timing stats] before test
--
1> [2019-06-18T19:12:12,612][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
1>   "stash" : {
1>     "body" : {
1>       "error" : {
1>         "root_cause" : [
1>           {
1>             "type" : "resource_not_found_exception",
1>             "reason" : "No known job with id 'old-cluster-job'",
1>             "stack_trace" : "ResourceNotFoundException[No known job with id 'old-cluster-job']
1> 	at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)
1> 	at org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)
1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
1> 	at java.base/java.lang.Thread.run(Thread.java:834)
1> "
1>           }
1>         ],
1>         "type" : "resource_not_found_exception",
1>         "reason" : "No known job with id 'old-cluster-job'",
1>         "stack_trace" : "ResourceNotFoundException[No known job with id 'old-cluster-job']
1> 	at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)
1> 	at org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)
1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)
1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
1> 	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)
1> 	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)
1> 	at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)
1> 	at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)
1> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
1> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
1> 	at java.base/java.lang.Thread.run(Thread.java:834)
1> "
1>       },
1>       "status" : 404
1>     }
1>   }
1> }]
1> [2019-06-18T19:12:12,649][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] There are still tasks running after this test that might break subsequent tests [xpack/ml/job[c]].
1> [2019-06-18T19:12:12,650][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] [p0=upgraded_cluster/30_ml_jobs_crud/Test get old cluster job's timing stats] after test
2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:v7.3.0#upgradedClusterTestRunner --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=upgraded_cluster/30_ml_jobs_crud/Test get old cluster job's timing stats}" -Dtests.seed=8DF4A47F62374ACB -Dtests.security.manager=true -Dtests.locale=kea-CV -Dtests.timezone=Asia/Nicosia -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.suite=upgraded_cluster
2> java.lang.AssertionError: Failure at [upgraded_cluster/30_ml_jobs_crud:114]: expected [2xx] status code but api [ml.get_job_stats] returned [404 Not Found] [{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"}],"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"status":404}]
 
Caused by:
java.lang.AssertionError: expected [2xx] status code but api [ml.get_job_stats] returned [404 Not Found] [{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"}],"type":"resource_not_found_exception","reason":"No known job with id 'old-cluster-job'","stack_trace":"ResourceNotFoundException[No known job with id 'old-cluster-job']\n\tat org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.missingJobException(ExceptionsHelper.java:22)\n\tat org.elasticsearch.xpack.ml.job.persistence.JobConfigProvider.lambda$expandJobsIds$5(JobConfigProvider.java:535)\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:321)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:51)\n\tat org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:213)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:120)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:171)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:165)\n\tat org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:206)\n\tat org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104)\n\tat org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:110)\n\tat org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:86)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:757)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"status":404}]

I tried the given REPRODUCE WITH line locally but it did not reproduce this failure.

@DaveCTurner DaveCTurner added >test-failure Triaged test failures from CI :ml Machine learning labels Jun 18, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@DaveCTurner
Copy link
Contributor Author

Muted in 631c63c.

@przemekwitek
Copy link
Contributor

AFAICT this is a duplicate of #43314 which I fixed yesterday with #43315.

I will unmute the test now

@DaveCTurner
Copy link
Contributor Author

Thanks @przemekwitek, yes, that does look to be the same issue. I'll close this.

@DaveCTurner
Copy link
Contributor Author

Oh wait, they're still muted. I'll let you close this once they're unmuted.

@DaveCTurner DaveCTurner reopened this Jun 19, 2019
@przemekwitek
Copy link
Contributor

They are muted as I'm waiting for the CI in #43358 to pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ml Machine learning >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants