Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Stop datafeeds running when their jobs are stale #37227

Merged
merged 3 commits into from
Jan 9, 2019
Merged

[ML] Stop datafeeds running when their jobs are stale #37227

merged 3 commits into from
Jan 9, 2019

Conversation

droberts195
Copy link
Contributor

We already had logic to stop datafeeds running against
jobs that were OPENING, but a job that relocates from
one node to another while OPENED stays OPENED, and this
could cause the datafeed to fail when it sent data to
the OPENED job on its new node before it had a
corresponding autodetect process.

This change extends the check to stop datafeeds running
when their job is OPENING or stale (i.e. has not had
its status reset since relocating to a different node).

Relates #36810

@droberts195
[ML] Stop datafeeds running when their jobs are stale
52f37d0 
We already had logic to stop datafeeds running against
jobs that were OPENING, but a job that relocates from
one node to another while OPENED stays OPENED, and this
could cause the datafeed to fail when it sent data to
the OPENED job on its new node before it had a
corresponding autodetect process.

This change extends the check to stop datafeeds running
when their job is OPENING _or_ stale (i.e. has not had
its status reset since relocating to a different node).

Relates elastic#36810
@droberts195 droberts195 requested a review from davidkyle January 8, 2019 14:58
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

dimitris-athanasiou
dimitris-athanasiou approved these changes Jan 8, 2019
View reviewed changes
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@droberts195 droberts195 mentioned this pull request Jan 8, 2019
Closed
davidkyle
davidkyle approved these changes Jan 8, 2019
View reviewed changes
Copy link
Member

@davidkyle davidkyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@droberts195
Copy link
Contributor Author

run gradle build tests 1

1 similar comment
@droberts195
Copy link
Contributor Author

run gradle build tests 1

@droberts195 droberts195 merged commit e0ce737 into elastic:master Jan 9, 2019
@droberts195 droberts195 deleted the stop_datafeed_running_against_stale_job branch January 9, 2019 10:42
droberts195 added a commit that referenced this pull request Jan 9, 2019
@droberts195
[ML] Stop datafeeds running when their jobs are stale (#37227)
2c4c345 
We already had logic to stop datafeeds running against
jobs that were OPENING, but a job that relocates from
one node to another while OPENED stays OPENED, and this
could cause the datafeed to fail when it sent data to
the OPENED job on its new node before it had a
corresponding autodetect process.

This change extends the check to stop datafeeds running
when their job is OPENING _or_ stale (i.e. has not had
its status reset since relocating to a different node).

Relates #36810
droberts195 added a commit that referenced this pull request Jan 9, 2019
@droberts195
[ML] Stop datafeeds running when their jobs are stale (#37227)
a582660 
We already had logic to stop datafeeds running against
jobs that were OPENING, but a job that relocates from
one node to another while OPENED stays OPENED, and this
could cause the datafeed to fail when it sent data to
the OPENED job on its new node before it had a
corresponding autodetect process.

This change extends the check to stop datafeeds running
when their job is OPENING _or_ stale (i.e. has not had
its status reset since relocating to a different node).

Relates #36810
@droberts195 droberts195 mentioned this pull request Jan 9, 2019
Merged
droberts195 added a commit that referenced this pull request Jan 9, 2019
@droberts195
[TEST] Unmute 60_ml_config_migration rolling upgrade (#37258)
d817b46 
This reverts commit d7efadc

The test should now work following the change made in #37227

Closes #36810
droberts195 added a commit that referenced this pull request Jan 9, 2019
@droberts195
[TEST] Unmute 60_ml_config_migration rolling upgrade (#37258)
335eeeb 
This reverts commit d7efadc

The test should now work following the change made in #37227

Closes #36810
droberts195 added a commit to droberts195/elasticsearch that referenced this pull request Jan 11, 2019
@droberts195
[ML] Wait for autodetect to be ready in the datafeed
de295f7 
This is a reinforcement of elastic#37227.  It turns out that
persistent tasks are not made stale if the node they
were running on is restarted and the master node does
not notice this.  The main scenario where this happens
is when minimum master nodes is the same as the number
of nodes in the cluster, so the cluster cannot elect a
master node when any node is restarted.

When an ML node restarts we need the datafeeds for any
jobs that were running on that node to not just wait
until the jobs are allocated, but to wait for the
autodetect process of the job to start up.  In the case
of reassignment of the job persistent task this was
dealt with by the stale status test.  But in the case
where a node restarts but its persistent tasks are not
reassigned we need a deeper test.

Fixes elastic#36810
@droberts195 droberts195 mentioned this pull request Jan 11, 2019
Merged
droberts195 added a commit that referenced this pull request Jan 11, 2019
@droberts195
[ML] Wait for autodetect to be ready in the datafeed (#37349)
1da59db 
This is a reinforcement of #37227.  It turns out that
persistent tasks are not made stale if the node they
were running on is restarted and the master node does
not notice this.  The main scenario where this happens
is when minimum master nodes is the same as the number
of nodes in the cluster, so the cluster cannot elect a
master node when any node is restarted.

When an ML node restarts we need the datafeeds for any
jobs that were running on that node to not just wait
until the jobs are allocated, but to wait for the
autodetect process of the job to start up.  In the case
of reassignment of the job persistent task this was
dealt with by the stale status test.  But in the case
where a node restarts but its persistent tasks are not
reassigned we need a deeper test.

Fixes #36810
droberts195 added a commit that referenced this pull request Jan 11, 2019
@droberts195
[ML] Wait for autodetect to be ready in the datafeed (#37349)
86b01f5 
This is a reinforcement of #37227.  It turns out that
persistent tasks are not made stale if the node they
were running on is restarted and the master node does
not notice this.  The main scenario where this happens
is when minimum master nodes is the same as the number
of nodes in the cluster, so the cluster cannot elect a
master node when any node is restarted.

When an ML node restarts we need the datafeeds for any
jobs that were running on that node to not just wait
until the jobs are allocated, but to wait for the
autodetect process of the job to start up.  In the case
of reassignment of the job persistent task this was
dealt with by the stale status test.  But in the case
where a node restarts but its persistent tasks are not
reassigned we need a deeper test.

Fixes #36810
droberts195 added a commit that referenced this pull request Jan 11, 2019
@droberts195
[ML] Wait for autodetect to be ready in the datafeed (#37349)
f141f3a 
This is a reinforcement of #37227.  It turns out that
persistent tasks are not made stale if the node they
were running on is restarted and the master node does
not notice this.  The main scenario where this happens
is when minimum master nodes is the same as the number
of nodes in the cluster, so the cluster cannot elect a
master node when any node is restarted.

When an ML node restarts we need the datafeeds for any
jobs that were running on that node to not just wait
until the jobs are allocated, but to wait for the
autodetect process of the job to start up.  In the case
of reassignment of the job persistent task this was
dealt with by the stale status test.  But in the case
where a node restarts but its persistent tasks are not
reassigned we need a deeper test.

Fixes #36810
2lambda123 pushed a commit to 2lambda123/elastic-elasticsearch that referenced this pull request May 2, 2024
@droberts195
[TEST] Unmute 60_ml_config_migration rolling upgrade
1e6565f 
This reverts commit d7efadc.

The test should now work following the change made in
elastic/elasticsearch#37227
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dimitris-athanasiou dimitris-athanasiou dimitris-athanasiou approved these changes

@davidkyle davidkyle davidkyle approved these changes

Assignees
No one assigned
Labels
>bug :ml Machine learning v6.6.0 v6.7.0 v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@droberts195 @elasticmachine @davidkyle @dimitris-athanasiou @colings86