-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator #39446
Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for DataprocCreateClusterOperator #39446
Conversation
Just have a quick discussion with @sunank200. We'll yield the event if CancelError was raised , and handle the TaskInstance state check and cancelation in |
47f4408
to
f30e854
Compare
That won't work as the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably should add a test to this. But others look good to me
f30e854
to
64975f2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add the test case for this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a short nitpick. We could create an issue and work on that later, but we'll need test cases for this PR
1dd7d02
to
2f69779
Compare
Added the test. |
87bda99
to
b32e305
Compare
…running or deferred state for DataprocCreateClusterOperator (apache#39446)
PR #39130 introduces a method for handling asyncio.CancelledError in a try/except block. However, this method is deemed unsafe, and it affects
DataprocCreateClusterOperator
operators, which enables external job cancellation if the triggerer restarts or crashes. This can cause weird behaviour like rescheduling deferred operators, as Airflow remains unaware of job cancellations.As a workaround, capturing
asyncio.CancelledError
cancels the job only if the TaskInstance is not in a running or deferred state. This prevents premature external job termination.More details at: #36090 (comment)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.