You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
if you stop a running GlueJobOperator task ( mark fail ) the operator do not call aws glue to stop the runnning task
so the airflow task stop but the aws glue job continue to run
Use case/motivation
Like the KPO would be nice to stop the aws glue job in case the operator is stop
currently the logs of operator are
[2023-06-26, 10:07:32 UTC] {glue.py:149} INFO - Initializing AWS Glue Job: XXXXXXXX. Wait for completion: True
[2023-06-26, 10:07:32 UTC] {glue.py:290} INFO - Checking if job already exists: XXXXXXXXXXX
[2023-06-26, 10:07:32 UTC] {base.py:73} INFO - Using connection ID 'XXXXXXXX' for task execution.
[2023-06-26, 10:07:32 UTC] {credentials.py:1051} INFO - Found credentials from IAM Role: XXXXXXXXXXXX
[2023-06-26, 10:07:33 UTC] {glue.py:169} INFO - You can monitor this Glue Job run at: https://console.aws.amazon.com/gluestudio/home?region=eu-west-1#/job/test_ip/run/XXXXXXXXXXXXXXX
[2023-06-26, 10:07:33 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:39 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:45 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:51 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:07:57 UTC] {glue.py:265} INFO - Polling for AWS Glue Job test_ip current run state with status RUNNING
[2023-06-26, 10:08:02 UTC] {local_task_job.py:276} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2023-06-26, 10:08:02 UTC] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 13175. PIDs of all processes in the group: [13175]
[2023-06-26, 10:08:02 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 13175
[2023-06-26, 10:08:02 UTC] {taskinstance.py:1488} ERROR - Received SIGTERM. Terminating subprocesses.
[2023-06-26, 10:08:02 UTC] {process_utils.py:79} INFO - Process psutil.Process(pid=13175, status='terminated', exitcode=0, started='10:07:31') (13175) terminated with exit code 0
Description
if you stop a running GlueJobOperator task ( mark fail ) the operator do not call aws glue to stop the runnning task
so the airflow task stop but the aws glue job continue to run
Use case/motivation
Like the KPO would be nice to stop the aws glue job in case the operator is stop
currently the logs of operator are
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: