BigQueryDataTransferServiceStartTransferRunsOperator got Access Denied error on long-running migration jobs.

[okayhooni]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

latest

Apache Airflow version

2.3.x ~ latest

Operating System

Debian GNU/Linux 11 (bullseye)

Deployment

Official Apache Airflow Helm Chart

Deployment details

deployed on EKS cluster with customized Airflow Helm chart based on the official chart

What happened

BigQueryDataTransferServiceStartTransferRunsOperator got Access Denied error on long-running migration jobs.
Especially, when BigQuery Data Transfer job triggered by Airflow operator, exceeds one hour, it fails due to the expiration of the credential (default lifespan = 1 hour).
But, I sometimes experienced the same Access Denied issue even on the job that has been submitted for less than 10 minutes.

Access Denied: Table projectid:table_name: Permission bigquery.tables.get denied on table projectid:datasetid.table_name (or it may not exist).

(those DTS job error logs can also be seen on BigQuery Data transfer console)

At first, I thought it was a token expiration issue, so I attempted to refresh the token; however, it did not have any effect on resolving the issue.

• Implement token refresh on BigQuery dts run operator to handle the heavy migration jobs over 1 hour #37538

What you think should happen instead

• BigQueryDataTransferServiceStartTransferRunsOperator task has to be success regardless of the running duration of migration jobs.
• As I mentioned above, I sometimes experienced the same Access Denied issue even on the job that has been submitted for less than 10 minutes. (So, I guess it is the issue on the GCP itself)

How to reproduce

• Runnining the BigQuery DTS job for migrating a sufficiently large data source that takes more than an hour.
• It can be sometimes reproduced on shorter DTS job (< 10 minutes)

Anything else

Related issue:

• Getting Unauthorized error messages with GKEStartPodOperator if pod execution is over 1 hour #31648

Related MR:

• Support providing creds lifetime for GCP connection token #31651
• Refresh GKE OAuth2 tokens #32673

Related Google BigQuery Docs:

• https://cloud.google.com/bigquery/docs/transfer-troubleshooting

Error: Access Denied: ... Permission bigquery.tables.get denied on table ...
Resolution: Confirm that the BigQuery Data Transfer Service service agent is granted the bigquery.dataEditor role on the target dataset. This grant is automatically applied when creating and updating the transfer, but it's possible that the access policy was modified manually afterwards. To regrant the permission, see Grant access to a dataset.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.