-
Notifications
You must be signed in to change notification settings - Fork 14.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mapped tasks with operator_extra_links
property object causes SerializationError
#25243
Comments
Instead of going through trouble trying to accomodate dynamic extra links, perhaps we should just detect those and mark them as dynamically and can’t be shown. |
I am seeing similar issues with the ECSOperator. |
Should be fixed by #25500 |
Reopening this issue. Trying to map |
|
BatchOperator
tasks causes SerializationError due to operator_extra_links
property objectoperator_extra_links
property object causes SerializationError
I'm facing the same issue |
Does anyone by chance know a work around for the issue (specifically w/ AWS Batch operator mentioned in original issue)? My current "work around" is just use the Amazon provider <=v3.4.0 which isn't ideal but does allow the Batch operator to be used in dynamic task mapping... |
|
Apache Airflow Provider(s)
amazon
Versions of Apache Airflow Providers
apache-airflow-providers-amazon-aws==4.1.0
Apache Airflow version
main (development)
Operating System
Debian GNU/Linux 11 (bullseye)
Deployment
Other Docker-based deployment
Deployment details
Using Breeze on
main
branch.What happened
Attempting to create dynamically-mapped tasks using the
BatchOperator
fails with the following DAG import error:What you think should happen instead
Users should be able to use Dynamic Task Mapping to generate
BatchOperator
tasks without a DAG import/serialization error.How to reproduce
BatchOperator
tasks are dynamically-mapped. Note this is a "toy" example, but it should be applicable to more "real-world" use cases.breeze start-airflow
Anything else
A similar issue was created previously with related fixes in #24676 and #25215.
I suspect the same behavior would occur using the
BigQueryExecuteQueryOperator
as well.Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: