Status of testing Providers that were prepared on April 09, 2023

[eladkal]

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 7.4.1rc1

• Fix RedshiftResumeClusterOperator deferrable implementation (#30370): @phanikumv
• add more info to quicksight errors (#30466): @vandonr-amz
• Add s3_bucket to template fields in S3 to SFTP operator (#30472): @happilyeverafter95
• Add s3_bucket to template fields in SFTP to S3 operator (#30444): @potiuk

Provider cncf.kubernetes: 6.0.0rc1

• Use default connection id for KubernetesPodOperator (#28848): @lwyszomi
• Set limits for XCOM pod (#28125): @dolfinus
• Add mechanism to suspend providers (#30422): @potiuk

Provider google: 9.0.0rc1

• Update DV360 operators to use API v2 (#30326): @lwyszomi
• Fix dynamic imports in google ads vendored in library (#30544): @potiuk
• Add deferrable mode to GKEStartPodOperator (#29266): @VladaZakharova
• BigQueryHook list_rows/get_datasets_list can return iterator (#30543): @vchiapaikeo
• Fix cloud build async credentials (#30441): @tnk-ysk

Provider microsoft.azure: 5.3.1rc1

• Fix AzureDataFactoryPipelineRunLink UI link generation (#30514): @hussein-awala

Provider smtp: 1.0.1rc1

• EmailOperator: fix wrong assignment of from_email (#30524): @vanducng
• Accept None for EmailOperator.from_email to load it from smtp connection (#30533): @hussein-awala

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@happilyeverafter95 @dolfinus @VladaZakharova @phanikumv @potiuk @tnk-ysk @hussein-awala @vandonr-amz @vanducng @vchiapaikeo @lwyszomi

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[vanducng]

Confirm #30524 bug already got fixed in smtp: 1.0.1rc1. Thanks.

[vandonr-amz]

👍

[vanducng]

google: 9.0.0rc1 also works well with our google ads and dv360 task.

[vchiapaikeo]

#30543 also still looks fine. Thank you, Elad!

[potiuk]

google: 9.0.0rc1 also works well with our google ads and dv360 task.

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

[potiuk]

Tested mine. All works.

[raphaelauv]

Use default connection id for KubernetesPodOperator (#28848)

work

[hussein-awala]

• Accept None for EmailOperator.from_email to load it from smtp connection #30533 -> works as expected
• Fix AzureDataFactoryPipelineRunLink UI link generation #30514 -> it works with the connections created via the Airflow CLI (as I did when I tested the PR before), but not directly with the connections created from the UI, I created Fix Azure data factory UI link by load subscription_id from extra__azure__subscriptionId #30556 to describe and fix the problem, and Azure connections extra fields are inconsistent between the doc, tests and the UI connections #30557 to clean the Azure provider code. Since we can create the connection from the CLI or the environment variables, and even we can edit the UI connection manually once it is created, no need to block the release and create a new one, we can add the fix (and maybe the full clean) to the next release.

[tnk-ysk]

I tested #30441, it works as expected

[eladkal]

#30514 -> it works with the connections created via the Airflow CLI (as I did when I tested the PR before), but not directly with the connections created from the UI, I created #30556 to describe and fix the problem, and #30557 to clean the Azure provider code. Since we can create the connection from the CLI or the environment variables, and even we can edit the UI connection manually once it is created, no need to block the release and create a new one, we can add the fix (and maybe the full clean) to the next release.

since this is the only fix in the release for azure I tend to drop azure from this wave. I think it's preferred to cut RC2 after the fixes are ready

[potiuk]

since this is the only fix in the release for azure I tend to drop azure from this wave. I think it's preferred to cut RC2 after the fixes are ready

We could also apply what we agreed about - shorter voting period for RC2+

• https://github.com/apache/airflow/pull/30037/files

We could potentially merge the fix now, prepare RC2 for azure only and release them all together when voting period ends. Might be good time to try this method to see if there is anything to fix in our process to follow this route.

[eladkal]

We could also apply what we agreed about - shorter voting period for RC2+

This is what I plan to do but this will work only if there aren't other changes for azure that were merged in the mean time. I didn't check yet.

[potiuk]

We will need an RC2 for google provider too - (found by @cgadam - fix in #30564).

[utkarsharma2]

Works well with AstroSDk.

[phanikumv]

#30370 works well.

2023-04-10, 14:00:26 UTC] {standard_task_runner.py:55} INFO - Started process 3123 to run task
[2023-04-10, 14:00:26 UTC] {standard_task_runner.py:82} INFO - Running: ['***', 'tasks', 'run', 'example_async_redshift_cluster_management', 'resume_redshift_cluster', 'manual__2023-04-10T13:55:51.894401+00:00', '--job-id', '10', '--raw', '--subdir', 'DAGS_FOLDER/example_redshift_cluster_management.py', '--cfg-path', '/tmp/tmpsyc34091']
[2023-04-10, 14:00:26 UTC] {standard_task_runner.py:83} INFO - Job 10: Subtask resume_redshift_cluster
[2023-04-10, 14:00:26 UTC] {task_command.py:389} INFO - Running <TaskInstance: example_async_redshift_cluster_management.resume_redshift_cluster manual__2023-04-10T13:55:51.894401+00:00 [running]> on host 101451e0c77b
[2023-04-10, 14:00:26 UTC] {redshift_cluster.py:472} INFO - resume_redshift_cluster completed successfully.
[2023-04-10, 14:00:26 UTC] {redshift_cluster.py:473} INFO - Resumed cluster successfully

[eladkal]

microsoft.azure and google will be excluded from rc1.
I will cut rc2 as soon as other providers are released.

[eladkal]

Thank you everyone.
The google and microsoft.azure providers are excluded from the release and RC2 will be cut for them shortly.
other providers are released.
I invite everyone to help improve providers for the next release, a list of open issues can be found here.