Amazon appflow

[igorborgest]

Hi!

It's a proposal to add Amazon AppFlow support for the apache-airflow-providers-amazon package.

This is an initial support tested only against two sources: Salesforce and Zendesk.

Adding 1 new hook and 6 new operators + all related tests, docs and examples:

• AppflowHook
• AppflowRunAfterOperator
• AppflowRunOperator
• AppflowRunFullOperator
• AppflowRunDailyOperator
• AppflowRunBeforeOperator
• AppflowRunAfterOperator
• AppflowRecordsShortCircuit

[ferruzzi]

Thanks for tackling this, it's a lot of work!

I have not used AppFlow, I assume you've tested these against the actual service and they work as you'd expect? I left some comments based on code conventions we use in the community and I'll have to finish looking at the Operators later. So far just a few smallish stylistic requests/suggestions.

[igorborgest]

Thanks for tackling this, it's a lot of work!

I have not used AppFlow, I assume you've tested these against the actual service and they work as you'd expect? I left some comments based on code conventions we use in the community and I'll have to finish looking at the Operators later. So far just a few smallish stylistic requests/suggestions.

@ferruzzi Yep! I developed this operator six months ago during an AWS Data Lab engagement alongside some AWS SAs. Since then, it has been running in production for Salesforce and Zendesk at VTEX.

[ferruzzi]

That's cool, thanks for contributing it back to the community!

Thanks for the changes, I know a lot of them were little nitpicks. I'll need to finish looking at the other half of the operators tomorrow. 👍

[o-nikolas]

Really nice contribution! Other than the minor notes from Dennis and Vincent this is a very solid PR with testing, docs, example dags, etc, all up to current standards 💯 Thanks for your work!

[igorborgest]

Thank you @o-nikolas @ferruzzi @vincbeck for all the great suggestions.

I think I've already addressed all of them. Are we good to move on? 😄

[vincbeck]

Awesome work! LGTM!

[ferruzzi]

I haven't had time to look at the second half of the operators, but most of my comments were stylistic, so as long as you carried them forward I think we're fine. We'll still need a Committer to approve and merge it.

[potiuk]

Can we please change example dags/system tests in AIP-47 compliant way? You can see many of PRs doing it recently and I think we are at the verge of makign it "default" way of adding the examples.

[josh-fell]

Lots of great work here! Overall, some minor comments and suggestions.

[josh-fell]

Looks great! 👍 Small comments re: using the functools compat module now and a question on the spelling wordlist change (if it's needed, I don't think it is).

Can you make sure to rebase when you push these changes too? We can merge after.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[igorborgest]

Thank you @josh-fell!
I think we are ready to merge 🙂

[josh-fell]

Nice work @igorborgest!

@vincbeck @o-nikolas @ferruzzi @Taragolis Thanks for all the feedback along the way!