Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provider Airbyte: breaking change, update provider to use Airbyte API Python SDK #41122

Merged
merged 28 commits into from
Aug 28, 2024

Conversation

marcosmarxm
Copy link
Contributor

@marcosmarxm marcosmarxm commented Jul 30, 2024

Hello 👋, Airbyte has finally consolidated API access across the open-source, cloud, and enterprise versions.
This means all of them have a single method client_id and client_secret to authenticate to the API now.
This pull request remove the dependency of HttpHook and uses the airbyte-api Python SDK library to execute all operations (create job, cancel job, etc).
The change is a breaking change to users as it removed the api_type parameter as it isn't required anymore.
The new authentication method will become soon the unique method to auth to the API so this is a required changed.
Unit tests were updated to run using the new code.


airflow/providers/airbyte/provider.yaml Outdated Show resolved Hide resolved
airflow/providers/airbyte/provider.yaml Show resolved Hide resolved
airflow/providers/airbyte/hooks/airbyte.py Show resolved Hide resolved
@marcosmarxm marcosmarxm requested a review from eladkal July 31, 2024 17:10
@marcosmarxm
Copy link
Contributor Author

@eladkal, can you take another look? Some tests are failing, but I can't determine if it’s due to my changes.

@potiuk
Copy link
Member

potiuk commented Aug 5, 2024

If in doubt - rebase:

Screenshot 2024-08-05 at 15 20 56

@marcosmarxm marcosmarxm force-pushed the marcos/airbyte-update-api-auth-method branch from 9338ffd to 06961bb Compare August 5, 2024 13:52
@eladkal eladkal linked an issue Aug 10, 2024 that may be closed by this pull request
2 tasks
@marcosmarxm
Copy link
Contributor Author

Before merging this change I'll run a final test with Airbye cloud and open source.

@eladkal eladkal force-pushed the marcos/airbyte-update-api-auth-method branch from 9dd3a18 to 6c40cb1 Compare August 10, 2024 19:15
@eladkal
Copy link
Contributor

eladkal commented Aug 10, 2024

Before merging this change I'll run a final test with Airbye cloud and open source.

Please do. If no further comments we are good to go

@potiuk
Copy link
Member

potiuk commented Aug 16, 2024

You need to rebase/run pre-commit

@potiuk potiuk requested a review from eladkal August 19, 2024 17:49
@marcosmarxm
Copy link
Contributor Author

Running final test with local deployment and cloud. Gonna 💚 🚀 after finish these tests.

@marcosmarxm
Copy link
Contributor Author

🟢 Tests are working
🟢 I executed DAGs with a local deployment and cloud, both are working.

Need rebase and 🚀

@marcosmarxm marcosmarxm force-pushed the marcos/airbyte-update-api-auth-method branch 2 times, most recently from 5bf3e2f to ae2825a Compare August 21, 2024 14:39
@eladkal eladkal force-pushed the marcos/airbyte-update-api-auth-method branch from c094c21 to b3962a3 Compare August 28, 2024 06:51
@marcosmarxm
Copy link
Contributor Author

Wooohooo 🎉 !!!

@eladkal eladkal merged commit c2d5ef4 into apache:main Aug 28, 2024
113 checks passed
@baugarcia
Copy link

Great! any ETA on when will the new version be deployed in the python repository? Version is still 3.9.0 there:
https://pypi.org/project/apache-airflow-providers-airbyte/

@marcosmarxm
Copy link
Contributor Author

@baugarcia probably the next window of releases is in 3~4 weeks. But keep this to the maintainers to provide a better answer.

@marcosmarxm marcosmarxm deleted the marcos/airbyte-update-api-auth-method branch September 26, 2024 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Providers] Airbyte new deployment version changed API endpoints
5 participants