Airbyte ol integration #40689
Airbyte ol integration #40689
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
Imbruced
force-pushed
the
airbyte-ol-integration
branch
from
57c5af8
to
64db2c1
Compare
Imbruced
force-pushed
the
airbyte-ol-integration
branch
from
64db2c1
to
8f05643
Compare
|
WIP |
There was a problem hiding this comment.
why is this file under hook directory?
There was a problem hiding this comment.
to keep it close the place where validation happens :)
There was a problem hiding this comment.
but this is not a hook... it's confusing.
Some users may not even know what OL is or what this file is about.
There was a problem hiding this comment.
Based on hook methods (validation is making sure the schema is ok for connection output). I am not pushing to keep it there, where do you think is the best place to put it ? Thanks
There was a problem hiding this comment.
Lets hear from @mobuchowski as I assume this isn't just about Airbyte but for any other integration
There was a problem hiding this comment.
I think it would be the best to put in under utils IMO, it's not really tied to particular hook other than "code proximity".
There was a problem hiding this comment.
I think it would be the best to put in under utils IMO, it's not really tied to particular hook other than "code proximity".
|
|
||
| valid_connection = is_connection_valid(res.json()) | ||
| if not valid_connection: | ||
| self.log.warning("Connection is not valid") |
There was a problem hiding this comment.
I think it would be best to explain more in the warning what means that connection is not valid - unless this is something well known by Airbyte users - who I am not 🙂
Imbruced
force-pushed
the
airbyte-ol-integration
branch
3 times, most recently
from
5c69a75
to
8cf05fb
Compare
Imbruced
force-pushed
the
airbyte-ol-integration
branch
2 times, most recently
from
ad16faf
to
4f7db51
Compare
Signed-off-by: pawelkocinski <pawel93kocinski@gmail.com>
Signed-off-by: pawelkocinski <pawel93kocinski@gmail.com>
Signed-off-by: pawelkocinski <pawel93kocinski@gmail.com>
Signed-off-by: pawelkocinski <pawel93kocinski@gmail.com>
Imbruced
force-pushed
the
airbyte-ol-integration
branch
from
671ec8a
to
bf2cae1
Compare
|
@lmaczulajtys can this wait until #41122 get done? I'm trying to replace the API auth method and start using the official Python API SDK to run all operations. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
Add open lineage data from airbyte operator
This mr aims to add ol support for airbyte operator. It adds schema and column level lineage and simple statistics with number rows. Code is tested with unit tests and also manually verified using local apache airflow instance.
PR is using v2 version of ol objects but OperatorLineage is only working with v1 version, so its locked until migration. What I understand it's blocked by this mr #39530.