Support for Airflow Datasets

[cicerojmm]

Implementation of Airflow Datasets (data-aware scheduling) functionality that became available from version 2.4.0.

In this implementation it is possible to build Datasets in two ways:

• The first is directly referencing the URI of the dataset in the appropriate parameters of both the producing tasks (outlets) and the consuming DAGs (schedule)
• the second is to have a specific YAML file to declare datasets (URI and Names) and thus reference this file indicating only the names of the datasets to be used by the producing tasks and consuming DAGs, as the URI will be selected automatically.

Referencied Issue: #151

[ericbuzato]

This feature is going to be really cool, I can't wait to use it in my pipelines! ☝️

[codecov-commenter]

Codecov Report

Patch coverage: 20.00% and project coverage change: -4.02 ⚠️

Comparison is base (7d2782b) 83.61% compared to head (da8d3b9) 79.60%.

❗ Current head da8d3b9 differs from pull request most recent head f169dab. Consider uploading reports for the commit f169dab to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #171      +/-   ##
==========================================
- Coverage   83.61%   79.60%   -4.02%     
==========================================
  Files           6        6              
  Lines         525      559      +34     
==========================================
+ Hits          439      445       +6     
- Misses         86      114      +28     

Impacted Files	Coverage Δ
dagfactory/utils.py	87.06% <15.38%> (-9.05%)	⬇️
dagfactory/dagbuilder.py	73.13% <22.72%> (-3.35%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ciceromoura]

@ajbosco
I added tests to increase coverage, I believe now the merge will pass

[ajbosco]

Thanks!

[karguru-aws]

Does the code factory support AWS Operators like Glue ?