Any thoughts about lineage?

[radbrt]

Because recap can read from lots of different data sources, I started to wonder about lineage. Any thought about adding a module to write lineage between datasets in the catalog?

dbt (etc) knows most of what happens in the DWH, but E/L is often standalone with little lineage. Simple libraries for parsing api output from fivetran/hightouch, singer logs, or logging from python scripts could go a long way in adding both lineage and possibly observability.

[criccomini]

Yes! I'm focusing on schema right now, but lineage is definitely something I want to cover eventually. :)