Explanation of Parsing vs. Compilation vs. Runtime

[dbeatty10]

🎩 Preview

Problem

There’s a lot of confusion about what dbt does at parse time vs. compile time / runtime. Even that separation is a relative simplification: parsing includes multiple steps, and while there are some distinctions between "compiling" and "running" a model, the two are very closely related.

It's come up many times before, and we expect it will keep coming up! A decent number of bug reports in dbt-core are actually rooted in a misunderstanding of when configs are resolved, especially when folks are using pre/post hooks, or configs that alter materialization behavior (partitions, merge_exclude_columns, etc).

Solution

So let's publish something that we can use as a reference! Markdown within the code base would be a good first step.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• Tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc)
• This PR includes type annotations for new and modified functions

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (408a789) 86.65% compared to head (4fc26eb) 86.43%.
Report is 16 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8744      +/-   ##
==========================================
- Coverage   86.65%   86.43%   -0.23%     
==========================================
  Files         176      176              
  Lines       25772    26009     +237     
==========================================
+ Hits        22332    22480     +148     
- Misses       3440     3529      +89     

Flag	Coverage Δ
integration	83.22% <ø> (-0.19%)	⬇️
unit	64.86% <ø> (-0.28%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 25 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jtcohen6]

Thanks for making this happen @dbeatty10! Talk is cheap, PRs are precious ;)

• I left a first round of comments, mostly catching things that have changed since I initially wrote this last December
• As far as where in the repo this should land - I wonder if the docs/architecture subfolder could be the right place? Perhaps (in the future) accompanied by some diagrams?

[dbeatty10]

• As far as where in the repo this should land - I wonder if the docs/architecture subfolder could be the right place? Perhaps (in the future) accompanied by some diagrams?

I'm guessing we'll want to move and/or rename this at least one more time. But in the interim, moved it here:
docs/eli64/parsing-vs-compilation-vs-runtime.md

when ELI5 just isn't detailed enough

[MichelleArk]

SQL (or Python!) + YAML

I can see how that can make the rest of the section a little more complicated to follow, so feel free to keep it simple

[dbeatty10]

Good call!

Attempted to address this in 4fc26eb