Feature/schema versions #2767
Feature/schema versions #2767
Conversation
There was a problem hiding this comment.
I don't mind dbt_schema_version. That said, if you did want to change it, I'd support something even more explicit, like dbt_artifact_schema_version.
I tested locally and saw dbt_version and dbt_schema_version propagated in every artifact except sources.json. Is that known, and to come in a future PR?
For my own edification: would our plan be to keep schema versions consistent across all artifacts, even if we only need to change one or two? Or is there a possibility of (e.g.) a v4 manifest, v3 run results, v2 catalog?
| self.manifest = WritableManifest.read(str(manifest_path)) | ||
| try: | ||
| self.manifest = WritableManifest.read(str(manifest_path)) | ||
| except IncompatibleSchemaException as exc: |
core/dbt/exceptions.py
Outdated
|
|
||
| msg = ( | ||
| f'Expected a schema version of "{self.expected}" in ' | ||
| f'{self.filename}, but found {found_str}' |
There was a problem hiding this comment.
What do you think of offering a hint to users who might be initially confused about what schema version means?
Expected a schema version of "https://schemas.getdbt.com/dbt/manifest/v1.json" in
.state/manifest.json, but found "https://schemas.getdbt.com/dbt/manifest/v2.json".
Are you running with a different version of dbt?
This will be fixed in the metadata PR. I didn't realize it was relevant here, but the freshness output has its own special schema that isn't the freshness results, which is used solely for writing, and that doesn't get the schema version. I'm going to coerce it into the standard-ish format as part of that.
I hadn't planned on keeping them consistent, though I can see the value in it. I think mixed versions are pretty reasonable things to have. If we change the catalog and your code only cares about the manifest, why should you get an error? This is especially relevant for |
Fair! |
resolves #2670
Description
Add schema versions and dbt versions dbt's artifact output.
Long-term, these belong in the
metadatafield described in #2761.I think I would like to get some automated mechanism for this into either hologram or the class, rather than this method of manually inserting things.
We can definitely change the name to something better, I don't love
dbt_schema_version.Checklist
CHANGELOG.mdand added information about my change to the "dbt next" section.