Update spec MetricNode for dbt x MetricFlow integration #7812
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -125,6 +144,102 @@ def __init__(self, schema_parser: SchemaParser, yaml: YamlBlock): | |||
| self.schema_parser = schema_parser | |||
| self.yaml = yaml | |||
|
|
|||
| def _get_input_measure( | |||
There was a problem hiding this comment.
These helper functions look good enough for a beta, but I think we should look at ways to make the code less repetitive and mechanical.
More specifically, if we are going to continue along these lines, it would be better to have a way to declare how the unparsed objects map to the parsed objects, rather than performing the conversion manually for these and every new object.
QMalcolm
force-pushed
the
qmalcolm--metric-nodes-match-dsi-spec
branch
from
3a9daad to
0ed5c91
Compare
|
Probably shouldn't have rebased to fixup some commits. People are in the middle of reviewing. Thats my bad 🤦 Edit: I didn't edit any commits. On squashed/fixup'd. So if you're reviewing commit by commit, you should be able to pickup where you left off 🤞 |
QMalcolm
force-pushed
the
qmalcolm--metric-nodes-match-dsi-spec
branch
from
bcb9d3b to
288cc76
Compare
With the integration with MetricFlow we're taking a dependency on `dbt-semantic-interfaces` which acts as the source of truth for protocols which MetricFlow and dbt-core need to agree on. Additionally we're hard pinning to 0.1.0.dev3 for now. We plan on having a less restrictive specification when dbt-core 1.6 hits GA.
… protocol This isn't the "final" implementation/refactor of the Metric node for this branch, the first of a few. This just gets us to a place where the MetricNode satisfies the DSI Metric protocol even if some things are left unimplemented. Additioanlly, this does not include the changes necessary for parsing
We're intentionally breaking the spec. There will be a separate tool provided for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming things like `type` to `calculation_method`. This is problematic because `type` is on the new spec, while `calculation_method` is not. Additionally, since we're intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be used for any property renaming.
We're breaking the metric nodes explicitly. A separate tool will be provided for migrating from core 1.5 (manifest v9) metrics to core 1.6 (manifest v10) metrics
…nifest" This reverts commit ccad3ff.
QMalcolm
force-pushed
the
qmalcolm--metric-nodes-match-dsi-spec
branch
from
288cc76 to
1aaaf29
Compare
QMalcolm
requested review from
emmyoop and
VersusFacit
and removed request for
a team
Going filter -> where_str_template -> str is unnecessary during the yaml specification. The `WhereFilter` object is an abstraction which is necessary for under the hood operations, but the end user doesn't need to jump through those extra hoops when we can easily derive it.
QMalcolm
force-pushed
the
qmalcolm--metric-nodes-match-dsi-spec
branch
from
57dac12 to
4add088
Compare
…etrics specification More often than not, when one specifies a measure, nominator, denominator, measures, or metrics in a metric's `type_params`. This changes allows for end users in these instances do ``` measure: <measure_name> ``` or similar instead of always having to do ``` measure: name: <measure_name> ```
We want end users to be able to specify metric windows like ``` window: 14 days ``` instead of having to do ``` window: count: 14 granularity: day ```
QMalcolm
force-pushed
the
qmalcolm--metric-nodes-match-dsi-spec
branch
from
4add088 to
81e0779
Compare
|
I'd prefer to rebase off of main, but for the sanity of those in the midst of reviewing, I'm gonna merge main into this branch |
3 tasks
core/dbt/contracts/graph/manifest.py
Outdated
| @@ -985,6 +988,20 @@ def analysis_lookup(self) -> AnalysisLookup: | |||
| self._analysis_lookup = AnalysisLookup(self) | |||
| return self._analysis_lookup | |||
|
|
|||
| @property | |||
| def pydantic_semantic_manifest(self) -> Optional[PydanticSemanticManifest]: | |||
There was a problem hiding this comment.
Borderline nitpicky, the return type is actually a non-Optional PydanticSemanticManifest, which is the more attractive return type anyway.
There was a problem hiding this comment.
Oh right, good catch! I had planned on having it return None if there were not any SemanticModel or Metric nodes 🤦 I think returning an empty semantic model is a pretty good default. I'll get the typing updated, and then also update write_semantic_manifest to not have to anticipate it ever being None
…nd propgate consequences
MichelleArk
reviewed
Jun 8, 2023
core/dbt/contracts/graph/nodes.py
Outdated
| @@ -566,7 +568,7 @@ class FileSlice(dbtClassMixin, Replaceable): | |||
|
|
|||
|
|
|||
| @dataclass | |||
| class SourceFileMetadata(dbtClassMixin, Replaceable): | |||
| class Metadata(dbtClassMixin, Replaceable): | |||
There was a problem hiding this comment.
is this rename intentional? re: #7769 (comment)
There was a problem hiding this comment.
Nope! looks like my merge of main into this branch did screw some things up 😞
MichelleArk
approved these changes
Jun 8, 2023
leahwicz
added a commit
to leahwicz/dbt-core
that referenced
this pull request
Sep 12, 2023
* break out a large test suite as a separate execution to avoid memory issues on windows CI runs (#7669)
* Remove `DelayedFileHandler` (#7661)
* remove DelayedFileHandler
* Changelog
* set_path to no-op
* more no-ops for rpc
* Clearer comments
* Model Deprecation (#7562)
* CT-2461: Work toward model deprecation
* CT-2461: Remove unneeded conversions
* CT-2461: Fix up unit tests for new fields, correct a couple oversights
* CT-2461: Remaining implementation and tests for model/ref deprecation warnings
* CT-2461: Changelog entry for deprecation warnings
* CT-2461: Refine datetime handling and tests
* CT-2461: Fix up unit test data
* CT-2461: Fix some more unit test data.
* CT-2461: Fix merge issues
* CT-2461: Code review items.
* CT-2461: Improve version -> str conversion
* fix #7502: write run_results.json for run operation (#7655)
* Honor `--skip-profile-setup` parameter when inside an existing project (#7609)
* Honor `--skip-profile-setup` parameter when inside an existing project
* Use project name as the profile name
* Use separate file connections for reading and writing
* Raise a custom exception when no adapters are installed
* Test skipping interactive profile setup when inside a dbt project
* Replace `assert_not_called()` since it does not work
* Verbose CLI argument for skipping profile setup
* Use separate file connections for reading and writing
* Check empty list in a Pythonic manner
* Include null checks in utils test base (#7672)
* Include null checks in utils test base
* Add tests for the schema test
* Add tests for this macro
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Bump ubuntu from 23.04 to 23.10 (#7675)
* Bump ubuntu from 23.04 to 23.10
Bumps ubuntu from 23.04 to 23.10.
---
updated-dependencies:
- dependency-name: ubuntu
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
* Add automated changelog yaml from template for bot PR
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
* Adding performance modeling for 1.3.4 to refs/heads/main (#7525)
* adding performance baseline for 1.3.4
* Add newline
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* Adding performance modeling for 1.4.1 to refs/heads/main (#7527)
* adding performance baseline for 1.4.1
* Add newline
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* Adding performance modeling for 1.3.0 to refs/heads/main (#7530)
* adding performance baseline for 1.3.0
* Add newline
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* Adding performance modeling for 1.4.6 to refs/heads/main (#7532)
* adding performance baseline for 1.4.6
* Add newline
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* CT 2516 ensure that paths in Jinja context flags object are strings (#7678)
* Relaxed the pyyaml dependency to >=5.3. (#7681)
* Updating CODEOWNERS to consolidate (#7693)
Consolidating to remove Language and Execution and instead default to the `core-team`
* CT 2590 write pub artifact to log (#7686)
* Bumping version to 1.6.0b2 and generate changelog
* Adding link to 1.5 release notes (#7707)
* Add `%` to adapter suite test cases for `persist_docs` (#7699)
* Test table/view/column-level comments with `%` symbol
* Test docs block with `%` symbol
* Changelog entry
* Update to reusable workflow for branch testing (#7676)
* fix overlooked node12 case with abandonded marketplace action
* update slack notification
* remove spaces per formatting
* replace with cli dispatch
* move conditional
* add explicit token, temp comment out slack
* add checkout
* checkout teh right branch
* switch to PAT
* add back repo checkout
* manually check workflow status
* fix notification
* swap to reusable workflow
* fix path
* swap permissions
* remove trigger
* fix secrets
* point to main
* fix #7300: Enable state for deferral to be separate from state for selectors (#7690)
* Target path should be relative to project dir, rather than current working directory (#7706)
* Adding performance modeling for 1.2.0 to refs/heads/main (#7560)
* adding performance baseline for 1.2.0
* Adding newline
---------
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: leahwicz <60146280+leahwicz@users.noreply.github.com>
* Readd exp_path for config deprecation warnings (#7536)
* Dropped support for Python 3.7 (#7623)
* Template rendering optimization (#7451)
* CT-2478: Template rendering optimization
* CT-2478: Fix type annotation, and accomodate non-string unit test cases.
* fix StopIteration error when publication not found (#7710)
* add ability to select models by access (#7739)
* add ability to select models by access
* changie
* Update core/dbt/graph/selector_methods.py
* pass optional sql_header to empty subquery sql rendering (#7734)
* Fix doc link in selector.py (#7755)
* Fix doc link in selector.py
* Ran changie to modify changelog entry
* Fix constraint rendering for expressions and foreign key constraint types (#7512)
* add project_name to manifest metadata (#7754)
* Version 0 for model works for latest_version (#7712)
* force dependency between test models (#7767)
* remove entire changes folder (#7766)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* log PublicationArtifactAvailable even when partially parsing & public models unchanged (#7783)
* Fixes #7299: dbt retry (#7763)
* Fix not equals comparison to be null-safe for adapters/utils tests (#7776)
* Fix names within functional test
* Changelog entry
* Test for implementation of null-safe equals comparison
* Remove duplicated where filter
* Fix null-safe equals comparison
* Fix tests for `concat` and `hash` by using empty strings () instead of `null`
* Remove macro namespace interpolation
* Improve warnings for constraints and materialization types (#7696)
* first pass
* debugging
* regen proto types
* refactor to use warn_supported flag
* PR feedback
* Package-namespaced generate_x_name macro resolution (#7509)
* generalize BaseModelConstraintsRuntimeEnforcement (#7805)
* Fixes #7551: Create `add_from_artifact` to populate `state_relation` field of nodes (#7796)
* Validate public models are not materialized as ephemeral (#7794)
* fix error message for empty/None: --warn-error-options handling (#7735)
* Adap 496/add test connection mode to debug (#7741)
* --connection-flag
* Standardize the plugin functions used by DebugTask
* Cleanup redundant code and help logic along.
* Add more output tests to add logic coverage and formatting.
* Code review
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* fix: removing dependency from dependencies.yml (#7743)
* ADAP-2: Materialized Views (#7239)
* changie
* ADAP-387: Stub materialized view as a materialization (#7211)
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* updating to feature main
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-387: reverting db_api implementation (#7322)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* updating to feature main
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* added tests
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* typo
* moved materialized view tests into adapter test framework
* add enum to relation for comparison in jinja
---------
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
* ADAP-391: Add configuration change option (#7272)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* move up pre-existing drop of backup
* change relation type to view to be consistent
* add base test case
* fix jinja exception message expression, basic test passing
* align names between postgres and default to same convention
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integration tests
* add on_configuration_change to expected_manifest to pass functional tests
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* moved materialized view tests into adapter test framework
* add alter materialized view macro
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* updated default option for on_configuration_change on NodeConfig
* removed explicit calls to enum values
* add test setup for testing fail config option
* updated `config_updates` to `configuration_changes` to align with `on_configuration_change` name
* setup configuration change framework
* skipped tests that are expected to fail without adapter implementation
* cleaned up log checks
---------
Co-authored-by: Mike Alfare <mike.alfare@dbtlabs.com>
* ADAP-388: Stub materialized view as a materialization - postgres (#7244)
* move the body of the default macros into the postgres implementation, throw errors if the default is used, indicating that materialized views have not been implemented for that adapter
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-402: Add configuration change option - postgres (#7334)
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* added materialized view stubs and test
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* updated postgres to use the new macros structure
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* init copy of pr 387 to begin 391 implementation
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* remove postgres-specific materialization in favor of core default materialization
* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies
* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integretion tests
* add on_configuration_change to expected_manifest to pass functuional tests
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* revert to current approach for materializations
* added tests
* move materialized view logic into the `/materializations` directory in line with `dbt-core`
* moved default macros in `dbt-core` into `dbt-postgres`
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* moved materialized view tests into adapter test framework
* updated materialized view tests to use adapter test framework
* add alter materialized view macro
* add alter materialized view macro
* change class name, and config setup
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* added in `on_configuration_change` setting
* updated default option for on_configuration_change on NodeConfig
* updated default option for on_configuration_change on NodeConfig
* fixed list being passed as string bug
* removed explicit calls to enum values
* removed unneeded test class
* fixed on_configuration_change to be picked up appropriately
* add test setup for testing fail config option
* remove breakpoint, uncomment tests
* update skip scenario to use empty strings
* update skip scenario to avoid using sql at all, remove extra whitespace in some templates
* push up initial addition of indexes for mv macro
* push slight change up
* reverting alt macro and moving the do create_index call to be more in line with other materializations
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402
# Conflicts:
# core/dbt/contracts/graph/model_config.py
# core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/base.py
# tests/functional/materializations/test_materialized_view.py
* merge feature branch into story branch
* merge feature branch into story branch
* added indexes into the workflow
* fix error in jinja that caused print error
* working on test messaging and skipping tests that might not fit quite into current system
* add drop and show macros for indexes
* add drop and show macros for indexes
* add logic to determine the indexes to create or drop
* pulled index updates through the workflow properly
* convert configuration changes to fixtures, implement index changes into tests
* created Model dataclass for readability, added column to swap index columns for testing
* fixed typo
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* ADAP-395: Implement native materialized view DDL (#7336)
* changie
* changie
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* init attempt at mv and basic forms of helper macros by mixing view and experimental mv sources
* remove unneeded return statement, rename directory
* remove unneeded ()
* responding to some pr feedback
* adjusting order of events for mv base work
* move up prexisting drop of backup
* change relatiion type to view to be consistent
* add base test case
* fix jinja exeception message expression, basic test passing
* added materialized view stubs and test
* response to feedback, removeal of refresh infavor of combined create_as, etc.
* updated postgres to use the new macros structure
* swapping to api layer and stratgeies for default implementation (basing off postgres, redshift)
* remove stratgey to limit need for now
* remove unneeded story level changelog entry
* add strategies to condtional in place of old macros
* macro name fix
* rename refresh macro in api level
* align names between postgres and default to same convention
* align names between postgres and default to same convention
* change a create call to full refresh
* pull adapter rename into strategy, add backup_relation as optional arg
* minor typo fix, add intermediate relation to refresh strategy and initial attempt at further conditional logic
* init copy of pr 387 to begin 391 implementation
* updating to feature main
* updating to feature main
* init set of Enum for config
* work on initial Enum class for on_configuration_change base it off ConstraintTypes which is also a str based Enum in core
* remove postgres-specific materialization in favor of core default materialization
* update db_api to use native types (e.g. str) and avoid direct calls to relation or config, which would alter the run order for all db_api dependencies
* add clarifying comment as to why we have a single test that's expected to fail at the dbt-core layer
* add on_configuration_change to unit test expected values
* make suggested name change to Enum class
* add on_configuration_change to some integretion tests
* add on_configuration_change to expected_manifest to pass functuional tests
* removing db_api and strategies directories in favor of matching current materialization setups
* macro name change
* revert to current approach for materializations
* revert to current approach for materializations
* added tests
* move materialized view logic into the `/materializations` directory in line with `dbt-core`
* moved default macros in `dbt-core` into `dbt-postgres`
* added `is_materialized_view` to `BaseRelation`
* updated materialized view stored value to snake case
* typo
* moved materialized view tests into adapter test framework
* updated materialized view tests to use adapter test framework
* add alter materialized view macro
* add alter materialized view macro
* added basic sql to default macros, added postgres-specific sql for alter scenario, stubbed a test case for index update
* change class name, and config setup
* change class name, and config setup
* play with field setup for on_configuration_change
* add method for default selection in enum class
* renamed get_refresh_data_in_materialized_view_sql to align with experimental package
* changed expected values to default string
* added in `on_configuration_change` setting
* change ignore to skip
* added in `on_configuration_change` setting
* updated default option for on_configuration_change on NodeConfig
* updated default option for on_configuration_change on NodeConfig
* fixed list being passed as string bug
* fixed list being passed as string bug
* removed explicit calls to enum values
* removed explicit calls to enum values
* removed unneeded test class
* fixed on_configuration_change to be picked up appropriately
* add test setup for testing fail config option
* remove breakpoint, uncomment tests
* update skip scenario to use empty strings
* update skip scenario to avoid using sql at all, remove extra whitespace in some templates
* push up initial addition of indexes for mv macro
* push slight change up
* reverting alt macro and moving the do create_index call to be more in line with other materializations
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-402
# Conflicts:
# core/dbt/contracts/graph/model_config.py
# core/dbt/include/global_project/macros/materializations/models/materialized_view/alter_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/create_materialized_view_as.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/refresh_materialized_view.sql
# core/dbt/include/global_project/macros/materializations/models/materialized_view/replace_materialized_view.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/base.py
# tests/functional/materializations/test_materialized_view.py
* merge feature branch into story branch
* merge feature branch into story branch
* added indexes into the workflow
* fix error in jinja that caused print error
* working on test messaging and skipping tests that might not fit quite into current system
* Merge branch 'feature/materialized-views/ADAP-2' into feature/materialized-views/ADAP-395
# Conflicts:
# core/dbt/include/global_project/macros/materializations/models/materialized_view/get_materialized_view_configuration_changes.sql
# plugins/postgres/dbt/include/postgres/macros/adapters.sql
# plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
# tests/adapter/dbt/tests/adapter/materialized_views/test_on_configuration_change.py
# tests/functional/materializations/test_materialized_view.py
* moved postgres implemention into plugin directory
* update index methods to align with the configuration update macro
* added native ddl to postgres macros
* removed extra docstring
* updated references to View, now references MaterializedView
* decomposed materialization into macros
* refactor index create statement parser, add exceptions for unexpected formats
* swapped conditional to check for positive state
* removed skipped test now that materialized view is being used
* return the results and logs of the run so that additional checks can be applied at the adapter level, add check for refresh to a test
* add check for indexes in particular for apply on configuration scenario
* removed extra argument
* add materialized views to get_relations / list_relations
* typos in index change logic
* moved full refresh check inside the build sql step
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* removing returns from tests to stop logs from printing
* moved test cases into postgres tests, left non-test functionality in base as new methods or fixtures
* fixed overwrite issue, simplified assertion method
* updated import order to standard
* fixed test import paths
* updated naming convention for proper test collection with the test runner
* still trying to make the test runner happy
* rewrite index updates to use a better source in Postgres
* break out a large test suite as a separate run
* update `skip` and `fail` scenarios with more descriptive results
* typo
* removed call to skip status
* reverting `exceptions_jinja.py`
* added FailFastError back, the right way
* removed PostgresIndex in favor of the already existing PostgresIndexConfig, pulled it into its own file to avoid circular imports
* removed assumed models in method calls, removed odd insert records and replaced with get row count
* fixed index issue, removed some indirection in testing
* made test more readable
* remove the "apply" from the tests and put it on the base as the default
* generalized assertion for reuse with dbt-snowflake, fixed bug in record count utility
* fixed type to be more generic to accommodate adapters with their own relation types
* fixed all the broken index stuff
* updated on_configuration_change to use existing patterns
* updated on_configuration_change to use existing patterns
* reflected update in tests and materialization logic
* reflected update in tests and materialization logic
* reverted the change to create a config object from the option object, using just the option object now
* reverted the change to create a config object from the option object, using just the option object now
* modelled database objects to support monitoring all configuration changes
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* updated "skip" to "continue", throw an error on non-implemented macro defaults
* reverted centralized framework, retained a few reusable base classes
* updated names to be more consistent
* readability updates
* added readme specifying that `relation_configs` only supports materialized views for now
---------
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
Co-authored-by: Matthew McKnight <91097623+McKnight-42@users.noreply.github.com>
* Add SemanticModel Node Type (#7769)
* Add dbt-semantic-interfaces as a dependency
With the integration with MetricFlow we're taking a dependency on
`dbt-semantic-interfaces` which acts as the source of truth for
protocols which MetricFlow and dbt-core need to agree on. Additionally
we're hard pinning to 0.1.0.dev3 for now. We plan on having a less
restrictive specification when dbt-core 1.6 hits GA.
* Add implementations of DSI Metadata protocol to nodes.py
* CT-2521: Initial work on adding new SemanticModel node
* CT-2521: Second rough draft of SemanticModels
* CT-2521: Update schema v10
* CT-2521: Update unit tests for new SemanticModel collection in manifest
* CT-2521: Add changelog entry
* CT-2521: Final touches on initial implementation of SemanticModel parsing
* Change name of Metadata class to reduce potential for confusion
* Remove "Replaceable" inheritance, per review
* CT-2521: Rename internal variables from semantic_models to semantic_nodes
* CT-2521: Update manifest schema to reflect change
---------
Co-authored-by: Quigley Malcolm <quigley.malcolm@dbtlabs.com>
* Update spec MetricNode for dbt x MetricFlow integration and begin outputting semantic manifest artifact (#7812)
* Refactor MetricNode definition to satisfy DSI Metric protocol
* Fix tests involving metrics to have updated properties
* Update UnparsedMetricNode to match new metric yaml spec
* Update MetricParser for new unparsed and parsed MetricNodes
* Remove `rename_metric_attr`
We're intentionally breaking the spec. There will be a separate tool provided
for migrating from dbt-metrics to dbt x metricflow. This bit of code was renaming
things like `type` to `calculation_method`. This is problematic because `type` is
on the new spec, while `calculation_method` is not. Additionally, since we're
intentionally breaking the spec, this function, `rename_metric_attr`, shouldn't be
used for any property renaming.
* Fix tests for Metrics (1.6) changes
* Regenerated v10 manifest schema and associated functional test artifact state
* Remove no longer needed tests
* Skip / comment out tests for metrics functionality that we'll be implementing later
* Begin outputting semantic manifest artifact on every run
* Drop metrics during upgrade_manifest_json if manifest is v9 or before
* Update properties of `minimal_parsed_metric_dict` to match new metric spec
* Add changie entry for metric node breaking changes
* Add semantic model nodes to semantic manifest
* Bumping version to 1.6.0b3 and generate changelog
* Rm spaces from NodeType strings (#7842)
* Rm space from NodeType strings
* Add changelog entry
* Add new index.html and changelog yaml files from dbt-docs (#7836)
* Move remaining unit tests to `/tests` directory (#7843)
* moved remaining unit tests to directory
* fix path
* Delete profiles.yml
* rename remaining test refs
* Use project directory in path selector instead of cwd (#7829)
* Use contextvar to store and get project_root for path selector method
* Changie
* Modify test to check Path selector with project-dir
* Don't set cv_project_root in base task if no config
* update adapters url (#7779)
* update adapters url
in response to [docs.getedbt.com pr 3465](https://github.com/dbt-labs/docs.getdbt.com/issues/3465), updating this error message to point to the correct URL, which was recently changed.
old URL: https://docs.getdbt.com/docs/supported-data-platforms#adapter-installation
new URL: https://docs.getdbt.com/docs/connect-adapters#install-using-the-cli
thank you @dbeatty10 for your 🦅 👀 !
* adding changie entry
* Update .changes/unreleased/Breaking Changes-20230612-161159.yaml
---------
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Allow ProjectDependency to have extra fields (#7834)
* Allow ProjectDependency to have extra fields
* changelog
* Respect column `quote` config in model contracts (#7537)
* Update SemanticModel node to match DSI 0.1.0dev3 protocols (#7848)
* Add tests to ensure our semantic layer nodes satisfy the DSI protocols
These tests create runtime checkable versions of the protocols defined in
DSI. Thus we can instantiate instances of our semantic layer nodes and
use `isinstance` to check that they satisfy the protocol. These `runtime_checkable`
versions of the protocols should only exist in testing and should never
be used in the actual package code.
* Update the `Dimension` object of `SemanticModel` node to match DSI protocol
* Make `UnparsedDimension` more strict and update schema readers accordingly
* Update the `Entity` object of `SemanticModel` node to match DSI protocol
* Make `UnparsedEntity` more strict and update schema readers accordingly
* Update the `Measure` object of `SemanticModel` node to match DSI protocol
* Make `UnparsedMeasure` more strict and update schema readers accordingly
* Update the `SemanticModel` node to match DSI protocol
A lot of the additions are helper functions which we don't actually
use in core. This is a known issue. We're in the process of removing
a fair number of them from the DSI protocol spec. However, in the meantime
we need to implement them to satisfy the protocol unfortunately.
* Make `UnparsedSemanticModel` more strict and update schema readers accordingly
* Changie entry for updating SemanticModel node
* Bumping version to 1.6.0b4 and generate changelog
* Fix: `dbt show --inline` with `private` models (#7838)
* Add functional test
* Check resource_type before DbtReferenceError
* Changelog entry
* Bump mypy from 0.981 to 1.0.1 (#7027)
* Bump mypy from 0.981 to 1.0.1
Bumps [mypy](https://github.com/python/mypy) from 0.981 to 1.0.1.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](https://github.com/python/mypy/compare/v0.981...v1.0.1)
---
updated-dependencies:
- dependency-name: mypy
dependency-type: direct:development
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* Add automated changelog yaml from template for bot PR
* upgrade mypy and fix all errors
* fixing some duplicate imports from conflict resolution
* fix mypy errors from merging in main
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Add AdapterRegistered event log message (#7862)
* Add AdapterRegistered event log message
* Add AdapterRegistered to unit test
* make versioning and logging consistent
* make versioning and logging consistent
* add to_version_string
* remove extra equals
* format fire_event
* Allow setting packages in dependencies.yml and move dependencies to runtime config (#7857)
* [CT-2696] Skip jinia parsing of metric filters (#7885)
* Update metric filters in testing fixtures
I incorrectly wrote the tests such that they didn't include curly
braces, `{{..}}`, around things like `dimension(..)` for filters.
This updates the tests fixtures to have proper filter specifications
* Skip jinja rendering of `filter` key of metrics
Note that `filter` can show up in multiple places: as a root key
on a metric (`metric.filter`), on a metric input (`metric.type_params.metrics[x].filter`),
denominator (`metric.type_params.denominator.filter`), numerator
(`metric.type_params.numerator.filter`), and a metric input measure
(`metric.type_params.measure.filter` and `metric.type_params.measures[x].filter`).
In this commit we skip all of them :)
* Add changie doc for skipping jinja parsing for metric filters
* Update yaml renderer test for metrics
* Ct-2690 Support dbt-semantic-interfaces 0.1.0dev5 (#7888)
* Upgrade to dbt-semantic-interfaces v0.1.0dev5
This is a fairly simple upgrade. Literally it's just pointing at the
the new versions. The v3 schemas are directly compatible with v5 because
there were no protocol level changes from v3 to v5. All the changers were
updates to tools MetricFlow uses from DSI, not tools that we ourselves
are using in core (yet).
* Add changie doc for DSI version bump
* Upgrade core to support dbt-semantic-interfaces `0.1.0dev7` (#7903)
* Bump DSI dependency version to 0.1.0dev7
* Cleaner DSI type enum importing
Previoulsy we had to use individual import paths for each type enum
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the type enums from a
singular path.
* Cleaner DSI protocol importing
Previoulsy we had to use individual import paths for each protocol
that dbt-semantic-interfaces provided. However, dbt-semantic-interfaces
has been updated to allow for importing all the protocols from a
singular path.
* Add semantic protocol satisifcation test for metric type params
* Replace `metric.type_params.measures` with `metric.type_params.input_measures`
In DSI 0.1.0dev7 `measures` on metric type params became `input_measures`.
Additionally `input_measures` should not be user specified but something
we compile at parse time, thus we've removed it from `UnparsedMetricTypeParams`.
Finally, actually populating `input_measures` is somewhat complicated due
to the existance of derived metrics, thus that work is being pushed
off to CT-2707.
* Update metric numerator/denominator to be `MetricInput`s
In DSI 0.1.0dev7 `metric.type_params.numerator` and `metric.type_params.denominator`
switched from being `MetricInputMeasure`s to `MetricInput`s. This
commit reflects that change. Additionally, some helper functions on
metric type params were removed related to the numerator and denominator.
Thus we've removed them respectively in this commit.
* Add protocol satisfaction tests for `MetricInput` and `MetricInputMeasure`
* Add `post_aggregation_reference` to `MetricInput` and fix typo in `MetricInputMeasure`
DSI 0.1.0dev7 added `post_aggregation_reference` to the `MetricInput` protocol,
thus we've added it to our implementation in core. Additionally, we had a typo
in a method name in our implementation of `MetricInputMeasure`, ironically
a similar function to the one we've added for `MetricInput`
* Changie doc for upgraded to DSI 0.1.0dev7
* Fix parsing of metric numerator and denominator in schema_yaml_readers
Previously numerator and denominator of a metric were `MetricInputMeasure`s,
now they're `MetricInput`s. Changing the typing isn't enough though.
We have parsing functions in `schema_yaml_readers` which were specifically
parsing the numerator and denominator as if they were `MetricInputMeasure`s.
Thus we had to updating the schema_yaml_readers to parse them as `MetricInput`s.
During this we had some logic in a parsing function `_get_metric_inputs` which
could be abstracted to newly added functions.
* Fixes #7753: Fix regression in `run-operation` to not require the name of the package to run (#7811)
* Reorganize, annotate, and revise dependency pins (#7368)
* Reorganize + annotate dependencies
* Loosen pins on other dbt-labs packages
* Minor pins, no patch pins
* Rm support for py37
* Fix protobuf pin
* Bump networkx upper bound #6551
* Do not jinja render packages from dependencies yml (#7910)
* Add some comments to methods constructing Project/RuntimeConfig
* Save flag that packages dict came from dependencies.yml
* Test for not rendering packages_dict
* Changie
* Ensure packages_yml_dict and dependencies_yml_dict are dictionaries
* Ensure "packages" passed to render_packages is a dict
* Remove --config-dir instead of fixing #7774 (#7793)
* Stringfy the dir always to solve the bug.
* Add changelog
---------
Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>
* Refactor/unify public and model nodes (#7891)
* Further Integrate Semantic Models (#7917)
* CT-2651: Add Semantic Models to the manifest and various pieces of graph linking code
* CT-2651: Finish integrating semantic models into the partial parsing system
* CT-2651: More semantic model details for partial parsing
* CT-2651: Remove merged references to project_dependencies
* CT-2651: Revise changelog entry
* CT-2651: Disable unit test until partial parsing of semantic models is complete.
* CT-2651: Temporarily disable an apparently-flaky test.
* Resolve SemanticModel refs in the same way as other refs (#7895)
* UninstalledPackagesFoundError references correct packages specified path (#7886)
* Fix: safe remove of external nodes from nodes.depends_on (#7923)
* Bumping version to 1.6.0b5 and generate changelog
* CT-2719: Rename the semantic_nodes collection on the manifest to semantic_models (#7927)
* Improved Semantic expr specification handling (#7876)
* Update semantic model parsing test to check different measure expr types
* Allow semantic model measure exprs to be defined with ints and bools in yaml
Sometimes the expr for a measure can defined in yaml with a bool or an int.
However, we were only allowing for strings. There was a work around for this,
which was wrapping your bool or int in double quotes in the yaml, but
this can be fairly annoying for the end user.
* Changie doc for fixing measure expr yaml specification
* Improved Semantic Model Measure Percentile defaults (#7877)
* Update semantic model parsing test to check measure agg params
* Make `use_discrete_percentile` and `use_approximate_percentile` non optional and default false
This was a mistake in our implementation of the MeasureAggregationParams.
We had defined them as optional and defaulting to `None`. However, as the
protocol states, they cannot be `None`, they must be a boolean value.
Thus now we now ensure them.
* Add changie doc for measure percentile fixes
* CT-2732: Fix selector methods to include semantic models (#7936)
* Bumping version to 1.6.0b6 and generate changelog
* update mashumaro to 3.8.1 (#7951)
* Update mashumaro to 3.8
* Change to 3.8.1
* Changie
* Fix comment for `dbt retry` (#7932)
* Factor Out Repeated Logic in the PartialParsing Class (#7952)
* CT-2711: Add remove_tests() call to delete_schema_source() so that call sites are more uniform with other node deletion call sites. This will enable further code factorization.
* CT-2711: Factor repeated code section (mostly) out of PartialParsing.handle_schema_file_changes()
* CT-2711: Factor a repeated code section out of schedule_nodes_for_parsing()
* Fix macro namespace search packages (#5804)
* allow setting enabled and depends_on_nodes from ModelNodeArgs (#7930)
* Set access model node args (#7966)
* Add Partial Parsing Support for Semantic Models (#7964)
* CT-2711: Add partial parsing support for semantic models
* CT-2711: Fix typo identified in code review
* Add merge incremental strategy for postgres (#6951)
* Add merge incremental strategy
* Expect merge to be a valid strategy for Postgres
---------
Co-authored-by: Anders Swanson <anders.swanson@dbtlabs.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Handle state:modified for external nodes (#7925)
* Add thread_id context var (#7942)
* Add thread_id context var
* Changie
* Fix context test
* Update .changes/unreleased/Features-20230623-173357.yaml
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Fix tests
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
* Add invocation command flag (#7939)
* Add invocation command flag
* Add changie entry
* Update .changes/unreleased/Features-20230623-111254.yaml
* consolidate cross-project ref entrypoint + plugin framework (#7955)
* Add restrict-access to dbt_project.yml (#7962)
* Bumping version to 1.6.0b7 and generate changelog
* Fix CTE insertion position when the model uses WITH RECURSIVE (#7350) (#7414)
* Replace space with underscore in NodeType strings (#7947)
* Use events.contextvar because of multiprocessing unable to pickle ContextVar (#7949)
* Add task contextvars to events/contextvars.py
* Use events.contextvars instead of task.contextvars
* Changie
* CT-2757: Fix unit test which broke due to merge issues (#7978)
* `dbt clone` (#7881)
Co-authored-by: Matthew McKnight <matthew.mcknight@dbtlabs.com>
* CT-2707: Populate metric input measures (#7984)
* Fix tests fixtures which were using measures for metric numerator/denominators
In our previous upgrade to DSI dev7, numerators and denominators for
metrics switched from being `MetricInputMeasure`s to `MetricInput`s.
I.e. metric numerators and denominators should references other metrics,
not semantic model measures. However, at that time, we weren't actually
doing anything with numerators and denominators in core, so no issue
got raised. The changes we are about to make though are going to surface
these issues..
* Add tests for ensuring a metric's `input_measures` gets properly populated
* Begin populating `metric.type_params.input_measures`
This isn't my favorite bit of code. Mostly because there are checks for
existence which really should be handled before this point, however a
good point for that to happen doesn't exist currently. For instance,
in an ideal world by the time we get to `_process_metric_node`, if a
metric is of type `RATIO` and the nominator and denominator should be
guaranteed.
* Update test checking that disabled metrics aren't added to the manifest metrics
We updated from the metric `number_of_people` to `average_tenure_minus_people` for
this test because disabling `number_of_people` raised other exceptions at parse
time due to a metric referencing a disabled metric. The metric `average_tenure_minus_people`
is a leaf metric, and so for this test, it is a better candidate.
* Update `test_disabled_metric_ref_model` to have more disabled metrics
There are metrics which depend on the metric `number_of_people`. If
`number_of_people` is disabled without the metrics that depend on it
being disabled, then a different (expected) exception would be raised
than the one this test is testing for. Thus we've disabled those
downstream metrics.
* Add test which checks that metrics depending on disabled metrics raise an exception
* Add changie doc for populating metric input measures
* Unified to UTC (#7665)
* UnifiedToUTC
* Check proximity of dbt_valid_to and deleted time
* update the message to print if the assertion fails
* add CHANGELOG entries
* test only if naive
* Added comments about naive and aware
* Generalize comparison of datetimes that are "close enough"
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* type + fix typo in ModelNodeArgs.unique_id (#7992)
* Remove pin of sqlparse, minor refactoring, add tests (#7993)
* [CT-2551] Make state selection MECE (#7773)
* ct-2551: adds old and unmodified state selection methods
* ct-2551: update check_unmodified_content to simplify
* add unit and integration tests for unmodified and old
* add changelog entry
* ct-2551: reformatting of contingent adapter assignment list
* Added note before running integration tests (#7657)
Co-authored-by: Kshitij Aranke <kshitij.aranke@dbtlabs.com>
* Add time spine table configuration to semantic manifest (#7996)
* Bump dbt-semantic-interface to dev8
* Create time_spline_table_configuration in semantic manifest
* Add metricflow_time_spin to semantic_models tests
* Remove skip from test
* Changie
* Update exception message
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
---------
Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
* allow on_schema_change: fail for incremental models with contracts (#8006)
* versioned node selection with underscore delimiting (#7995)
* Bumping version to 1.6.0b8 and generate changelog
* fire proper event for inline query error (#7960)
* CT-2752: Extract methods to new SemanticManifest class for better encapsulation of details. (#8012)
* CT-2780 [Docs] Fix-toc-links-in-contributing-md (#8017)
* docs(contributing): fix-toc-link-in-contributing-md
* docs(contributing-md): fix-link2
* Fix-typo
* Remove backtick from href
* changie new
* Cough commit / trigger CI
* Nicer error message for contracted model missing 'columns' (#8024)
* ModelNodeArgs.unique_id - include v (#8038)
* pin click (#8050)
* pin click
* changelog
* add workflow for flaky test testing (#8044)
* add workflow for flaky test testing
* improve docs
* rename workflow
* update default input
* add min passing tests
* partial parse file path (#8032)
* Update the PR template (#7892)
* add interface changes section to the PR template
* update entire template
* split up choices for tests and interfaces
* minor formatting change
* add line breaks
* actually put in line breaks
* revert split choices in checklist
* add line breaks to top
* move docs link
* typo
* Semantic Model Validation (#8049)
* Use dbt-semantic-interface validations on semantic models and metrics defined in Core.
* Remove empty test, since semantic models don't generate any validation warnings.
* Add changelog entry.
* Temporarily remove requirement that there must be semantic models definied in order to define metrics
* Fix accidental propagation of log messages to root logger (#7882)
* Fix accidental propagation of log messages to root logger.
* Add changelog entry
* Fixed an issue which blocked debug logging to stdout with --log-level debug, unless --debug was also used.
* add negative part number test case for split part cross db util (#7200)
* add negative test case
* changie
* missed a comma
* Update changelog entry
* Add a negative number (rather than subtract a positive number)
---------
Co-authored-by: Doug Beatty <44704949+dbeatty10@users.noreply.github.com>
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* Remove `create_metric` as a public facing `SemanticModel.Measure` property (#8068)
* Remove `create_metric` as a public facing `SemanticModel.Measure` property
We want to add `create_metric`. The `create_metric` property will be
incredibly useful. However, at this time it is not hooked up, and we don't
have time to hook it up before the code freeze for 1.6.0rc of core. As
it doesn't do anything, we shouldn't allow people to specify it, because
it won't do what one would expect. We plan on making the implementation
of `create_metric` a priority for 1.7 of core
* Changie doc for the removal of create_metric property
* Flaky Test Workflow (#8055)
* add permissions
* replace db setup
* try with bash instead of just pytest flags
* fix test command
* remove spaces
* remove force-flaky flag
* add starting vlaues
* add mac and windows postgres isntall
* define use bash
* fix typo
* update output report
* tweak last if condition
* clarify failures/successful runs
* print running success and failure tally
* just output pytest instead of capturing it
* set shell to not exit immediately on exit code
* add formatting around results for easier scanning
* more output formatting
* add matrix to unlock parallel runners
* increase to ten batches
* update debug
* add comment
* clean up comments
* Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release (#8071)
* Use Ubuntu 22.04.2 LTS (Jammy Jellyfish) since it is a long-term supported release
* Changelog entry
* Fixes #7785: fail-fast behavior (#8066)
* Add validate_sql method to base adapter with implementation for SQLAdapters (#8001)
* Add dry_run method to base adapter with implementation for SQLAdapters
resolves #7839
In the CLI integration, MetricFlow will issue dry run queries as
part of its warehouse-level validation of the semantic manifest,
including all semantic model and metric definitions.
In most cases, issuing an `explain` query is adequate, however,
BigQuery does not support the `explain` keyword and so we cannot
simply pre-pend `explain` to our input queries and expect the
correct behavior across all contexts.
This commit adds a dry_run() method to the BaseAdapter which mirrors
the execute() method in that it simply delegates to the ConnectionManager.
It also adds a working implementation to the SQLConnectionManager and
includes a few test cases for adapter maintainers to try out on their own.
The current implementation should work out of the box with most
of our adapters. BigQuery will require us to implement the dry_run
method on the BigQueryConnectionManager, and community-maintained
adapters can opt in by enabling the test and ensuring their own
implementations work as expected.
Note - we decided to make these concrete methods that throw runtime
exceptions for direct descendants of BaseAdapter in order to avoid
forcing community adapter maintainers to implement a method that does
not currently have any use cases in dbt proper.
* Switch dry_run implementation to be macro-based
The common pattern for engine-specific SQL statement construction
in dbt is to provide a default macro which can then be overridden
on a per-adapter basis by either adapter maintainers or end users.
The advantage of this is users can take advantage of alternative
SQL syntax for performance or other reasons, or even to enable
local usage if an engine relies on a non-standard expression and
the adapter maintainer has not updated the package.
Although there are some risks here they are minimal, and the benefit
of added expressiveness and consistency with other similar constructs
is clear, so we adopt this approach here.
* Improve error message for InvalidConnectionError in test_invalid_dry_run.
* Rename dry_run to validate_sql
The validate_sql name has less chance of colliding with dbt's
command nomenclature, both now and in some future where we have
dry-run operations.
* Rename macro and test files to validate_sql
* Fix changelog entry
* Update validate sql test classes to new nomenclature (#8076)
The original implementation of validate_sql was called dry_run,
but in the rename the test classes and much of their associated
documentation still retained the old naming.
This is mainly cosmetic, but since these test classes will be
imported into adapter repositories we should fix this now before
the wrong name proliferates.
* Remove volume declaration (#8069)
* Remove volume declaration
* Changelog entry
---------
Co-authored-by: Doug Beatty <doug.beatty@dbtlabs.com>
* update contributing.md reference to test/integration (#8073)
* Er/ct 2675 test custom target (#8079)
* remove skip
* fix retry test
* 8030/fix contract checksum (#8072)
* Add semantic_models to tracked resource counts (#8078)
* Add semantic_models to tracked resource counts
* Add changelog entry
* Simplify node statistic tabulation.
* Remove review comment. Replace with explanation.
---------
Co-authored-by: Peter Allen Webb <peter.webb@dbtlabs.com>
* CT-2691 Fix the populating of a `Metric`'s `depends_on` property (#8015)
* Add metrics from metric type params to a metric's depends_on
* Add Lookup utility for finding `SemanticModel`s by measure names
* Add the `SemanticModel` of a `Metric`'s measure property to the `Metric`'s `depends_on`
* Add `SemanticModelConfig` to `SemanticModel`
Some tests were failing due to `Metric`'s referencing `SemanticModels`.
Specifically there was a check to see if a referenced node was disabled,
and because `SemanticModel`'s didn't have a `config` holding the `enabled`
boolean attr, core would blow up.
* Checkpoint on test fixing
* Correct metricflow_time_spine_sql in test fixtures
* Add check for `SemanticModel` nodes in `Linker.link_node`
Now that `Metrics` depend on `SemanticModels` and `SemanticModels`
have their own dependencies on `Models` they need to be checked for
in the `Linker.link_node`. I forget the details but things blow up
without it. Basically it adds the SemanticModels to the dependency
graph.
* Fix artifacts/test_previous_version_state.py tests
* fix access/test_access.py tests
* Fix function metric tests
* Fix functional partial_parsing tests
* Add time dimension to semantic model in exposures fixture
* Bump DSI version to a minimum of 0.1.0dev10
DSI 0.1.0dev10 fixes an incoherence issue in DSI around `agg_time_dimension`
setting. This incoherence was that `measure.agg_time_dimension` was being
required, even though it was no longer supposed to be a required attribute
(it's specificially typed as optional in the protocol). This was causing
a handful of tests to fail because the `semantic_model.defaults.agg_time_dimension`
value wasn't being respected. Pulling in the fix from DSI 0.1.0dev10 fixes
the issue.
Interestingly after bumping the DSI version, the integration tests were
still failing. If I ran the tests individually they passed though. To get
`make integration` to run properly I ended up having to clear my `.tox`
cache, as it seems some outdated state was being persisted.
* Add test specifically for checking the `depends_on` of `Metric` nodes
* Re-enable test asserting calling metric nodes in models
* Migrate `checked_agg_time_dimension` to `checked_agg_time_dimension_for_measure`
DSI 0.1.0dev10 moved `checked_agg_time_dimension` from the `Measure`
protocol to the `SemanticModel` protocol as `checked_agg_time_dimension_for_measure`.
This finishes a change where for a given measure either the `Measure.agg_time_dimension`
or the measure's parent `SemanticModel.defaults.agg_time_dimension` needs to be
set, instead of always require the measure's `Measure.agg_time_dimension`.
* Add changie doc for populating metric
---------
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
* add __test__ = False to non-test classes that start with Test (#8086)
* gitignore user.yml and profiles.yml (#8087)
* [CT-2821] Support `dbt-semantic-interfaces~=0.1.0rc1` (#8085)
* Bump version support for `dbt-semantic-interfaces` to `~=0.1.0rc1`
* Add tests for asserting WhereFilter satisfies protocol
* Add `call_parameter_sets` to `WhereFilter` class to satisfy protocol
* Changie doc for moving to DSI 0.1.0rc1
* [CT-2822] Fix `NonAdditiveDimension` Implementation (#8089)
* Add test to ensure `NonAdditiveDimension` implementation satisfies protocol
* Fix typo in `NonAdditiveDimension`: `window_grouples` -> `window_groupings`
* Add changie doc for typo fix in NonAdditiveDimension
* Fix query comment tests (#7928) (#7928)
* add env vars for datadog ci visibility (#8097)
* add env vars for datadog ci visibility
* modify pytest command for tracing
* fix posargs
* move env vars to job that needs them
* add test repeater to DD
* swap flags
* Apply new integration tests to existing framework to identify supported features (#8099)
* applied new integration tests to existing framework
* applied new integration tests to existing framework
* generalized tests for reusability in adapters; fixed drop index issue
* generalized tests for reusability in adapters; fixed drop index issue
* removed unnecessary overrides in tests
* adjusted import to allow for usage in adapters
* adjusted import to allow for usage in adapters
* removed fixture artifact
* generalized the materialized view fixture which will need to be specific to the adapter
* unskipped tests in the test runner package
* corrected test condition
* corrected test condition
* added missing initial build for the relation type swap tests
* bumpversion 1.7.0a1 (#8111)
* Cleanup main after cutting new 1.6.latest branch (#8102)
* Publish coverage results to codecov.io (#8127)
* wrap deprecation warnings in warn_or_error calls (#8129)
* Add new index.html and changelog yaml files from dbt-docs (#8131)
* Rm docs changelog entry (#8147)
* Rearrange pp_versioned_models test (#8150)
* format exception from dbtPlugin.initialize (#8143)
* Hotfix for 372: Use JSONEncoder in json.dumps (#8151)
* Correcting spelling of partition (#8101)
* Correcting spelling of partition
* Changie entry
* add a node status (#8174)
* Add implementation issue template (#8176)
* [CT-1849] `_connection_exception_retry` handles `EOFError` exceptions (#8182)
* Add test for checking that `_connection_exception_retry` handles `EOFError`s
* Update `_connection_exception_retry` to handle `EOFError` exceptions
* Add changie docs for `_connection_exception_retry` handling `EOFError` exceptions
* Fixed double underline (#7944)
* Fixed double-underline
* backward compatibility postgres_get_relations
* Remove invalid comments
* compatibility adapter and get_relation
* fix generic for call
* fix adapter dispatch grammar issue
* [CT-1483] Let macro names include word materialization (#8181)
* Add test asserting that a macro with the work materializtion doesn't cause issues
* Let macro names include the word `materialization`
Previously we were checking if a macro included a materialization
based on if the macro name included the word `materialization`. However,
a macro name included the word `materialization` isn't guarnteed to
actually have a materialization, and a macro that doesn't have
`materialization` in the name isn't guaranteed to not have a materialization.
This change is to detect macros with materializations based on the
detected block type of the macro.
* Add changie doc materialization in macro detection
* [CT-2594] Fix serialization of `warn_error_options` on Contexts (#8180)
* Add test ensuring `warn_error_options` is dictified in `invocation_args_dict` of contexts
* Add dictification specific to `warn_error_options` in `args_to_dict`
* Changie doc for serialization changes of warn_error_options
* Check for existing_relation immediately prior to renaming (#8193)
* add param to control maxBytes for single dbt.log file (#8200)
* add param to control maxBytes for single dbt.log file
* nits
* nits
* Update core/dbt/cli/params.py
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
---------
Co-authored-by: Peter Webb <peter.webb@dbtlabs.com>
* Update Core for Latest dbt-extractor with Version Parsing (#8206)
* Update dbt-extractor requirement, and adjust ref handling accordingly
* Add changelog entry.
* Update implementation-ticket.yml (#8208)
* up timeout (#8218)
* Bump mypy from 1.3.0 to 1.4.0 (#7912)
* Bump mypy from 1.3.0 to 1.4.0
Bumps [mypy](https://github.com/python/mypy) from 1.3.0 to 1.4.0.
- [Commits](https://github.com/python/mypy/compare/v1.3.0...v1.4.0)
---
updated-dependencies:
- dependency-name: mypy
dependency-type: direct:development
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Add automated changelog yaml from template for bot PR
* add to pre-commit config
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <buildbot@fishtownanalytics.com>
Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
* Initialize sqlparse lexer and tweak order of setting compilation fields (#8215)
* Split integration tests into parallel groups / jobs (#6346)
* Ensure that target_schema from snapshot config is promoted to node level (#8117)
* pin upper bound for sqlparse (#8236)
* pin upper bound for sqlparse
* changelog
* [CT-2879] Fix unbound variable error in `checked_agg_time_dimension_for_measure` (#8235)
* Fix unbound variable error in `checked_agg_time_dimension_for_measure`
* Improve assertion error message in `checked_agg_time_dimension_for_measure`
* Add changie doc for checked_agg_time_dimension_for_measure unbound variable fix
…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #7500 #7404
Description
Checklist
changie newto create a changelog entry