Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] new configuration to run tests on only the "new" data for snapshots and incremental models #10877

Open
graciegoheen opened this issue Oct 17, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@graciegoheen
Copy link
Contributor

graciegoheen commented Oct 17, 2024

Describe the feature

A more general solution for #10236 and #10864

Folks want to be able to test just their "new" data before it's inserted into their existing table. This is relevant for:

  • snapshots
  • incremental models

For example, you may want to test that your unique_key is actually unique for the "new" data. If that test fails don't insert the "new" data into the existing table.

We should create a new data test configuration to allow you to configure a given data to only run on the "new" data.

models:
  - name: my_model
    columns:
      - name: id
        data_tests:
          - unique:
              new_records_only: true  # name/spec TBD

Acceptance criteria:

  • you can set this config on a data test for an incremental model
  • you can set this config on a data test for a snapshot
  • when set, the data test will only run on "new" data before insert/merge
  • if data test fails, the subsequent insert/merge will be skipped
@graciegoheen graciegoheen added the enhancement New feature or request label Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant