Skip to content

Latest commit

 

History

History
326 lines (268 loc) · 15.4 KB

CHANGELOG.md

File metadata and controls

326 lines (268 loc) · 15.4 KB
Raw

Release [1.4.9], 2023-10-27

Improvement

  • Lots of work on Distributed table materializations. Big thanks to gfunc for the additional PR and Zhenbang for code review and suggestions. See the README for details on how to use the new functionality.

Bug Fix

Release [1.4.8], 2023-08-22

Bug Fix

  • Fixed issues with experimental Distributed table materializations. Closes ClickHouse#179. Thanks to Zhebnang for the report and for contributing to the fix with gfunc.

Release [1.4.7], 2023-08-09

Bug Fix

  • Fixed an exception in "legacy" incremental materializations that are not distributed

Release [1.4.6], 2023-07-27

Bug fix

  • Lightweight deletes could fail in environments where the HTTP session was not preserved (such as clusters behind a non-sticky load balancer). This has been fixed by sending the required settings with every request instead of relying on a SET statement. A similar approach has been used to persist the 'insert_distributed_sync' setting for Distributed table materializations.

Release [1.4.5], 2023-07-27

Improvement

  • Adds additional experimental support for Distributed table engine models and incremental materialization. See the README for details. Thanks to gladkikhtutu for the contribution!

Release [1.4.4], 2023-07-19

Bug Fixes

  • Fixed two logging/exception handling issues that would cause exception on startup or when handling some exceptions from the ClickHouse server. Partially addresses ClickHouse#169.
  • Fixed issue with the on_cluster macro that would break the exchange tables step of incremental materializations with an active cluster. Thanks to Andrew Davis for the PR. Closes ClickHouse#167

Release [1.4.3], 2023-06-27

Bug Fix

Improvement

  • Added macros for creating distributed tables. See the distributed_table.sql include file. Thanks to gladkikhtutu for the contribution.

Release [1.4.2], 2023-05-14

Bug fixes

  • Create initial dbt database (if not found) on the defined cluster on first run, instead of just the execution node. Thanks to Jens Hoevenaars for the PR
  • Fix the SYSTEM SYNC REPLICA statement when exchanging tables ON CLUSTER for incremental materializations. Thanks to Schum for PR. Closed ClickHouse#157.

Release [1.4.1], 2023-05-11

Improvements

  • Reduce the number of SQL calls for Modify Comment operations. Thanks to Konstantin Ilchenko.
  • Add "on cluster" to several additional macros to better support distributed tables. Thanks to Saurabh Bikram
  • Add the unique invocation id to temporary "new data" used in delete+insert incremental materializations to allow parallel transformations on the same table. In general parallel transformations are risky, so this approach should only be used when transformations are explicitly limited to non-overlapping data ranges. Closes ClickHouse#150

Release [1.4.0], 2023-02-06

Improvements

  • Support dbt [1.4.1] ClickHouse#135
    • Adds support for Python 3.11
    • Adds additional dbt 1.4.0 tests
    • Adds support for incremental_predicates. This only applies to delete+insert incremental strategy. Note that incremental predicates that depend on "non-deterministic" data (such as a subquery using a table that is accepting inserts) could lead to unexpected results for ReplicatedMergeTree tables depending on the timing of the incremental materialization.
    • Replaces deprecated Exception classes
  • Setting the use_lw_deletes profile value to True will now attempt to enable the allow_experimental_lightweight_delete setting for the dbt session (if user has such permissions on the ClickHouse server). See ClickHouse#133

Bug Fix

  • Composite unique keys specified as a list would not work with incremental materializations. This has been fixed.

Release 1.3.3, 2023-01-18

Documentation Update

  • The documentation has been updated to reflect that dbt-clickhouse does support ephemeral models, and ephemeral model tests do pass. However, due to a ClickHouse limitation, CTEs will not work directly with INSERT statements so table models will fail if they include ephemeral models in the SELECT. View models and other SQL statements using ephemeral models should work correctly.

Bug Fix

  • Client connections would incorrectly reuse a previous session_id when initializing, causing session locked errors. This has been fixed. This closes ClickHouse#127. Multiple threads should now work correctly in dbt projects, and dbt-clickhouse automated tests now use threads=4 (soon to be the dbt default).

Release 1.3.2, 2022-12-23

Improvements

  • Added experimental support for the delete+insert incremental strategy. In most cases this strategy should be significantly faster than the existing "legacy" custom strategy (which is still the default). To enable delete+insert incremental materialization, the flag allow_experimental_lightweight_delete must be enabled on your ClickHouse server. This flag is NOT currently considered production ready, so use at your own risk. To use this strategy as the "default" incremental strategy, the new configuration value use_lw_deletes must be set to True. Otherwise, to use it for a particular model, set the model's incremental_strategy to delete+insert. Important caveats about this strategy:
    • This strategy directly uses lightweight deletes on the target model/table. It does not create a temporary or intermediate table. Accordingly, if there is an issue during that transformation, the materialization can not be rolled back and the model may contain inconsistent data.
    • If the incremental materialization includes schema changes, or lightweight deletes are not available, dbt-clickhouse will fall back to the much slower "legacy" strategy.
  • Allow append as an incremental strategy. This is the same as using the custom model configuration value inserts_only.
  • New s3source macro. This simplifies the process of using the ClickHouse s3 table function in queries. See the tests for example usage.

Bug Fixes

  • The ON CLUSTER clause has been added to additional DDL statements including incremental models processing. Closes ClickHouse#117 and should close ClickHouse#95 for Replicated tables that use the {uuid} macro in the path to avoid name conflicts. Thanks to Saurabh Bikram
  • The apply and revoke grants macros now correctly work with roles as well as user. Again thanks to Saurabh Bikram
  • A compound unique_key (such as key1, key2) now works correctly with incremental models

Release 1.3.1, 2022-11-17

Improvements

  • Improve error message when atomic "EXCHANGE TABLES" operation is not supported

Release 1.3.0, 2022-10-30

Improvements

  • Support dbt 1.3.0 ClickHouse#105
    • Adds additional dbt 1.3.0 core tests
    • Adds array utility macros ported from dbt-utils
    • Does NOT add support for Python models (not implemented)
    • Does NOT utilize default/standard incremental materialization macros (standard strategies do not work in ClickHouse)

Bug Fix

  • Require exact match for relations. ClickHouse databases and tables are all case-sensitive, so all searches are now case-sensitive. Closes ClickHouse#100 and ClickHouse#110



Release 1.2.1, 2022-09-19

Improvements

  • Support dbt 1.2.1 ClickHouse#79
    • Grants support on models
    • Support the cross database dbt-core macros (migrated from dbt-utils)
    • Add docs tests
    • Validate Python 3.10 support
    • Implement retry logic for small set of "retryable" connection errors
  • Don't close connection on release to improvement model performance
  • Allow specifying database engine for schema creation
  • Extend support for ClickHouse Cloud and Replicated databases

Bug Fix

  • Validate that atomic EXCHANGE TABLES is supported on the target ClickHouse server ClickHouse#94



Release 1.1.8, 2022-09-02

Improvement

  • Support use of atomic EXCHANGE TABLES when supported by ClickHouse



Release 1.1.7, 2022-07-11

Improvement

  • Support ClickHouse Arrays with Nullable value types

Bug Fix

  • Fix brackets regex in columns)

Release 1.1.6, 2022-07-04

Improvement

  • Add support for CREATE AS SELECT in Replicated Databases



Release 1.1.5, 2022-06-27

Bug Fixes

  • Ensure database exists before connection
  • Safely handle dropping working database



Release 1.1.4, 2022-06-27

Improvement

  • Allow selection of either native protocol driver (clickhouse-driver) or http driver (clickhouse-connect)



Release 1.1.2, 2022-06-22

Rename from v 1.1.0.2



Release 1.1.0.2, 2022-06-17

Improvements

  • Support for inserting SETTINGS section to CREATE AS and INSERT INTO queries through model configuration
  • Support adding custom session settings to the connection through profile credentials.
  • Allow using custom databases/schemas for connection
  • Support Boolean fields in CSV seeds

Bug Fix

  • Fixed prehooks and empty fields in CSV seeds



Release 1.1.0, 2022-06-02

Release 1.1.0.1, 2022-06-09

Bug Fix

  • Fixed prehooks and empty fields in CSV seeds



Release 1.1.0, 2022-06-02

Improvements

  • 1.1.0 dbt support
  • Snapshots timestamps
  • Replace temporary in-memory tables with MergeTree table - removing memory limitations over Incremental model and snapshots creation
  • Moved to use clickhouse-connect driver - an officially supported HTTP driver by ClickHouse Inc.

Release 1.0.4, 2022-04-02

Improvements

  • 1.0.4 dbt support
  • New logger

Release 1.0.1 - 2022-02-09

Improvement

  • Support 1.0.1 dbt

Bug Fixes

Release 1.0.0, 2022-01-02

Add

  • dbt 1.0.0 support

Release 0.21.1, 2022-01-01

Improvements

  • dbt 0.21.1 support
  • Extended settings for clickhouse-driver ClickHouse#27

Bug Fixes

Release 0.21.0, 2021-11-18

Improvement

  • Support 0.21.0 dbt

Bug Fixes

Release 0.20.2, 2021-10-16

Improvement

  • dbt 0.20.1 support

Improvement

Bug Fixes

Release 0.20.1, 2021-08-15

Improvement

  • dbt 0.20.1 support

Release 0.20.0, 2021-08-14

Improvement

  • dbt 0.20.0 support

Release 0.19.1.1, 2021-08-13

Improvement

  • Add verify and secure to connection configuration

Bug Fix

Release 0.19.1, 2021-05-07

Improvement

  • Add support the ON CLUSTER clause for main cases

Usage Change

  • Engine now require brackets ()

Bug Fix

  • Fix a missing sample profile

Release 0.19.0.2, 2021-04-03

Bug Fix

  • Fix name of partition

Release 0.19.0.1, 2021-03-30

Bug Fix

  • Fix version of clickhouse-driver in setup.py

Release 0.19.0, 2021-02-14

Initial Release