dbt 0.19.0
dbt 0.19.0 - Kiyoshi Kuromiya (January 27, 2021)
Links:
Breaking changes
- The format for
sources.json
,run-results.json
,manifest.json
, andcatalog.json
has changed:- Each now has a common metadata dictionary (#2761, #2778). The contents include: schema and dbt versions (#2670, #2767);
invocation_id
(#2763, #2784); custom environment variables prefixed withDBT_ENV_CUSTOM_ENV_
(#2764, #2785); cli and rpc arguments in therun_results.json
(#2510, #2813). - Remove
injected_sql
from manifest nodes, usecompiled_sql
instead (#2762, #2834)
- Each now has a common metadata dictionary (#2761, #2778). The contents include: schema and dbt versions (#2670, #2767);
- Defer if and only if upstream reference does not exist in current environment namespace (#2909, #2946)
- Rationalize run result status reporting and clean up artifact schema (#2493, #2943)
- Add adapter specific query execution info to run results and source freshness results artifacts. Statement call blocks return
response
instead ofstatus
, and the adapter methodget_status
is nowget_response
(#2747, #2961)
Deprecations
- Removed support for
config-version: 1
ofdbt_project.yml
, which was deprecated in v0.17.0. Useconfig-version: 2
in all projects and installed packages.
Features
Core
- dbt will compare configurations using the un-rendered form of the config block in
dbt_project.yml
. (#2713, #2735) - Added state and defer arguments to the RPC client, matching the CLI (#2678, #2736)
- Added ability to snapshot hard-deleted records (opt-in with
invalidate_hard_deletes
config option). (#249, #2749) - Added revival for snapshotting hard-deleted records. (#2819, #2821)
- Improved error messages for YAML selectors (#2700, #2781)
- Added
--defer
flag fordbt test
as well (#2701, #2954) - Added native python
re
module for regex in jinja templates (#1755, #1755) - Store resolved node names in manifest (#2647, #2837)
- Save selectors dictionary to manifest, allow descriptions (#2693, #2866)
- Normalize cli-style-strings in manifest selectors dictionary (#2879, #2895)
Docs website
- Add select/deselect option in DAG view dropups. (docs#98, docs#138)
- Fixed issue where sources with tags were not showing up in graph viz (docs#93, docs#139)
- Use
compiled_sql
instead ofinjected_sql
for "Compiled" (docs#146, docs#148) - Fixed issue where data tests with tags were not showing up in graph viz (docs#147, docs#157)
BigQuery
- Added
dbt_invocation_id
for each BigQuery job to enable performance analysis (#2808, #2809) - Added support for BigQuery connections using refresh tokens (#2344, #2805)
- Added macro
get_partitions_metadata(table)
to return partition metadata for BigQuery partitioned tables (#2552, #2596) - Hourly, monthly and yearly partitions available in BigQuery (#2476, #2903)
- Allow BigQuery to default to the environment's default project (#2828, #2908)
Fixes
- Respect
--project-dir
indbt clean
command (#2840, #2841) - Fix Redshift adapter
get_columns_in_relation
macro to push schema filter down to thesvv_external_columns
view (#2854, #2854) - Increased the supported relation name length in postgres from 29 to 51 (#2850)
dbt list
command always return0
as exit code (#2886, #2892)- Set default
materialized
for test node configs totest
(#2806, #2902) - Allow
docs
blocks inexposure
descriptions (#2913, #2920) - Use original file path instead of absolute path as checksum for big seeds (#2927, #2939)
- Fix KeyError if deferring to a manifest with a since-deleted source, ephemeral model, or test (#2875, #2958)
- Fix regression with defining exposures and other resources with the same name (#2969, #3009)
- Remove ellipses printed while parsing (#2971, #2996)
Under the hood
- Save
manifest.json
at the same time we save therun_results.json
at the end of a run (#2765, #2799) - Added strategy-specific validation to improve the relevancy of compilation errors for the
timestamp
andcheck
snapshot strategies. ((#2787, #2791) - Changed rpc test timeouts to avoid locally run test failures (#2803,#2804)
- Added a
debug_query
on the base adapter that will allow plugin authors to create custom debug queries (#2751,#2871) - Add
unixodbc-dev
package to testing docker image (#2859) - Add event tracking for project parser/load times (#2823,#2893)
- Bump
cryptography
version to>= 3.2
and bump snowflake connector to2.3.6
(#2896, #2922) - Widen supported Google Cloud libraries dependencies (#2794, #2877).
- Bump
hologram
version to0.0.11
. Addscripts/dtr.py
(#2888,#2889) - Bump
hologram
version to0.0.12
. Add testing support for python3.9 (#2822,#2960) - Bump the version requirements for
boto3
in dbt-redshift to the upper limit1.16
to match dbt-redshift and thesnowflake-python-connector
as of version2.3.6
. (#2931, (#2963)) - Rewrite macro for snapshot_merge_sql to make compatible with other SQL dialects (#3003)
- Rewrite logic in
snapshot_check_strategy()
to make compatible with other SQL dialects (#3000, #3001) - Remove version restrictions on
botocore
(#3006) - Include
exposures
in start-of-invocation stdout summary:Found ...
(#3007, #3008) - Cleanup docker resources, use single
docker/Dockerfile
for publishing dbt as a docker image (dbt-release#3, #3019)
Contributors
Thank you all for your contributions to dbt! 🎉
- @brangisom (#2855)
- @db-magnus (#2892)
- @dbeatty10 (dbt-adapter-tests#10)
- @elexisvenator (#2850)
- @feluelle (#2841)
- @franloza (#2837)
- @hochoy (#2851)
- @joellabes (#2913)
- @joelluijmes (#2749, #2821)
- @jplynch77 (docs#139)
- @kingfink (#2791)
- @max-sixty (#2877, #2908)
- @mikaelene (#3003)
- @Mr-Nobody99 (docs#138)
- @plotneishestvo (#2896)
- @rsella (#2892)
- @stpierre (#3006)
- @swanderz (#3000)
- @tyang209 (#2931)
- @zmac12 (#2817)
- ran-eh (#2596)
Kiyoshi Kuromiya (1943–2000)
Thanks to @amychen1776 for writing this biography of Kiyoshi Kuromiya―a famous Philadelphian, and the namesake of this dbt release.
To boil down all of Kiyoshi Kuromiya's accomplishments to a few sentences would be an act not unlike trying to boil down the Pacific Ocean to mere salt. There will be many remnants missing but you will at least have a glimpse at how he has shaped the world.
Born at a Japanese American internment camp in Wyoming, Kiyoshi's life began in conflict. He would go on to live a life centered around being a voice for the marginalized. His intersectional activism span from racial injustice to LGBTQA rights as well as protesting war. Though not born and raised in West Philadelphia, Kiyoshi became a Philadelphian in 1961 when he started at the University of Pennsylvania. There, he began to explore his multi-faceted identities as well as progress his activism.
One of the ways he attracted attention to his causes was theatrics. He once staged an event on the library steps at Penn, threatening to burn a dog alive. When thousands turn to protest, he flipped the narrative by delivering the message: "Congratulations on your anti-napalm protest. You saved the life of a dog. Now, how about saving the lives of tens of thousands of people in Vietnam?"
Kiyoshi was also one of the few Asian Americans who openly supported the Civil Rights Movement. He was a close confidant to Dr. Martin Luther King Jr, to the point of taking care of his children in the days after King's assassination. As an openly gay delegate to the Black Panther Convention, he played a part in the organization's recognization of the connections between racial injustice and the Gay Rights Movement. He was also a founding member of the Gay Liberation Front - Philadelphia group.
Outside of his involvement in the civil rights movement, Kiyoshi was widely known for his work for the AIDS community. He was actively involved in several AIDS organizations in Philadelphia, including being the editor of the ACT-UP Standard of Care, the first such publication for the care of people living with HIV. He was a treatment activist and clinical trials participant, fighting for research that involved the community in its design. His advocacy for people of color, drug users, women, and incarcerated individuals would ensure hundreds of folks had access to up-to-date treatment information and resources.
He passed away in 2000 due to AIDS-related complications but his legacy to social justice will forever live on. ✊🏿