Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable BigQuery CDC configuration for Python BigQuery sink #32529

Merged
merged 40 commits into from
Oct 15, 2024
Merged

Enable BigQuery CDC configuration for Python BigQuery sink #32529

merged 40 commits into from
Oct 15, 2024

Commits on Sep 20, 2024

  1. include CDC configuration on the storage write transform provider

    @prodriguezdefino
    prodriguezdefino committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    745cd58 View commit details
    Browse the repository at this point in the history

  2. adding the primary key configuration for CDC and tests

    @prodriguezdefino
    prodriguezdefino committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    504d2a4 View commit details
    Browse the repository at this point in the history

  3. fixing List.of references to use ImmutableList

    @prodriguezdefino
    prodriguezdefino committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    a2eabf9 View commit details
    Browse the repository at this point in the history

  4. fixing test, missing calling the cdc info row builder() method

    @prodriguezdefino
    prodriguezdefino committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    01703c1 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. fix test, add config validations

    @prodriguezdefino
    prodriguezdefino committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    51360d5 View commit details
    Browse the repository at this point in the history

  2. added the xlang params to storage write python wrapper

    @prodriguezdefino
    prodriguezdefino committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    542a49e View commit details
    Browse the repository at this point in the history

  3. adding missing comma

    @prodriguezdefino
    prodriguezdefino committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    fe36fe8 View commit details
    Browse the repository at this point in the history

  4. shortening property name

    @prodriguezdefino
    prodriguezdefino committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    1004f91 View commit details
    Browse the repository at this point in the history

  5. changing xlang config property

    @prodriguezdefino
    prodriguezdefino committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    de9e948 View commit details
    Browse the repository at this point in the history

  6. set use cdc schema property as nullable, added safe retrieval method

    @prodriguezdefino
    prodriguezdefino committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    ab40dd9 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2024

  1. fixes property name reference and argument type definition

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    27e5634 View commit details
    Browse the repository at this point in the history

  2. python format fix

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    e804618 View commit details
    Browse the repository at this point in the history

  3. adding xlang IT with BQ

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    7b7255b View commit details
    Browse the repository at this point in the history

  4. adding missing primary key column to test

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    2b901eb View commit details
    Browse the repository at this point in the history

  5. python format fix

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    bb16979 View commit details
    Browse the repository at this point in the history

  6. format xlang test

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    12dce1b View commit details
    Browse the repository at this point in the history

  7. more format xlang test fixes

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    27a86db View commit details
    Browse the repository at this point in the history

  8. and more format xlang test fixes

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    e5766e7 View commit details
    Browse the repository at this point in the history

  9. adding missing import

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    09f298a View commit details
    Browse the repository at this point in the history

  10. missing self reference

    @prodriguezdefino
    prodriguezdefino committed Sep 22, 2024
    Configuration menu
    Copy the full SHA
    e972c89 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. enabled create if needed functionality for CDC python integration, im… 

    …plemented table constraint support on the bigquery fake dataset services
    @prodriguezdefino
    prodriguezdefino committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    ec3373b View commit details
    Browse the repository at this point in the history

  2. Update bigquery.py

    @prodriguezdefino
    prodriguezdefino authored Sep 24, 2024
    Configuration menu
    Copy the full SHA
    23ea4c3 View commit details
    Browse the repository at this point in the history

  3. triggering the xlang tests

    @prodriguezdefino
    prodriguezdefino committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    143566a View commit details
    Browse the repository at this point in the history

  4. fixing lint

    @prodriguezdefino
    prodriguezdefino committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    4e390c7 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. addressing few comments

    @prodriguezdefino
    prodriguezdefino committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    51bae40 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. cdc info is added after row transformation now

    @prodriguezdefino
    prodriguezdefino committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    6331691 View commit details
    Browse the repository at this point in the history

  2. remove not used param

    @prodriguezdefino
    prodriguezdefino committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    56fe2b6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into python_xlang_cdc_bigquery

    @prodriguezdefino
    prodriguezdefino committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    ce85c88 View commit details
    Browse the repository at this point in the history

  4. removed typing information for callable

    @prodriguezdefino
    prodriguezdefino committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    bd324c5 View commit details
    Browse the repository at this point in the history

  5. adding test for cdc using dicts as input and cdc write callable

    @prodriguezdefino
    prodriguezdefino committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    3e5e31c View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. simplifying the xlang configuration from python perspective, will add… 

    … callable on a future PR
    @prodriguezdefino
    prodriguezdefino committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    0073348 View commit details
    Browse the repository at this point in the history

  2. spotless apply

    @prodriguezdefino
    prodriguezdefino committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    9dd4cfa View commit details
    Browse the repository at this point in the history

  3. wrong property passed to xlang builder

    @prodriguezdefino
    prodriguezdefino committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    4208517 View commit details
    Browse the repository at this point in the history

  4. missing self

    @prodriguezdefino
    prodriguezdefino committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    cd7b6cd View commit details
    Browse the repository at this point in the history

  5. fixing xlang it

    @prodriguezdefino
    prodriguezdefino committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    72d1bf4 View commit details
    Browse the repository at this point in the history

  6. fixes wrong property reference

    @prodriguezdefino
    prodriguezdefino committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    1487acc View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. change cdc xlang test to use beam.io.WriteToBigQuery

    @prodriguezdefino
    prodriguezdefino committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    96882c5 View commit details
    Browse the repository at this point in the history

  2. force another build

    @prodriguezdefino
    prodriguezdefino committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    538ef3c View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. modifying comment to trigger build.

    @prodriguezdefino
    prodriguezdefino committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    1774f8f View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. addressing PR comments, included new dicts based test for xlang pytho… 

    …n tests, included the CDC configurations into the existing RowDynamicDestinations object, improved error message for mutation information schema checks.
    @prodriguezdefino
    prodriguezdefino committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    1d99fc6 View commit details
    Browse the repository at this point in the history