Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix spark dbt project path #9

Merged
merged 1 commit into from
Apr 11, 2019
Merged

fix spark dbt project path #9

merged 1 commit into from
Apr 11, 2019

Conversation

drewbanin
Copy link
Contributor

Closes #8 by changing the setup.py package_data path to match the actual path of the Spark dbt project.

@drewbanin drewbanin requested a review from cmcarthur April 11, 2019 13:04
@drewbanin drewbanin merged commit 30f9948 into master Apr 11, 2019
@drewbanin drewbanin deleted the fix/spark-project-path branch April 11, 2019 14:15
Fokko pushed a commit to Fokko/dbt-spark that referenced this pull request Dec 20, 2022
@dparent1
Merge pull request dbt-labs#9 from Fokko/fd-fix-incremental-runs
294b7fb 
Fix incremental runs
Fokko added a commit to Fokko/dbt-spark that referenced this pull request Feb 2, 2023
@Fokko
Squashed commit of the following:
b727a9e 
commit 8e1a8f4
Merge: 28909f2 908d9fe
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Thu Feb 2 13:22:23 2023 -0500

    Merge pull request dbt-labs#16 from Fokko/fd-revert-some-changes

    Revert some stuff

commit 908d9fe
Merge: b286fd0 f877d1e
Author: Fokko Driesprong <fokko@tabular.io>
Date:   Thu Feb 2 10:00:18 2023 -0800

    Merge branch 'main' of https://github.com/dbt-labs/dbt-spark into fd-revert-some-changes

commit b286fd0
Author: Fokko Driesprong <fokko@tabular.io>
Date:   Wed Feb 1 12:00:21 2023 -0800

    Revert some stuff

    I noticed that two Spark tests are failing, so tried to revert
    some of the changes that seemed related. They are now passing
    and also my Iceberg dbt project is still running fine

commit 28909f2
Merge: 3ba0a72 3c03d9a
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Mon Jan 30 15:05:00 2023 -0500

    Merge branch 'main' into dparent1/iceberg

    - in impl.py changed:
        List[agate.Row]
        to
        AttrDict

    This fixed the pre-commit error after merging in changes from main.

commit 3c03d9a
Merge: 10c9dac 4d179e0
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Mon Jan 30 14:31:18 2023 -0500

    Merge branch 'dbt-labs:main' into main

commit 3ba0a72
Merge: 83f2d61 10c9dac
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Mon Jan 23 16:39:22 2023 -0500

    Merge branch 'main' into dparent1/iceberg

    - Fixing up merge conflicts around Exception/Error types
    - Putting back accidental newline removal in tox.ini

commit 10c9dac
Merge: 3a7ca7c 39800e0
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Mon Jan 23 15:25:29 2023 -0500

    Merge branch 'dbt-labs:main' into main

commit 83f2d61
Merge: beac4b4 3a7ca7c
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Mon Jan 9 11:12:28 2023 -0500

    Merge branch 'main' into dparent1/iceberg

commit 3a7ca7c
Merge: cad11e2 d9a3d76
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Mon Jan 9 11:02:55 2023 -0500

    Merge branch 'dbt-labs:main' into main

commit beac4b4
Merge: 29610a5 6e6daf5
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Tue Dec 20 16:24:54 2022 -0500

    Merge pull request dbt-labs#12 from Fokko/fd-comments

    Cleanup based on comments

commit 6e6daf5
Author: Fokko Driesprong <fokko@tabular.io>
Date:   Tue Dec 20 21:49:01 2022 +0100

    Cleanup based on comments

commit 29610a5
Merge: 34c5549 cad11e2
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Tue Dec 20 13:26:20 2022 -0500

    Merge branch 'main' into dparent1/iceberg

    Fixed conflicts:
    dbt/adapters/spark/impl.py
    dbt/adapters/spark/relation.py
    dbt/include/spark/macros/materializations/incremental/incremental.sql
    dbt/include/spark/macros/materializations/incremental/strategies.sql

    Updated tox.ini removing my own addition of allowlist_externals, main
    has it set now.

commit cad11e2
Merge: 37307d1 512b3d0
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Tue Dec 20 12:50:27 2022 -0500

    Merge branch 'dbt-labs:main' into main

commit 34c5549
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Thu Dec 8 10:47:25 2022 -0500

    Allowing the use of /bin/bash by tox

    tox on certain platforms will complain that /bin/bash is not allowed to
    be used.  I'm allowing it to be used with this change.

commit 3ffcf5b
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Thu Dec 8 10:36:57 2022 -0500

    Removed use of ParsedSourceDefinition, add iceberg

    - upstream dbt changed breaking the use of ParsedSourceDefinition, using
      SourceDefinition appears to work instead
    - Added in change to include iceberg in adapters.sql:
       macro spark__alter_column_comment

commit edb6f01
Merge: 65eac08 37307d1
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Thu Dec 8 10:02:36 2022 -0500

    Merge changes in main to bring branch up to date

commit 37307d1
Merge: d482d66 9511847
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Thu Dec 8 09:22:16 2022 -0500

    Merge branch 'dbt-labs:main' into main

commit 65eac08
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Thu Dec 8 09:05:48 2022 -0500

    Backing out previous merge which broke unit tests

commit 513293b
Merge: 294b7fb b936b7a
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Thu Dec 8 08:55:36 2022 -0500

    Merge pull request dbt-labs#10 from Fokko/fd-enable-docs

    Add Iceberg to the list

commit b936b7a
Author: Fokko Driesprong <fokko@tabular.io>
Date:   Wed Dec 7 19:04:24 2022 +0100

    Add Iceberg to the list

commit 294b7fb
Merge: 8d3984f d282b21
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Mon Nov 28 09:55:51 2022 -0500

    Merge pull request dbt-labs#9 from Fokko/fd-fix-incremental-runs

    Fix incremental runs

commit d282b21
Author: Fokko Driesprong <fokko@tabular.io>
Date:   Fri Nov 25 15:27:14 2022 +0100

    Fix incremental runs

commit d482d66
Author: dparent1 <65294587+dparent1@users.noreply.github.com>
Date:   Mon Nov 21 16:05:10 2022 -0500

    Set up CI with Azure Pipelines

    [skip ci]

commit 8d3984f
Merge: 93d6b97 b759267
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Mon Nov 7 14:35:22 2022 -0500

    Merge branch 'main' into dparent1/iceberg

commit 93d6b97
Merge: 402be01 7f233b1
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Thu Sep 29 09:58:24 2022 -0400

    Merge branch 'main' into dparent1/iceberg

commit 402be01
Merge: 50f7b94 80dc029
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Wed Sep 28 09:02:05 2022 -0400

    Merge branch 'main' into dparent1/iceberg

commit 50f7b94
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Tue Sep 27 14:18:53 2022 -0400

    Removing the is_iceberg check it is not needed

    Upon further investigation this check is not needed since
    self.database will not be set.

commit bbf79e4
Merge: 6764e55 ebd011e
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Mon Sep 12 11:21:35 2022 -0400

    Merge branch 'main' into dparent1/iceberg

commit 6764e55
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Fri Aug 19 14:13:58 2022 -0400

    Adding changelog entry.

commit ee8c7b0
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Fri Aug 19 13:45:24 2022 -0400

    Renaming macro and method name

    On second look I wasn't happy with my name choices for macro name and
    method, hopefully what I have now makes more sense.

    [CT-276] Apache Iceberg Support dbt-labs#294

commit 6f96705
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Fri Aug 19 10:09:02 2022 -0400

    Setting _schema rather than replacing it

    [CT-276] Apache Iceberg Support dbt-labs#294

    The _schema variable was used for non-iceberg tables but was being
    overridden by work for iceberg v2 tables.  I've made it so the iceberg
    condition will set _schema rather than blanket changing the schema for
    all providers.

commit 35bc12e
Author: Dan Parent <65294587+dparent1@users.noreply.github.com>
Date:   Tue Aug 16 12:51:53 2022 -0400

    Adding in additional support for iceberg v2 tables

    Found a way to identify iceberg tables given that spark returns
    an error when trying to execute "SHOW TABLE EXTENDED..."  See
    https://issues.apache.org/jira/browse/SPARK-33393

    Instead of show table extended a "DESCRIBE EXTENDED" is
    performed to retrieve the provider information.  This allows
    for identification of iceberg through an is_iceberg member
    variable.

    Allow for multiple join conditions to allow for mutliple columns to
    make a row distinct

    Use is_iceberg everywhere handling iceberg tables differs from other
    sources of data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cmcarthur cmcarthur cmcarthur approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@drewbanin @cmcarthur