Incremental macros missing from packaged dbt-spark==0.19.0

[highfly22]

Describe the bug

When a incremental model is running, the following error is reported

  Runtime Error
    Database Error
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'my_table__dbt_tmp' not found in database 'FooSchema';

For example, the schema is configured as "FooSchema". The temporary view is created as "my_table__dbt_tmp". The describe command tries to find 'FooSchema.my_table__dbt_tmp', which doesn't exists.

    create temporary view my_table__dbt_tmp as
    
    describe extended FooSchema.my_table__dbt_tmp

Steps To Reproduce

1. configure spark profile
2. create a incremental model
3. invoke dbt run -m my_table twice

Expected behavior

No error is reported.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

The output of dbt --version:

installed version: 0.19.0
   latest version: 0.19.0

Up to date!

Plugins:
  - spark: 0.19.0
  - postgres: 0.19.0
  - redshift: 0.19.0
  - snowflake: 0.19.0
  - bigquery: 0.19.0

Python 3.6.3

The operating system you're using:

The output of python --version:
Python 3.6.3

Additional context

    dev:
      type: spark
      method: thrift
      schema: FooSchema
      host: localhost
      port: 10000
      user: Foo
      auth: NONE

Thrift server is created by spark 2.4.7

[jtcohen6]

Hey @highfly22, thanks for the issue, I was just about to open this!

This is actually a problem with the distribution of dbt-spark==0.19.0 that we're hoping to fix very soon. In particular, this line in package_data:
https://github.com/fishtown-analytics/dbt-spark/blob/46b2881656ff9028d8850a9f3bda23b51986088f/setup.py#L59
Failed to glob for the three files in include/spark/macros/materializations/incremental/*.sql.

Because those files are missing from the packaged plugin, dbt instead falls back to the default incremental materialization, which doesn't work on Spark and raises the error you're seeing here.

We're going to fix this (#151) and ship dbt-spark==v0.19.0.1 ASAP. In the meantime, you can either:

• Downgrade to dbt-spark==0.18.1
• Copy the three incremental macro files into your own project's macros/ folder, as dbt will prioritize your versions

[kwigley]

closed by #151!

[kwigley]

Fix is included in 0.19.0.1 which is now available on PyPi