[CT-276] Apache Iceberg Support

[cccs-jc]

Any plans to support iceberg table format. In preliminary testing it seems to work but I wonder if there are particularities which has not been addressed.

Also Iceberg uses catalogs (database) which the current dbt-spark implementation does not support. https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs

Adding support for catalogs (database) in dbt-spark would be required to properly support iceberg tables.

[jtcohen6]

It sounds like the three-part namespacing, with catalogs, may be coming to the Apache Spark standard as well: #294

dbt supports three-part namespaces by default, so it's actually much simpler from the standpoint of plugin code. What might be tricky is ongoing conditional support for both 2- or 3-part namespacing, configurable depending on the cluster / file format you're using

[cccs-jc]

okay so what I understand is that there is no show stoppers with supporting 3-part namespacing. But the adapter should be able to work in either mode based on table format and sql engine.

I mostly wanted to know if there would be any objections in adding such support.

[kbendick]

Hi all! I work on Iceberg project and recently came across this issue. I also work very closely with many contributors to the Spark project and occasionally contribute there as well.

I'm relatively green to dbt admittedly, but having dbt work properly with Iceberg is important to us, as dbt continues to be a more and more common part of the standard data stack.

I can confirm that Spark 3 makes heavy use of catalogs / multipart namespacing and is by all intents "the future" (and really the present) of Spark. If no catalog is declared, spark 3.x uses the implicit spark_catalog (which is a SessionCatalog). Happy to answer more questions about it as well.

I'm going to play around over the weekend with dbt a bit and will follow up after that. 👍

[cccs-jc]

Hey @kbendick I've updated my repo with iceberg support. It's incomplete work but feel free to take all the iceberg specific parts. It's in the main branch here https://github.com/cccs-jc/dbt-spark

I will focus on support for a pyspark driver means of interfacing with spark. #305

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[kbendick]

Hi all! I have been out of office for health reasons but I might have some cycles now to look at this. Is the linked PR still relevant or is there some area specifically I can help out in?

[stevenayers]

Hi all, is there any update on this?

[VersusFacit]

Twas merged, to be sent out in the beta, and thus, at last, we may close this.

[akshay-kokate-06]

Does this also support the incremental models ??

[Fokko]

@akshay-kokate-06 it does :)

[QuangVuMinh]

How can i config catalog for iceberg table ? catalog_type(hadoop, hive), catalog_name, ...

[Fokko]

You need to set up a catalog through the spark configuration and then set the default catalog to the newly created catalog. An example through the spark-default.conf can be found here: https://github.com/tabular-io/docker-spark-iceberg/blob/main/spark/spark-defaults.conf You can also set it using startup arguments, as shown in https://tabular.io/blog/dbt/

Once #781 has been merged, you can also set it when configuring the session:

dbt_tabular:
  outputs:
    dev:
      method: session
      schema: dbt_tabular
      type: spark
      host: NA
      server_side_parameters:
        "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
        "spark.sql.defaultCatalog": "sandbox"
        "spark.sql.catalog.sandbox": "org.apache.iceberg.spark.SparkCatalog"
        "spark.sql.catalog.sandbox.catalog-impl": "org.apache.iceberg.rest.RESTCatalog"
        "spark.sql.catalog.sandbox.uri": "https://api.tabular.io/ws/"
        "spark.sql.catalog.sandbox.credential": "abc"
        "spark.sql.catalog.sandbox.warehouse": "sandbox"

Also, once #755 is in, we can also select different catalogs.