[CT-594] Support for Iceberg sort order

[cccs-jc]

When creating iceberg tables you can specify a sort order. This is important because a table that is partition will implicitly be sorted by iceberg (overriding the sort statement in your query).

Sort order are specified via ALTER TABLE statement. In an iceberg implementation this feature should be configurable via a dbt-config.

Reference: https://iceberg.apache.org/docs/latest/spark-ddl/

Example statement:

ALTER TABLE prod.db.sample WRITE ORDERED BY category ASC NULLS LAST, id DESC NULLS FIRST

[jtcohen6]

Hey @cccs-jc, thanks for opening!

Sure, this sounds like a reasonable config option within the table materialization. It also sounds like something you could get working with a post_hook in the meantime.

Following the thread in #294: dbt-spark doesn't (yet) officially support Iceberg, but it seems like this is something you've managed to get working over in https://github.com/cccs-jc/dbt-spark. There are a few moving pieces in play:

• Adding support for three-part catalogs (How to support SQL Standard Catalogs? #281) — a requirement for Iceberg as well
• Reworking caching ([CT-202] Workaround for some limitations due to list_relations_without_caching method #228) + cataloging ([CT-385] Limit catalog generation to specific relations #300) queries, to move away from show tables extended like '*', which is unbearably slow at scale — which has implications for the is_iceberg checks I see in your fork

Perhaps it makes sense to wait on adding support for this specific config, until we have the right pieces in place to officially support Iceberg?

[cccs-jc]

I agree. I mostly wanted to create the issue so we don't loose track of it. It's my understanding that the guys at Iceberg plan to contribute support for Iceberg in dbt-spark. I've pointed them to my POC which does include some tidbits but is not complete or production grade.

For now yes we will use a pre-hook so that when we insert the table is always configured correctly.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.