The road to deltalake v1.0.0 🦀

[rtyler]

I have been thinking about what API stability and what a celebration-worthy 1.0 release for the deltalake crate would entail. I think we're making great progress and have largely kept strong APIs over the past year or so. When I think about 1.0, the things I would like to see (and maybe implement) would be:

High level features:

• Change Data Feed support Support CDC #604 feat: introduce CDC write-side support for the Update operations #2486
• Constraints feat: check constraints #1915
• Schema evolution for all Rust/Python writers Schema evolution on upsert (merge) #2282
• Deletion vector support Support deletion vector #1094
• Rust-based writer engine for Python by default milestone #10
• Stablized API for DataFusion integration
• Stabilized API for configuration via environment variables for object store

High level bug fixes:

• Full timestamp support (INT96/Timestamp/Timestamp with Timezone), currently a persistent thorn in the side of multi-reader/writer setups.
• Adoption of AWS supported Rust SDKs for dependencies.

Depending on progress made by some projects driven by our friends at Databricks, I think it would also be exciting to incluhde:

• Preliminary delta-kernel-rs adoption, which would hopefully add deletion vector support. feat: adopt kernel schema types #2495
• Liquid clustering support, pending completion of the design doc. (Currently as written it just looks like it's a minimal scope ZORDER operation executed on writes, which we have the pieces for already.)
• Unity catalog integration pending credential vending and Unity supporting the APIs necessary for 3rd parties to interoperate.

I think that's "close enough" to be impement-able by 1.0 around Summit.

I wanted to share this with everybody to get a sense of what you think we need for a 1.0 release?

[ion-elgreco]

I only have one remark, if I remember correctly, Will mentioned that deletion vectors would require a new kind of reader on the python side. So am I not sure if that should be part of v1.0.0, since it could potentially take some time. Maybe deletion vectors can be v1.1

Also I started looking into the timestamp tz issue, it's not a huge change but it will have a breaking effect on current writers, so we should definitely do that sooner than later.

[rtyler]

@ion-elgreco My suggestion here is for the Rust library. I don't think we should hold Deletion Vectors back from the Rust implementation on account of more work being needed by the Python library 😄

I think that's a good call out for us to keep in mind with reader and writer changes in 🦀 since there are implications for 🐍

[jordandakota]

Is there somewhere to read about the concerns around supporting deletion vectors on the python side (obv rust needs it first)? We've found a lot of support for pulling spark out of a lot of our deltalake workloads since merge was introduced into delta-rs. With deletion vectors going enabled by default in most databricks workspaces, more external tables are being set up with deletion vectors which reduces the reach of delta-rs for us.

[rtyler]

Is there somewhere to read about the concerns around supporting deletion vectors on the python side (obv rust needs it first)? We've found a lot of support for pulling spark out of a lot of our deltalake workloads since merge was introduced into delta-rs. With deletion vectors going enabled by default in most databricks workspaces, more external tables are being set up with deletion vectors which reduces the reach of delta-rs for us.

@jordandakota There's #1094 which is kind of the placeholder for this. I know that @zachschuermann wrote some Deletion Vector support in the early days of the delta-kernel-rs work, but what is missing is the bringing of that work into delta-rs either through kernel (which @roeap is starting on #2489) or bringing the code over.

If you're open to rolling up your sleeves, that would be a great contribution to the project 😉

[yeshsriram]

Hope this #2006 makes it.

[sim-san]

Will be Column Mapping and Generated Columns feature of the writer be part the version 1.0 ?

[ion-elgreco]

@sim-san for which binding are you asking this?

[sim-san]

@ion-elgreco For the Python binding

[ion-elgreco]

There is no one actively working on column mapping or generated columns at the moment, but it will eventually come in the Python binding.

Column mapping reading though won't be possible with our python scan interface