Delta Lake Support Roadmap

[kevinzwang]

Features

• Basic reads
• Partitioned reads
• Basic writes
• [Catalogs] [Delta Lake] Add support for reading tables with deletion vectors #1954
  • Blocked by delta-rs
• Utilizing statistics from table metadata for reads
• [PERF] [Catalogs] Push partition filter expressions into catalog file URI fetches #1953
• [Catalogs] [Delta Lake] Add support for reading tables with column mappings #1955
• Partitioned writes
• Upserts/MERGE
  • Copy-on-write
  • Deletion vector writing
    • Blocked by delta-rs
• Add support for writing new DeltaLake tables to catalogs such as AWS Glue and Unity Catalog

Bugs and Testing

• Reading only the partition column from an Iceberg/Delta table fails #2129
• Add end-to-end tests for reading+writing #2348

[jaychia]

(After offline discussion with delta-rs team) other things we might want on the roadmap that might be useful to support:

1. Variant type support
2. Row IDs

[keen85]

Hi, are there no plans to support MERGE for Delta Tables?

[kevinzwang]

Hi, are there no plans to support MERGE for Delta Tables?

Hi @keen85 , the roadmap does contain two types of upsert operations we would like to support! Copy-on-write and deletion vectors. Once either of these are implemented, Daft will be able to do MERGE and DELETE operations

[jaychia]

@kevinzwang should we add it into the tasks for this roadmap issue?

[kevinzwang]

Updated it to be more clear