Databricks Unity Catalog Support

[dhirschfeld]

I'm opening this to discuss the state of Unity Catalog support in delta-rs.

In my ideal world I'd be able get a DeltaTable object from Unity Catalog, with permissions controlled by UC access policies.

[dhirschfeld]

xref:

• fix: remove the get_data_catalog() function #1941

[dhirschfeld]

Extract the session credentials from your cluster

• https://kb.databricks.com/en_US/notebooks/access-s3-temp-session-token

[rtyler]

I have had a number of discussions with Databricks on the topic. The 🦀 code has support for table lookup but it's kind of useless IMHO without credential vending support from the Unity APIs. Basically a user today would need to have an access token for Unity and their own separate storage credentials for accessing the table in object storage.

What is missing is a public API from Databricks which would allow the actor running delta-rs to retrieve credentials for use in accessing storage.

So yes, I would like to see this too, but I'm marking this on-hold until Databricks provides us the necessary REST APIs for Unity Catalog

[dhirschfeld]

Thanks for the info @rtyler. I'll continue to monitor this thread in the hope that something changes on the Databricks side...

[wmoreiraa]

Hi sir, @rtyler. Did Databricks gave any updates on the last months on this topic? I think we'all on databricks ecosystem are waiting for this.

[nicornk]

https://github.com/duckdb/uc_catalog/blob/4a2ba143eb8686becf69f759caa731742c2d7c81/src/uc_api.cpp#L120

There seems to be an API now to get temporary storage credentials?

[ion-elgreco]

Those APIs aren't public yet if you check the docs. But it's promising

[ion-elgreco]

Actually they seems already to be working : ) I'll start working on it soon ^^

[nicornk]

I also found zero google results on the APIs, just wanted to share this here.

[ion-elgreco]

No luck:

{'error_code': 'UNAUTHENTICATED',
 'message': "Request to generate access credential for table '<redacted>' from outside of Databricks Unity Catalog enabled compute environment is denied for security. Please contact Databricks support for integrations with Unity Catalog.",
 'details': [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo',
   'reason': 'UNITY_CATALOG_EXTERNAL_GENERATE_TABLE_CREDENTIALS_DENIED',
   'domain': 'unity-catalog.databricks.com',
   'metadata': {'tableId': '<redacted>'}},
  {'@type': 'type.googleapis.com/google.rpc.RequestInfo',
   'request_id': '<redacted>',
   'serving_data': ''}]}

[abhiaagarwal]

Unity Catalog open-source is available:
https://github.com/unitycatalog/unitycatalog/

[ion-elgreco]

Yup, but we likely only can support the OSS version and not databricks unity

[abhiaagarwal]

I didn't realize there was a difference :(

[dhirschfeld]

An interesting blog-post from DuckDB:

• https://duckdb.org/2024/06/10/delta.html

They're working with Databricks to implement native Delta Lake support in DuckDB.

It seems very cool but the examples are all using direct access to a local filesystem or S3. IIUC DuckDB still wouldn't have access to Unity Catalog managed tables.

[ion-elgreco]

@abhiaagarwal see this comment: #2066 (reply in thread)

I already gave the new API credentials fetching api a go, but it didn't allow me to fetch them

[roeap]

since unity is now oss and should have some features that were holding us back in adpoting this, i am looking to soon spend some time on reviving our integration - unless someone else is faster :)

[roeap]

they are the same. the reason we may be unblocked is more realted to some features that were associated with teh release as well.

[roeap]

the api surface is nove much clearer (and better documented) and fully accessible to clients.

[ion-elgreco]

They are the same indeed, but the main thing we need is fetching credentials, you can see a couple posts above that I already tried this, but it wasn't working unfortunately.

[roeap]

I'll try anyhow 😀

[rahulnbhandari]

any luck did it work @roeap

[abhiaagarwal]

They mentioned on the UC community call that DBX will be rolling out remote support soon and it's already in private feature preview, our long national nightmare is over

[ion-elgreco]

DBX?

[abhiaagarwal]

Databricks haha, I think that's a pretty common abbreviation for them!

[ion-elgreco]

Lol, first time seeing that

What do you mean with remote support though? Afaik, credential fetching is still blocked even with the new API

[abhiaagarwal]

They mentioned the credential feature will be implemented

[ion-elgreco]

They mentioned the credential feature will be implemented

Let's see, I wouldn't bet my money on it being the thing we want it to be ^^

[dhirschfeld]

Does the existence of duckdb/uc_catalog imply that it's possible to access a table in UC with just a Databricks token (without already having direct access to the underlying storage)?

[jordandakota]

https://github.com/unitycatalog/unitycatalog/releases/tag/v0.2.0

OSS UC now supports identity and credential vending. I'm assuming this will come to dbx UC soon. Additionally, there's some docs pages that show the api:

https://www.getdaft.io/projects/docs/en/latest/user_guide/integrations/unity-catalog.html

https://docs.unitycatalog.io/integrations/unity-catalog-daft/

https://github.com/spiceai/quickstarts/blob/trunk/catalogs%2Fdatabricks%2FREADME.md

Credential Vending
Expanded support for credential vending APIs
Added support for vending S3, Azure and GCS credentials
Added support for vending credentials for models and paths
Credentials are now temporary and scoped down to give clients access to only the asset directory

[yefetBenTili]

@ion-elgreco @dhirschfeld
Any update on this?
https://docs.databricks.com/en/data-governance/unity-catalog/access-open-api.html
With all the limitations, there is a glimmer of hope

[ion-elgreco]

I'm not working on this

[dhirschfeld]

It sounds like @tunayokumus might be looking into it:

• Can't read a Delta table from Azure Unity Catalog #1628 (comment)

[jordandakota]

Daft addressed the same issue, can see here

Eventual-Inc/Daft#2903

Eventual-Inc/Daft#3142