write_deltalake - Support azure managed identities

[rbushri]

Description

Since object_store added support to azure managed identities in the next release see: apache/arrow-rs#3581 and this feature will release in object_store_0.5.4 version, it'll be nice if delta-rs will support Azure managed identities as well.

Use Case
Write delta table using authorized access to blob data with managed identities

write_deltalake(
            f"az://container/delta/test",
            df,
            storage_options={"account_name": account_name},
        )

[roeap]

Hi @rbushri - yes, once we do the next python release, that feature will be included.

I do have one question, In the PR you mentioned, I was able to validate that the workload identities work, but did not have the chance to test the managed identities. I have a python wrapper around the object store crate that I use to validate everything works on the python side. Would it be possible for you to test if that works with managed identities as well?

[roeap]

After this fix apache/arrow-rs#4193 - i.e. once we integrate the next object store release, managed identities should start to work ...

[Jaegin]

Could you please confirm whether managed identities now work?

[roeap]

@Jaegin - latest object store is integrated on main, and the functionality has been validated over there.

I was out for a few weeks, so not sure, on what exactly has been released in the meantime, but if it is not already released, it will be included in the next ones.

[Jaegin]

Thanks, @roeap - appreciate the update and I'll take a look.

In the meantime, any chance you're also involved with the kafka-delta-ingest project? I have the following issue and seeking some help: delta-io/kafka-delta-ingest#151

Many thanks,
Steve

[mightyshazam]

Managed identity should be working because I am using it currently.

[roeap]

Hey @mightyshazam! great to hear - i guess this gives us enough confidence to close this for now 😆