[Blobfusev2] Unable to authenticate with User-Assigned Identity when System-Assigned Identity is enabled

[AGELadviseurs]

Which version of blobfuse was used?

v2

Which OS distribution and version are you using?

Ubuntu 20.04.5 LTS

If relevant, please share your mount command.

blobfuse2 mount input/ --config-file=config-input.yaml

What was the issue encountered?

Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]

Have you found a mitigation/solution?

Not yet

Please share logs if available.

I'm trying to mount using MSI. This is the relevant part from the config file:

azstorage:
type: adls
account-name: container
container: input
endpoint: https://container.blob.core.windows.net
mode: msi
appid:
#objid:

The VM has a managed identity assigned. I've tried using both the appid as the objid parameter, but both give me the same error. Also tried clientid but that aswell gave me the same error.
Additional info: the VM has both a system-assigned identity enabled as 2 managed identities assigned.

[vibhansa-msft]

If your VM has Managed Identity assigned to it, you don't need to provide any appid/clientid in the configuration.
Make sure your identity has "Storage blob data contributor" role assigned to your storage account.

[AGELadviseurs]

If your VM has Managed Identity assigned to it, you don't need to provide any appid/clientid in the configuration. Make sure your identity has "Storage blob data contributor" role assigned to your storage account.

But I have 2 managed identities AND a system assigned identity assigned to the VM? How do I configure the config file to use the specific managed identity?

[vibhansa-msft]

We have not encountered such use-case so far so to be honest not sure about this one. My guess here would be specifying the resource-id or client-id shall work.

[AGELadviseurs]

This is weird. I just tried with an Access Key for the sake of testing it, but also does not work. This is the azstorage part (endpoint/container/account-key replaced) in my config file:

azstorage:
type: adls
account-name: input
container: input
endpoint: https://input.blob.core.windows.net
mode: key
account-key: accountkey

The error:
Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]

Grepping blob on syslog does not yield any results.

[vibhansa-msft]

In /var/log you might have a file blobfuse2.log, you can check for logs there. Also, add "sdk-trace: true" to your 'azstorage' section in config file. You can remove the endpoint part as anyway blobfuse2 can create it on its own based on account type and name,

[vibhansa-msft]

Can you try the connection outside of this VM. As VM has the identity assigned it might be some privilege related issue I assume causing this failure. Or if possible, remove all assigned identities to the VM and then try using key as auth mode.

[AGELadviseurs]

Ok I tried several things. First of all I tried to get the access-key part working.

1. It worked after I changed the type parameter from 'adsl' to 'block'. Not sure I quite understand this, but I enabled data lake storage when creating the storage container.
   No authentication errors anymore, but it still did not mount.
2. I then tried running the blobfuse2 command with 'sudo', now it mounted succesfully

Now I tried figuring out the identity stuff.

1. As we are using a Landing Zone, system-assigned identities are enabled for monitoring through policy by our corporate team. We cannot disable this.
2. I tried assigning the system-assigned identity the proper permissions on the storage container, voila it worked.

It seems if you have system-assigned AND user-assigned identities on a VM, the blobfuse driver will always try to authenticate through the system-assigned identity, even though you provide 'appid' or 'objid' of the user-assigned identity. It seems it discards these values. Perhaps they will work when you don't have a system-assigned identity on the VM, but unfortunately I cannot test this due to above explained reasons.

[vibhansa-msft]

When multiple identities are assigned to VM, I doubt it's something in the backend that overrides or choose which identity to use. As per blobfuse code we use what user has given and you can validate that part in debug logs. May be something inherence in azure vm overrides this is my guess.
For your question on ADLS vs Block part, if your account is HNS enabled then you shall use "type: adls". You shall not put "endpoint" as adls and block uses different endpoints so better leave that to blobfuse to decide.
Sudo and non-Sudo user both shall be able to mount with blobfuse. May be its some privileges configuration on system that is preventing it here.

[vibhansa-msft]

This multiple identify scenario we have not tested or tried out from end so far, so not confident on what will be the expected behavior here.

[AGELadviseurs]

When I use 'mode: MSI', I always see the same MSI token in the logfile, regardless if I also enter 'appid' or 'objid'. So it really looks like there is some kind of override when it VM has a system-assigned identity enabled. Perhaps you could test this.

Regarding ADSL vs block: as per your instructions (and using sudo now) I can see a different outcome in the logfile (connection reset by peer). I believe we need this different endpoint whitelisted first on our side.

[vibhansa-msft]

for ADLS accounts you need to whitelist the ".dfs" endpoint in addition to ".blob" endpoint.

[vibhansa-msft]

Did enabling dfs endpoint work for you.

[AGELadviseurs]

Did enabling dfs endpoint work for you.

Yes, it did. Sorry for not giving feedback.

Only remaining "problem" is the initial question with identities. Solved it for now by adding Data Contributor RBAC to the system-assigned identity (need it on the user-assigned identity).

[vibhansa-msft]

Yes, for that I guess there is some overriding at system level and system-assigned identity is taking precedence. I do not see any way of controlling that from blobfuse code as of now. Will put this as a feature-request when VM has multiple identity associated provide control on which one to use.

[markti]

Also facing challenges when using both SystemAssigned and UserAssigned Managed Identifies...

"failed to initialize new pipeline"

I will try adding RBAC to the SystemAssigned identity. I have attempted to pipe in the UserAssigned managed ID but it doesn't work.

[vibhansa-msft]

As per above discussion if VM has both system and user assigned identities, Blobfuse is taking system assigned one so your storage account needs to have "Storage blob data contributor" role assigned to that particular identity.

[markti]

As per above discussion if VM has both system and user assigned identities, Blobfuse is taking system assigned one so your storage account needs to have "Storage blob data contributor" role assigned to that particular identity.

this is not good. I'm using terraform to create a user assigned identity specifically for the Blob Storage Access. The System Assigned I only use for AAD login for Linux. If I pass the appid, blobfuse should use it 100% of the time.

[vibhansa-msft]

Its not controlled by Blobfuse is what we observed earlier. Whatever is provided in config file blobfuse was using it but there was some higher level overriding happening which went into using system identity.

[sk3016]

Did enabling dfs endpoint work for you.

Yes, it did. Sorry for not giving feedback.

Only remaining "problem" is the initial question with identities. Solved it for now by adding Data Contributor RBAC to the system-assigned identity (need it on the user-assigned identity).

Do you think adding "Data Contributor RBAC role to Storage account " which is not in the same subscription also should work ? My VM is in abc subscription and the storage account that i am trying to access is in different subscription .

[AGELadviseurs]

Did enabling dfs endpoint work for you.

Yes, it did. Sorry for not giving feedback.
Only remaining "problem" is the initial question with identities. Solved it for now by adding Data Contributor RBAC to the system-assigned identity (need it on the user-assigned identity).

Do you think adding "Data Contributor RBAC role to Storage account " which is not in the same subscription also should work ? My VM is in abc subscription and the storage account that i am trying to access is in different subscription .

It should work if both subscriptions are within the same tenant.

[sk3016]

Did enabling dfs endpoint work for you.

Yes, it did. Sorry for not giving feedback.
Only remaining "problem" is the initial question with identities. Solved it for now by adding Data Contributor RBAC to the system-assigned identity (need it on the user-assigned identity).

Do you think adding "Data Contributor RBAC role to Storage account " which is not in the same subscription also should work ? My VM is in abc subscription and the storage account that i am trying to access is in different subscription .

It should work if both subscriptions are within the same tenant.

ok, thank you for the update. let me try.

[trexx00]

Hi,
A related issue:
Trying to use the objid. In my case I only have the System assigned identity. The Roles assigned are:
"Storage Blob Reader", "Storage Blob Data Read/Write".
Mounting fails with :
Error: failed to initialize new pipeline [failed to get credential]
Should this be failing ? and is it due because of a restriction with "Contributor" vs. "Read/Write" ?

[vibhansa-msft]

you can try assigning "storage blob data contributor" role to the identity and verify that works fine or not. This is just to cross-validate there is no other issue with any sort of config or setup. We do recommend this role to our customers as read/write will have certain limitations.

[vibhansa-msft]

This limitation is coming from the SDK itself. We are planning to integrate with latest SDK soon, hopefully that will resolve this issue.

[HeyangQin]

I encountered the same issue. I later realized that the problem was simply due to using the incorrect account key🤣.

[vibhansa-msft]

With our latest track-2 migration this will be resolved.

[vibhansa-msft]

This has been resolved in our latest 2.3.0~preivew.1 release.