Support for blob index

[dacspiir]

Which service(blob, file, queue, table) does this issue concern?

Blob service

Which version of the Azurite was used?

docker latest - image aca174491773

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

DockerHub

What's the Node.js version?

Don't know (it's inside docker)

What problem was encountered?

The query tags feature ( https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-find-blobs?tabs=azure-portal ) is not available in Azurite and this makes it impossible to test code that relies on it.

Any request related to tags will get denied with an HTTP error 400.

Steps to reproduce the issue?

• Create a container "testcontainer"
• GET /devstoreaccount1/testcontainer?restype=container&comp=list&prefix=testprefix&maxresults=100&include=copy%2Cmetadata%2Ctags HTTP/1.1

Actual result: HTTP 400
Expected result: HTTP 200

Debug logs:

2020-11-30T09:55:32.177Z 9bb16b0d-9e44-4deb-8a9b-3cfbcf2f1596 error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
2020-11-30T09:55:32.177Z 9bb16b0d-9e44-4deb-8a9b-3cfbcf2f1596 error: ErrorMiddleware: ErrorName=DeserializationError ErrorMessage=tags is not a valid value for include. The valid values are: ["copy","deleted","metadata","snapshots","uncommittedblobs"]. ErrorHTTPStatusCode=400 ErrorHTTPStatusMessage=undefined ErrorHTTPHeaders=undefined ErrorHTTPBody=undefined ErrorStack="Error: tags is not a valid value for include. The valid values are: ["copy","deleted","metadata","snapshots","uncommittedblobs"].\n at serializeEnumType (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:747:15)\n at Serializer.serialize (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:527:27)\n at serializeSequenceType (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:820:35)\n at Serializer.serialize (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:539:27)\n at Object.deserialize (/opt/azurite/dist/src/blob/generated/utils/serializer.js:25:25)\n at Object.deserializerMiddleware [as default] (/opt/azurite/dist/src/blob/generated/middleware/deserializer.middleware.js:29:18)\n at /opt/azurite/dist/src/blob/generated/ExpressMiddlewareFactory.js:62:46\n at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)\n at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:317:13)\n at /opt/azurite/node_modules/express/lib/router/index.js:284:7"
2020-11-30T09:55:32.177Z 9bb16b0d-9e44-4deb-8a9b-3cfbcf2f1596 error: ErrorMiddleware: Set HTTP code: 400
2020-11-30T09:55:32.177Z 9bb16b0d-9e44-4deb-8a9b-3cfbcf2f1596 error: ErrorMiddleware: Set HTTP body: undefined

Have you found a mitigation/solution?

No. The feature is missing. It makes local testing of blob storage code that uses this feature impossible, as some of our business logic depends on the capability of querying by tag.
It basically forces our developers to create private blob accounts for testing in their local computers.

[blueww]

@dacspiir
Thanks for raising this issue!

Azurite still not support blob tag now.
We will evaluate it.

We welcome contribution to Azurite.
It would be great if you want to add it by raise PR to this repo.

[SeanFeldman]

With Blob Index Tags available it would be really good to see Azurite adding the support. Running any tests against Azurite that contain Blob Index Tags logic cannot be exercised.

[julien-piccaluga]

It will be great to have this Blob Tags feature for our integration tests.

[SeanFeldman]

Blob index tag feature went GA. Any plans to officially support it with Azurite @blueww?

[SeanFeldman]

/cc @XiaoningLiu

[blueww]

@SeanFeldman

Blob Index is a big feature to implement, it's still in planning.

[SeanFeldman]

@blueww any updates on this? Blob Index Tag is GA-ed for quite a while. Thanks.

[blueww]

Thanks for bringing it up, sorry we don't have an accurate date for this feature yet : (

[MarioArriaga92]

Project I work on also needs this, any updates or workarounds/suggestions?

[blueww]

@MarioArriaga92
Thanks for bringing it up again!
We still don't have an accurate date for this feature yet.

Would you please share the scenarios you need use Azurite with blob index tag? This might can help us evaluate this request.

[fbgoode]

We're using blob index tags in a new project as well and we're unable to use Azurite for testing without this feature.

[ConDar15]

Just encountered this problem with Azurite and wanted to share our use case in the hopes that it'll move this closer to being implemented.

• We have a storage of user documents within Azure Blob storage and one of the things we store is user photos.
• We have a need to identify which we consider their "primary" photo for handling and display purposes.
• We use tagging to track which of the photos is the primary photo to avoid having to keep a separate data store just for that bit of metadata, which could get out of sync, etc...
• I would like to use Azurite for local development, however the lack of tag support means I am forced into using either our Azure blob storage instance or the Azure Storage Emulator which Azurite supersedes.

Please add support for blob tags as ours is but one use case of many I can think of that will require the use of tags on blogs for metadata storage.

[jbox5]

• We have a storage of user documents within Azure Blob storage and one of the things we store is user photos.
• We have a need to identify which we consider their "primary" photo for handling and display purposes.
• We use tagging to track which of the photos is the primary photo to avoid having to keep a separate data store just for that bit of metadata, which could get out of sync, etc...
• I would like to use Azurite for local development, however the lack of tag support means I am forced into using either our Azure blob storage instance or the Azure Storage Emulator which Azurite supersedes.

This is exactly my use case as well. Really would like to see this feature.

[RonPeters]

Our use case is images and documents. We tag the blobs with the ID of the entity and sub-entity that "owns" the blob. This allows us to query storage with the entity and/or sub-entity IDs and return all the associated blobs.

[markheath]

With the release of 3.17 Azurite nearly does everything I want it to, but the tagging support is the missing piece. We have code that whenever it copies a blob it also copies any tags that it might have, and we generate SAS Uris with the tag permission, and all of that causes Azurite to reject requests with XmlExceptions being thrown by the C# SDK. We do a lot of our testing currently with Azurite, but as we're rolling out more usage of index tagging, it is going to prevent us using Azurite at all.

[giometrix]

Lack of tag support is preventing me from using azurite, is there a timeline for this?

[blueww]

@giometrix
Thanks for bringing it up again!
We still don't have an accurate date for this feature yet.

[aignatovich]

One of the desired features within the scope of index tags is Tag Conditional Headers. Specifically x-ms-if-tags header support.

Tag expression evaluation does not lead to a success for tag-based access conditions now.
Having this feature would enable the adoption of Azurite for me.

[blueww]

@aignatovich
Per my test on latest Azurite 3.17.1, run some blob API (like get blob properties) with "x-ms-if-tags" header will success.
What's the change you would like? Do you want Azurite to block request with "x-ms-if-tags" header?

[aignatovich]

@blueww I am looking for the blob operation (particularly delete) to be performed only if predicate in the "x-ms-if-tags" header evaluates to true (for instance "Author" = 'Jack'). Source

[blueww]

@aignatovich
Thanks for the information!
Currently Azurite still not support set blob tag. Do you want Azurite to fail on the request with "x-ms-if-tags" header, when not start with "--loose"?

[aignatovich]

@blueww indeed storing blob tags is not yet supported. I think that indicating that particular feature is not supported in the strict mode by replying with http code (the way it is working now) is desired behavior.

Amazing remark about --loose startup parameter. Thanks for the tip!

[aLifeOnMars]

any news regarding implementation of blob tags?

[blueww]

@aLifeOnMars
Thanks for bringing it up again!
We still don't have an accurate date for this feature yet.

[ludvigbartholdsson]

Please fix!

[miroslavvojtus]

Actually this is a reason we are not going to use Azurite for our app.
Sorry to say that but waiting 2 years for feature which is part of standard API makes emulator unusable during development. There is ton of features you do not support which may be show stopper for us but this one is for now key.
I hope you change strategy in future in order to make this solid tool for development and testing.

[blueww]

@miroslavvojtus
Thanks for raise the request again!
Blob index tag is already in our plan of following several months.

Would you like to share some scenarios you use blob index tag, like which API normally you will use with blob index tag? which query you will use?
This can help us better design the feature for Azurite.

[miroslavvojtus]

Hello. I like you are asking 👍 Well, our use case is for sort of transaction logic on top of the blob storage. We have a process which is uploading and deleting several new files in blob storage. As the new files should be visible after the operation and analogously the to be deleted should be erased. We are tagging the blobs with operation id and operation type. At the end we either batch delete the to be deleted items and corresponding tags or in case of failure we revert the operations. Unfortunately we were not aware of the "feature"/drawback of blob indexes and tags the eventual consistency. It led us to drop this idea and rather keep our "taggs" in our ACID DB. Thanks to that we are getting back to Azurite even without the feature :) But anyway, I would appreciate if the emulator would mimic as close to 100% of features as possible. Thanks again for considering consumers' opinion. Huge 👍 po 31. 10. 2022 o 4:31 Wei Wei ***@***.***> napísal(a):

…

@miroslavvojtus <https://github.com/miroslavvojtus> Thanks for raise the request again! Blob index tag is already in our plan of following several months. Would you like to share some scenarios you use blob index tag, like which API normally you will use with blob index tag? which query you will use? This can help us better design the feature for Azurite. — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABS3MS33WGP2JR2ZAKXAUDWF44SPANCNFSM4UHLXVEQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Miroslav Vojtuš

[sivajrm]

Hi @blueww
Is this feature supported yet in Azurite docker installation?
I am seeing the 400 error whenever I try to set tags using

blobClient.setTags(Collections.singletonMap("hello", "world"));

Outgoing request printed on the log is
uri=http://127.0.0.1:10000/devstoreaccount1/community-0?restype=container, method=PUT

Response
HTTP/1.1 400 Bad Request
Server: <filtered>
Date: <filtered>
Connection: <filtered>
Keep-Alive: <filtered>
Content-Length: <filtered>

Thanks,
Siva

[blueww]

@sivajrm

This feature is still not supported in Azurite.
We are working on a plan to implement it.
However, as the feature impact many existing APIs, and also add several new APIs, the implementation will take time.

[sivajrm]

Okay, please keep this thread updated.

[kzu]

Uh... it's been a LOOONG time and it still isn't implemented :(

[msamsonse]

Any update when tags support will be implemented in Azurite?

[cool-html5]

Tag query is actually the only way to meaningfully search for a blob. The alternative is to query for a list of all blobs and compare the blob names, which in services with millions of blobs would take a lot of resources. Please prioritize this.

And yes, I do realize that we are talking about development environment here, but for testing purposes this would be very important.

[blueww]

We are working on it.
However, we might will split it into 2 phases, and implement the Get/Set Tag API in Phase 1, then implement search blob by Tag in Phase 2.

See details in the track items:
#1871
#1872
#1873
#1874

[adeellis]

Hi is there any update on this feature?

[blueww]

@adeellis

Get/Set Blob Tag (Phase 1) is already released in latest Azurite.
Search/filter blob by Tag (Phase 2) is still in development.

[adeellis]

Awesome thanks

[rony-dennis]

Hi team, any updates on this?

[blueww]

@rony-dennis

A draft PR for blob tag query/filter: #2311 is already raised by @EmmaZhu .
However, there are still many items missed for this PR, see more details in the PR. We will have formal review after all missed items are added.

It would be great if you could try/review the implementation and give some feedback!

[black-snow]

bump

[Squiggleh]

Any updates on this?

[blueww]

Blob Tag include filter by tag and query blob by tag already supported in Azurite 3.33.0.

Feel free to raise a new issue if you see any problem when use blob tag related API in Azurite.