Unable to list blobs in azurite

[sindre-nistad]

Which service(blob, file, queue, table) does this issue concern?

blob

Which version of the Azurite was used?

3.10.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

Docker

What's the Node.js version?

v12.19.0

What problem was encountered?

Listing all blobs gives a 400

storage-emulator_1  | 2021-02-02T14:26:49.841Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: BlobStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/github-archives?restype=container&comp=list&include=copy%2Cmetadata%2Ctags RequestHeaders:{"host":"127.0.0.1:10000","x-ms-version":"2020-02-10","x-ms-client-request-id":"5ccebd6a-595b-4a27-88d4-e2b89b23b3f7","x-ms-return-client-request-id":"true","user-agent":"azsdk-net-Storage.Blobs/12.7.0 (.NET 5.0.2; Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64)","x-ms-date":"Tue, 02 Feb 2021 14:26:49 GMT","authorization":"SharedKey devstoreaccount1:glKyU7JCOiXUDVf0s1BWeq5oQ6Z+lYUq9NVHXSHmIaI="} ClientIP=192.168.112.1 Protocol=http HTTPVersion=1.1
storage-emulator_1  | 192.168.112.1 - - [02/Feb/2021:14:26:49 +0000] "GET /devstoreaccount1/github-archives?restype=container&comp=list&include=copy%2Cmetadata%2Ctags HTTP/1.1" 400 -
storage-emulator_1  | 2021-02-02T14:26:49.841Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=github-archives Blob=
storage-emulator_1  | 2021-02-02T14:26:49.841Z cd0755c4-f303-403a-b4a5-cba6d564e67d verbose: DispatchMiddleware: Dispatching request...
storage-emulator_1  | 2021-02-02T14:26:49.843Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: DispatchMiddleware: Operation=Container_ListBlobFlatSegment
storage-emulator_1  | 2021-02-02T14:26:49.843Z cd0755c4-f303-403a-b4a5-cba6d564e67d verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
storage-emulator_1  | 2021-02-02T14:26:49.843Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: PublicAccessAuthenticator:validate() Start validation against public access.
storage-emulator_1  | 2021-02-02T14:26:49.843Z cd0755c4-f303-403a-b4a5-cba6d564e67d debug: PublicAccessAuthenticator:validate() Getting account properties...
storage-emulator_1  | 2021-02-02T14:26:49.843Z cd0755c4-f303-403a-b4a5-cba6d564e67d debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: github-archives, blob: 
storage-emulator_1  | 2021-02-02T14:26:49.844Z cd0755c4-f303-403a-b4a5-cba6d564e67d debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container github-archives
storage-emulator_1  | 2021-02-02T14:26:49.844Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
storage-emulator_1  | 2021-02-02T14:26:49.845Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-client-request-id:5ccebd6a-595b-4a27-88d4-e2b89b23b3f7\nx-ms-date:Tue, 02 Feb 2021 14:26:49 GMT\nx-ms-return-client-request-id:true\nx-ms-version:2020-02-10\n/devstoreaccount1/devstoreaccount1/github-archives\ncomp:list\ninclude:copy,metadata,tags\nrestype:container"
storage-emulator_1  | 2021-02-02T14:26:49.846Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:glKyU7JCOiXUDVf0s1BWeq5oQ6Z+lYUq9NVHXSHmIaI=
storage-emulator_1  | 2021-02-02T14:26:49.846Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
storage-emulator_1  | 2021-02-02T14:26:49.846Z cd0755c4-f303-403a-b4a5-cba6d564e67d verbose: DeserializerMiddleware: Start deserializing...
storage-emulator_1  | 2021-02-02T14:26:49.847Z cd0755c4-f303-403a-b4a5-cba6d564e67d error: ErrorMiddleware: Received a MiddlewareError, fill error information to HTTP response
storage-emulator_1  | 2021-02-02T14:26:49.847Z cd0755c4-f303-403a-b4a5-cba6d564e67d error: ErrorMiddleware: ErrorName=DeserializationError ErrorMessage=tags is not a valid value for include. The valid values are: ["copy","deleted","metadata","snapshots","uncommittedblobs"].  ErrorHTTPStatusCode=400 ErrorHTTPStatusMessage=undefined ErrorHTTPHeaders=undefined ErrorHTTPBody=undefined ErrorStack="Error: tags is not a valid value for include. The valid values are: [\"copy\",\"deleted\",\"metadata\",\"snapshots\",\"uncommittedblobs\"].\n    at serializeEnumType (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:747:15)\n    at Serializer.serialize (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:527:27)\n    at serializeSequenceType (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:820:35)\n    at Serializer.serialize (/opt/azurite/node_modules/@azure/ms-rest-js/dist/msRest.node.js:539:27)\n    at Object.deserialize (/opt/azurite/dist/src/blob/generated/utils/serializer.js:25:25)\n    at Object.deserializerMiddleware [as default] (/opt/azurite/dist/src/blob/generated/middleware/deserializer.middleware.js:29:18)\n    at /opt/azurite/dist/src/blob/generated/ExpressMiddlewareFactory.js:62:46\n    at Layer.handle [as handle_request] (/opt/azurite/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/opt/azurite/node_modules/express/lib/router/index.js:317:13)\n    at /opt/azurite/node_modules/express/lib/router/index.js:284:7"
storage-emulator_1  | 2021-02-02T14:26:49.847Z cd0755c4-f303-403a-b4a5-cba6d564e67d error: ErrorMiddleware: Set HTTP code: 400
storage-emulator_1  | 2021-02-02T14:26:49.847Z cd0755c4-f303-403a-b4a5-cba6d564e67d error: ErrorMiddleware: Set HTTP body: undefined
storage-emulator_1  | 2021-02-02T14:26:49.847Z cd0755c4-f303-403a-b4a5-cba6d564e67d info: EndMiddleware: End response. TotalTimeInMS=7 StatusCode=400 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.10.0"}
storage-emulator_1  | 2021-02-02T14:26:51.661Z   info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
storage-emulator_1  | 2021-02-02T14:26:51.661Z   info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
storage-emulator_1  | 2021-02-02T14:26:51.662Z   info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
storage-emulator_1  | 2021-02-02T14:26:51.662Z   info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
storage-emulator_1  | 2021-02-02T14:26:51.662Z   info: QueueGCManager:markSweepLoop() Start new mark and sweep.
storage-emulator_1  | 2021-02-02T14:26:51.662Z   info: QueueGCManger:markSweep() Get all extents.
storage-emulator_1  | 2021-02-02T14:26:51.663Z   info: QueueGCManager:marksweep() Get 0 extents.
storage-emulator_1  | 2021-02-02T14:26:51.663Z   info: QueueGCManager:markSweep() Get referred extents, then remove from allExtents.
storage-emulator_1  | 2021-02-02T14:26:51.663Z   info: QueueGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
storage-emulator_1  | 2021-02-02T14:26:51.663Z   info: QueueGCManager:markSweepLoop() Mark and sweep finished, take 1ms.
storage-emulator_1  | 2021-02-02T14:26:51.663Z   info: QueueGCManager:markSweepLoop() Sleep for 60000

Steps to reproduce the issue?

Program.cs

using System;
using System.Linq;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;

namespace Foo
{
    internal static class Program
    {
        private static void Main(string[] args)
        {
            string StorageKey =
                "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;";
            string BlobContainer = "github-archives";
            
            BlobServiceClient blobServiceClient = new BlobServiceClient(StorageKey);
            // Assuming you have already created a container named "GitHub-archive", otherwise, that can be done
            // in the storage simulator, or by making this method async, and use
            // `await BlobServiceClient.CreateBlobContainerAsync(BlobContainer)` instead.
            BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(BlobContainer);
            Console.WriteLine(containerClient.GetBlobs(BlobTraits.All).ToList());
        }
    }
}

docker-compose.yml

version: "3.8"

services:
  storage-emulator:
    image: mcr.microsoft.com/azure-storage/azurite
    volumes:
      - "blobStorage:/data"
    ports:
      - "10000:10000"  # Blobs
    command: [
        "azurite",
        "--blobHost", "0.0.0.0",
        "--oauth", "basic",
        "--loose",
        "--debug", "/dev/stdout"
    ]

When containerClient.GetBlobs(BlobTraits.All).ToList() is called, I get the exception Exception of type 'System.Xml.XmlException' was thrown

Have you found a mitigation/solution?

No

[blueww]

@sindre-nistad
Azurite still not support blob tag, see detail in Support Matrix
It failed since you use BlobTraits.All in list blob, and SDK will set request with include=copy%2Cmetadata%2Ctags in the Uri to Azurite, but include tags is not supported by Azurite.

To workaround this, would suggest not use BlobTraits.All, but use BlobTraits combination without tags like BlobTraits.Metadata | BlobTraits.CopyStatus

[sindre-nistad]

Thanks for letting me know, and explain the issue for me 😄
Do you know if this will be supported? If so, is there a timeline?

[sindre-nistad]

I've tried with BlobTraits.Metadata | BlobTraits.CopyStatus, and BlobTraits.Metadata (as we need some of the metadata), and that worked like a charm

[blueww]

@sindre-nistad
Blob tag is already tracked in issue: #647

As your scenarios to list blob works now, do you mind to close this issue, and we can track blob tag in #647.

Besides that, we welcome contribution!
It would be great if you would like to join the effort to add blob tag support to Azurite.