Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Globalization issues when using dotnet/aspnet:9.0-azurelinux3.0-distroless-extra on ARM64 #6023

Closed
craigktreasure opened this issue Nov 2, 2024 · 10 comments
Assignees

Comments

@craigktreasure
Copy link

craigktreasure commented Nov 2, 2024

Describe the bug

I have a simple Blazor application (using Blazorise in this case) that ran just fine on .NET 8 in either the 8.0-azurelinux3.0-distroless-extra or 8.0-cbl-mariner2.0-distroless-extra containers. After upgrading the application to .NET 9 RC 2 using the 9.0-azurelinux3.0-distroless-extra container image (Mariner 2.0 is not available for .NET 9), I began encountering globalization errors that I haven't been able to resolve. There appears to be something different between the globalization resources installed for 9.0-azurelinux3.0-distroless-extra compared to the others, which is unexpected when upgrading to .NET 9 or switching from another .NET 9 container image.

Update: After further investigation, this issue only appears when running the ARM64 variant of the image on an ARM64 device.

Which .NET image(s) are you using?

mcr.microsoft.com/dotnet/aspnet:9.0-azurelinux3.0-distroless-extra with sha256:e006bfc2276280f9f768c555a2870d3484b85a871ef9d3635eab29e9a5795c97

Steps to reproduce

See the minimal repro here and run on an ARM64 device.

Other information

The same code works in both .NET 8 and .NET 9 with the following container images (not exhaustive):

  • mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0-distroless-extra
  • mcr.microsoft.com/dotnet/aspnet:8.0-azurelinux3.0-distroless-extra
  • mcr.microsoft.com/dotnet/aspnet:9.0-noble-chiseled-extra

The same code does not work on .NET 9 with the following container images (not exhaustive):

  • mcr.microsoft.com/dotnet/aspnet:9.0-azurelinux3.0-distroless-extra

The error encountered is:

System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
cs is an invalid culture identifier.
   at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
   at Blazorise.Localization.TextLocalizerService.AddLanguageResource(String cultureName)
   at Blazorise.Localization.TextLocalizerService.ReadResource()
   at Blazorise.Localization.TextLocalizerService..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.AspNetCore.Components.ComponentFactory.<>c__DisplayClass9_0.<CreatePropertyInjector>g__Initialize|1(IServiceProvider serviceProvider, IComponent component)
   at Microsoft.AspNetCore.Components.ComponentFactory.InstantiateComponent(IServiceProvider serviceProvider, Type componentType, IComponentRenderMode callerSpecifiedRenderMode, Nullable`1 parentComponentId)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.InstantiateChildComponentOnFrame(RenderTreeFrame[] frames, Int32 frameIndex, Int32 parentComponentId)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewComponentFrame(DiffContext& diffContext, Int32 frameIndex)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewSubtree(DiffContext& diffContext, Int32 frameIndex)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& diffContext, Int32 oldStartIndex, Int32 oldEndIndexExcl, Int32 newStartIndex, Int32 newEndIndexExcl)
   at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.ComputeDiff(Renderer renderer, RenderBatchBuilder batchBuilder, Int32 componentId, ArrayRange`1 oldTree, ArrayRange`1 newTree)
   at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment, Exception& renderFragmentException)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()

I encountered this on:

  • Raspberry Pi running Ubuntu 24.04 (arm64)
  • Windows 11 24H2 26100.2033 (arm64)

Output of docker version

Client:
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 18:10:55 2024
OS/Arch: windows/arm64
Context: desktop-linux

Server: Docker Desktop 4.35.1 (173168)
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:19 2024
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.21
GitCommit: 472731909fa34bd7bc9c087e4c27943f9835f111
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Output of docker info

Client:
Version: 27.3.1
Context: desktop-linux
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe
compose: Docker Compose (Docker Inc.)
Version: v2.29.7-desktop.1
Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe
debug: Get a shell into any image or container (Docker Inc.)
Version: 0.0.37
Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe
desktop: Docker Desktop commands (Alpha) (Docker Inc.)
Version: v0.0.15
Path: C:\Program Files\Docker\cli-plugins\docker-desktop.exe
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.2
Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.27
Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe
feedback: Provide feedback, right in your terminal! (Docker Inc.)
Version: v1.0.5
Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe
init: Creates Docker-related starter files for your project (Docker Inc.)
Version: v1.3.0
Path: C:\Program Files\Docker\cli-plugins\docker-init.exe
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe
scout: Docker Scout (Docker Inc.)
Version: v1.14.0
Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 12
Server Version: 27.3.1
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
Kernel Version: 5.15.153.1-microsoft-standard-WSL2
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 12
Total Memory: 15.48GiB
Name: docker-desktop
ID: 906f5a08-c33f-41da-a52b-d8fa8d97bdfd
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Labels:
com.docker.desktop.address=npipe://\.\pipe\docker_cli
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

@lbussell
Copy link
Contributor

lbussell commented Nov 4, 2024

Hi @craigktreasure, thank you for the detailed issue. I was not able to reproduce this issue using the instructions in the repo you provided.

I did have one suggestion that may help - I noticed that your Dockerfile relies on building the app locally and then copying binaries to the image. This can cause differences in the build from machine-to-machine.

I would recommend building the app inside the Dockerfile. You can check out our globalization sample for an example: https://github.com/dotnet/dotnet-docker/blob/0f8eee07e5c6fcc7243e10459a84b4b8090e506c/samples/globalapp/Dockerfile.azurelinux-distroless

I tested building and running the app with this Dockerfile as well and didn't run into any issues:

ARG sdkImageTag=9.0-azurelinux3.0
ARG baseImageTag=9.0-azurelinux3.0-distroless-extra

FROM mcr.microsoft.com/dotnet/sdk:${sdkImageTag} AS build
ARG TFM=net9.0
WORKDIR /source

# Copy project file and restore as distinct layers
COPY *.csproj .
RUN dotnet restore

# Copy source code and publish app
COPY . .
RUN dotnet publish --configuration Release --framework $TFM --no-restore -o /app

FROM mcr.microsoft.com/dotnet/aspnet:${baseImageTag}
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["./MyNewBlazoriseApp"]

You may also want to add a .dockerignore file so that you don't accidentally copy over the contents of any previous local builds:

**/bin
**/obj
**/out

@craigktreasure
Copy link
Author

@lbussell I just tried it on an x64 machine and I didn't see it repro either. Do you happen to have an arm64 machine you can try it on? It might be specific to the arm64 image.

@craigktreasure
Copy link
Author

@lbussell Updated my repro with your suggestions and retried on a few more machines. I can only reproduce it on ARM64 machines, so I believe it's specific to the ARM64 image produced. I'll update the title and description.

@craigktreasure craigktreasure changed the title Globalization issues when using dotnet/aspnet:9.0-azurelinux3.0-distroless-extra Globalization issues when using dotnet/aspnet:9.0-azurelinux3.0-distroless-extra on ARM64 Nov 5, 2024
@akoken
Copy link

akoken commented Nov 5, 2024

I got the same error on my machine.

OS: MacOS Sequoia 15.1 (Apple Silicon)
Environment: Docker Desktop v4.35.1
Emulator: Rosetta

❯ docker run --rm -it -p 8080:8080 -e ASPNETCORE_ENVIRONMENT=Development mynewblazoriseapp:net9-azurelinux3
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/home/app/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {63bec15f-2796-416f-b922-12d5b5a08611} may be persisted to storage in unencrypted form.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:8080
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
      Failed to determine the https port for redirect.
warn: Microsoft.AspNetCore.Components.Server.Circuits.RemoteRenderer[100]
      Unhandled exception rendering component: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
      cs is an invalid culture identifier.
      System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
      cs is an invalid culture identifier.
         at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
         at Blazorise.Localization.TextLocalizerService.AddLanguageResource(String cultureName)
         at Blazorise.Localization.TextLocalizerService.ReadResource()
         at Blazorise.Localization.TextLocalizerService..ctor()
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
         at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
         at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite callSite, RuntimeResolverContext context)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
         at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
         at Microsoft.AspNetCore.Components.ComponentFactory.<>c__DisplayClass9_0.<CreatePropertyInjector>g__Initialize|1(IServiceProvider serviceProvider, IComponent component)
         at Microsoft.AspNetCore.Components.ComponentFactory.InstantiateComponent(IServiceProvider serviceProvider, Type componentType, IComponentRenderMode callerSpecifiedRenderMode, Nullable`1 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.Renderer.InstantiateChildComponentOnFrame(RenderTreeFrame[] frames, Int32 frameIndex, Int32 parentComponentId)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewComponentFrame(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InitializeNewSubtree(DiffContext& diffContext, Int32 frameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.InsertNewFrame(DiffContext& diffContext, Int32 newFrameIndex)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.AppendDiffEntriesForRange(DiffContext& diffContext, Int32 oldStartIndex, Int32 oldEndIndexExcl, Int32 newStartIndex, Int32 newEndIndexExcl)
         at Microsoft.AspNetCore.Components.RenderTree.RenderTreeDiffBuilder.ComputeDiff(Renderer renderer, RenderBatchBuilder batchBuilder, Int32 componentId, ArrayRange`1 oldTree, ArrayRange`1 newTree)
         at Microsoft.AspNetCore.Components.Rendering.ComponentState.RenderIntoBatch(RenderBatchBuilder batchBuilder, RenderFragment renderFragment, Exception& renderFragmentException)
         at Microsoft.AspNetCore.Components.RenderTree.Renderer.ProcessRenderQueue()

@lbussell
Copy link
Contributor

lbussell commented Nov 5, 2024

Thanks for the additional context. We'll try to repro this specifically on ARM64.

@mthalman
Copy link
Member

mthalman commented Nov 5, 2024

Found the problem. The .NET Dockerfile is misconfigured:

FROM $REPO:9.0.0-rc.2-azurelinux3.0-distroless-arm64v8

This should be based on the extra runtime image tag instead: 9.0.0-rc.2-azurelinux3.0-distroless-extra-arm64v8

Because it doesn't have the right base, it causes it to have DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true.

@lbussell
Copy link
Contributor

lbussell commented Nov 6, 2024

@craigktreasure we've just published a fix in the nightly repo. Can you please try your scenario with the image: mcr.microsoft.com/dotnet/nightly/aspnet:9.0-azurelinux3.0-distroless-extra-arm64v8@sha256:47100c385f5865ef1422114e84a75ad3b7942fda05e92ea4bd057e748d0bdd40?

If it's all working, then we'll publish the fix in the main repo (non-nightly) next week.

@craigktreasure
Copy link
Author

@lbussell, just tried it locally (removed the duplicate sha256) and it did work. Assuming the 9.0-azurelinux3.0-distroless-extra-arm64v8 and 9.0-azurelinux3.0-distroless-extra (for the ARM64 platform) are built the same, that should do it! Sadly, MCR doesn't show multi-platform image details, so it's not totally clear whether they're equivalent or not without looking at the build process

craigktreasure added a commit to craigktreasure/azurelinux3-invariant-repro that referenced this issue Nov 6, 2024
This change tests the proposed fix mentioned [here](dotnet/dotnet-docker#6023 (comment)).
@lbussell
Copy link
Contributor

lbussell commented Nov 6, 2024

Oops yes, that was a typo. They are built the same, just wanted to be as specific as possible.

PS> docker buildx imagetools inspect mcr.microsoft.com/dotnet/nightly/aspnet:9.0-azurelinux3.0-distroless-extra

Name:      mcr.microsoft.com/dotnet/nightly/aspnet:9.0-azurelinux3.0-distroless-extra
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:1b20007a9f3e6cc6ede5731ee533f768b591775cf004c1baa7a5c160e6abe7a4

Manifests: 
  Name:      mcr.microsoft.com/dotnet/nightly/aspnet:9.0-azurelinux3.0-distroless-extra@sha256:9525d423ae642db4ca17cb03d19d0a9261ed38756e5fc24720e635062080a792
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      mcr.microsoft.com/dotnet/nightly/aspnet:9.0-azurelinux3.0-distroless-extra@sha256:47100c385f5865ef1422114e84a75ad3b7942fda05e92ea4bd057e748d0bdd40
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

I'm working on better tests for this scenario now. Thanks for the detailed bug report and for testing out our images early!

@lbussell lbussell moved this from Backlog to Rollout in .NET Docker Nov 7, 2024
@lbussell lbussell self-assigned this Nov 7, 2024
@lbussell
Copy link
Contributor

These fixes were rolled out yesterday with .NET 9. Feel free to re-open if that is not the case. Thanks again for the report!

@github-project-automation github-project-automation bot moved this from Rollout to Done in .NET Docker Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

4 participants