[NETSDKE2E][ARM64][intermittent]Get "Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt." When running the command quickly. #41588
Comments
dotnet-issue-labeler
bot
added
Area-Workloads
untriaged
|
After joining other people's accounts, try 6 times and can reproduce. |
|
This issue also repro's on .net9.0.100-preview.6.24314.10SDK |
|
This issue also repro's on .net9.0.100-preview.6.24328.19SDK and I found it seems to be easier to reproduce in an environment with VS+SDK. |
|
I tried this with today's 9.0 preview on an x64 machine and couldn't reproduce this. I suspect it's ARM64-specific. |
|
This might be related to the same parallelism issue we found in the template engine with the Aspire template initially. They were able to workaround it but we never found the underlying issue. @v-ainigao does this only repro on arm64? What version of VS are you using? |
Yes, only repro on ARM64 and the vs I use is for daily testing main branch. |
marcpopMSFT
removed
untriaged
|
This issue also repro's on latest .net9preview7SDK |
|
This issue also repro's on net9.0.100-rc.1.24406.16SDK. |
|
From 2nd September, we experience the same issue on the GitHub action build of the Reqnroll project. A few of our tests perform a It only comes on our Windows job, not on the Linux. For the Windows agents we use It might be related to the .NET SDK installed on the agent: For the failing builds the agent contains .NET SDK 8.0.400, .NET Runtime 8.0.8, but I have checked earlier builds and it was .NET SDK 8.0.303, .NET Runtime 8.0.7. It seems that .NET SDK 8.0.400 is used by the agents since 27th August. |
|
This issue also repro's on net9.0.100-rc.1.24452.12(runtime-9.0.0-rc.1.24431.7)SDK. |
@Forgind We can reproduce it using the standard
So I don't think it would be ARM specific. |
|
Can it be anyhow related to the Visual Studio 17.11 version maybe? Because that was also recently updated (from 17.10) in the GitHub actions image. @v-ainigao Which VS version do you have where you can reproduce the problem? |
It will repro in the cmd when running the code too fast. The vs I am using today is 17.12.0preview2.0[35305.153.d17.12] and the SDK it carries net9.0.100-rc.1.24452.12(runtime-9.0.0-rc.1.24431.7) SDK |
What do you mean by that exactly? (Because in our case we run it from a test and maybe that is also "too fast".) |
It can be understood that you enter the code too quickly or just copy a code and press Enter in the cmd. It is easier to repro.
|
|
I see. I tried to put it even into a PowerShell loop, but locally I cannot reproduce, just on GitHub Actions... 😞 |
I just tried PowerShell and can repro it too in an ARM64OS created using Azure |
|
I also encounter this issue on the main build -Version 17.12.0 Preview 3.0 [35325.140.main] which integrated with 9.0.100-rc.2.24474.11 SDK.
|
|
This issue does not repro on .net9RTM sdk. |
|
@marcpopMSFT Could you please confirm that you have made some changes that could have fixed this as @v-ainigao reported? Because I have just reproduced it locally on my x64 Intel processor with .NET 8.0 just a few days ago. (I don't even have .NET 9 runtime or SDK installed on this machine.) Is there a chance that the same fix can be also applied to .NET 8 SDK? |
I didn't make any changes, and it doesn't repro as usual. |
|
We have not made any changes to resolve this. Our best guess earlier is that it was a race condition but we had trouble reproducing it so weren't able to rack down the issue. Given it's an unreliable repro, that explains why it might have gone away in .net9 in some configurations but still be showing up in other existing net8 configurations. I don't think we know how to investigate with next steps as I'm not sure what information to have you collect. Without a repro we can consistently reproduce, we can't get the debug information to determine what is causing the access violation. |
|
Thank you @marcpopMSFT for the response and it is quite understandable. I think normally this does not cause a big problem as users can retry or use Visual Studio to create a new project when it does not work. Maybe our open-source project (Reqnroll) is the only one that based their testing on generating test projects with We finally decided to avoid using It is also to be noticed that |
|
If you're running Did reducing your usage of new help with the reliability here? @baronfel any thoughts on why new might be slow? Is it the restore step needing to hit the network a second time or potentially something else? |
|
I don't know much about the perf characteristics of dotnet new - we've mostly been focused on functionality in the past. We probably owe a perf trace to find hot spots. It does do an automatic restore as a post action (which should be able to be skipped IIRC). |
@marcpopMSFT For the first look it did. But I could not fully finish that because of some side-effects my workaround caused and I haven't yet got time to investigate those. But I will send an update when things are more visible. |
@baronfel My suspicion is that most of the performance problems rooting from the restore post-action, but I don't have evidence for that. Nevertheless the option of skipping this restore would definitely be helpful, because we anyway add new packages to the created project and do a build afterwards, so the restore has to be run anyway so I don't think we get too much benefit of the restore during If you are interested in performance results, you can download the test result TRX file of our test execution, where you will see the different commands with a time stamp in the test std out. For example in our latest build they are here: https://github.com/reqnroll/Reqnroll/actions/runs/11369395224/artifacts/2064457469 |
|
I believe most or all of the templates have a |
|
@dsplaisted Wow! Thank you! This is cool and I did not know about that. Indeed the Thanks for the hint. Really useful. I will give it a quick try. |
FYI @baronfel, looks like the |
|
We have very explicit control over the help output for top-level |
|
Feedback on performance: I did some basic measurements, and it seems that using |
|
Reactivating as I was able to reproduce this issue and get a crash dump of https://github.com/jaredpar/complog/actions/runs/11713633968 The core bug seems to be that Microsoft.VisualStudio.Setup.Configuration.Native is not thread safe but the wstring s_str;
const wstring& GetStr() {
if (s_str.empty()) {
...
s_str = ...;
}
return s_str;
}This is not safe to call from multiple threads as the assignment can race with Crashing frame: Microsoft.VisualStudio.Setup.Configuration.Native.dll!memcpy() Line 470 Unknown
[Inline Frame] Microsoft.VisualStudio.Setup.Configuration.Native.dll!wmemmove(wchar_t *) Line 247 C++
[Inline Frame] Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::_WChar_traits<wchar_t>::move(wchar_t * const) Line 347 C++
[Inline Frame] Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::wstring::assign(const wchar_t * const) Line 2663 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::wstring::assign(const wchar_t * const _Ptr) Line 2676 C++
[Inline Frame] Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::wstring::operator=(const wchar_t * const) Line 2498 C++
> [Inline Frame] Microsoft.VisualStudio.Setup.Configuration.Native.dll!Policy::get_DefaultCachePath() Line 174 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!Policy::get_CachePath() Line 191 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!StateManager<FileStateManager>::GetRootDirectory() Line 68 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!FileStateManager::GetInstanceIds() Line 18 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!StateManager<FileStateManager>::GetInstanceIds() Line 49 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!EnumSetupInstances::Initialize() Line 47 C++
Microsoft.VisualStudio.Setup.Configuration.Native.dll!EnumSetupInstances::Next(unsigned long celt, ISetupInstance * * rgelt, unsigned long * pceltFetched) Line 69 C++Other thread accessing this same data: Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests() Line 246 C#
Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests() Line 165 C#
Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetAvailableWorkloadDefinitions() Line 568 C#
System.Linq.dll!System.Linq.Enumerable.SelectEnumerableIterator<(Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadDefinition, Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifest), string>.MoveNext() Unknown
System.Linq.dll!System.Linq.Enumerable.Contains<string>(System.Collections.Generic.IEnumerable<string> source, string value, System.Collections.Generic.IEqualityComparer<string> comparer) Unknown
> dotnet.dll!Microsoft.DotNet.Workloads.Workload.VisualStudioWorkloads.GetInstalledWorkloads(Microsoft.NET.Sdk.WorkloadManifestReader.IWorkloadResolver workloadResolver, Microsoft.DotNet.Workloads.Workload.List.InstalledWorkloadsCollection installedWorkloads, Microsoft.NET.Sdk.WorkloadManifestReader.SdkFeatureBand? sdkFeatureBand) Line 59 C#
dotnet.dll!Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.AddInstalledVsWorkloads(System.Collections.Generic.IEnumerable<Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadId> sdkWorkloadIds) Line 66 C#
dotnet.dll!Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.InstalledAndExtendedWorkloads.get() Line 38 C#
dotnet.dll!Microsoft.DotNet.Tools.New.WorkloadsInfoProvider.GetInstalledWorkloadsAsync(System.Threading.CancellationToken cancellationToken) Line 27 C# |
|
Based on the stack trace, it looks like this is caused by the same parallelism issue as dotnet/templating#7946 |
Build info
.net8.0.400-preview.0.24312.1(runtime-8.0.5) sdk
Steps:
1.install net8 preview 4 sdk
2.Quickly run the following command. (Direct Paste)
Expected Result:
The command runs successfully without returning any errors.
Actual Result:
Get "Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt." When running the command quickly.
dotnet --info
``` dotnet --info C:\Users\v-ainigao>dotnet --info .NET SDK: Version: 8.0.400-preview.0.24312.1 Commit: f13e0daa5c Workload version: 8.0.400-manifests.c1c70047 MSBuild version: 17.11.0-preview-24310-01+74e23a98dRuntime Environment:
OS Name: Windows
OS Version: 10.0.22631
OS Platform: Windows
RID: win-arm64
Base Path: C:\Program Files\dotnet\sdk\8.0.400-preview.0.24312.1\
.NET workloads installed:
Configured to use loose manifests when installing new manifests.
[wasm-tools]
Installation Source: SDK 8.0.400-preview.0
Manifest Version: 8.0.5/8.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.workload.mono.toolchain.current\8.0.5\WorkloadManifest.json
Install Type: FileBased
Host:
Version: 8.0.5
Architecture: arm64
Commit: 087e15321b
.NET SDKs installed:
8.0.400-preview.0.24312.1 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 8.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found:
None
Environment variables:
Not set
global.json file:
Not found
Learn more:
https://aka.ms/dotnet/info
Download .NET:
https://aka.ms/dotnet/download
The text was updated successfully, but these errors were encountered: