Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race in RuntimeTypeCache / metadata reading #81884

Closed
marek-safar opened this issue Feb 9, 2023 · 8 comments · Fixed by #81947
Closed

Race in RuntimeTypeCache / metadata reading #81884

marek-safar opened this issue Feb 9, 2023 · 8 comments · Fixed by #81947

Comments

@marek-safar
Copy link
Contributor

It happened to me twice but could not reproduce it reliably.

__DistroRid: osx-arm64
  Determining projects to restore...
  Unhandled exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at System.RuntimeType.get_IsGenericTypeDefinition()
     at System.RuntimeType.MakeGenericType(Type[] instantiation)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.PopulateInterfaces(Filter filter)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.GetListByName(Char* pName, Int32 cNameLen, Byte* pUtf8Name, Int32 cUtf8Name, MemberListType listType, CacheType cacheType)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.Populate(String name, MemberListType listType, CacheType cacheType)
     at System.RuntimeType.GetInterfaces()
     at System.Text.Json.Reflection.ReflectionExtensions.GetCompatibleGenericInterface(Type type, Type interfaceType)
     at System.Text.Json.Serialization.IAsyncEnumerableConverterFactory.CanConvert(Type typeToConvert)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetBuiltInConverter(Type typeToConvert)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetConverterForType(Type typeToConvert, JsonSerializerOptions options, Boolean resolveJsonConverterAttribute)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreateJsonTypeInfo(Type type, JsonSerializerOptions options)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options)
     at System.Text.Json.JsonSerializerOptions.GetTypeInfoNoCaching(Type type)
     at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
     at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Boolean resolveIfMutable)
     at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions options, Type inputType)
     at System.Text.Json.JsonSerializer.GetTypeInfo[T](JsonSerializerOptions options)
     at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
     at Microsoft.DotNet.ToolPackage.LocalToolsResolverCache.GetCacheTable(String packageCacheFile)
     at Microsoft.DotNet.ToolPackage.LocalToolsResolverCache.TryLoad(RestoredCommandIdentifier restoredCommandIdentifier, RestoredCommand& restoredCommand)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.PackageHasBeenRestored(ToolManifestPackage package, String targetFramework)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.InstallPackages(ToolManifestPackage package, Nullable`1 configFile)
     at System.Linq.Parallel.SelectQueryOperator`2.SelectQueryOperatorResults.GetElement(Int32 index)
     at System.Linq.Parallel.QueryResults`1.get_Item(Int32 index)
     at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
     at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()
     at System.Linq.Parallel.SpoolingTaskBase.Work()
     at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location ---
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
     --- End of inner exception stack trace ---
     at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
     at System.Linq.Parallel.SpoolingTask.SpoolStopAndGo[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, SynchronousChannel`1[] channels, TaskScheduler taskScheduler)
     at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
     at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
     at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
     at System.Linq.Parallel.QueryOperator`1.ExecuteAndGetResultsAsArray()
     at System.Linq.ParallelEnumerable.ToArray[TSource](ParallelQuery`1 source)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.Execute()
     at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at Microsoft.DotNet.Cli.Parser.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseHelp>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__17_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<<UseParseDirective>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass7_0.<<UseExceptionHandler>b__0>d.MoveNext()

Version: 8.0.100-alpha.1.23068.1

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Feb 9, 2023
@ghost
Copy link

ghost commented Feb 9, 2023

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

It happened to me twice but could not reproduce it reliably.

__DistroRid: osx-arm64
  Determining projects to restore...
  Unhandled exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at System.RuntimeType.get_IsGenericTypeDefinition()
     at System.RuntimeType.MakeGenericType(Type[] instantiation)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.PopulateInterfaces(Filter filter)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.GetListByName(Char* pName, Int32 cNameLen, Byte* pUtf8Name, Int32 cUtf8Name, MemberListType listType, CacheType cacheType)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.Populate(String name, MemberListType listType, CacheType cacheType)
     at System.RuntimeType.GetInterfaces()
     at System.Text.Json.Reflection.ReflectionExtensions.GetCompatibleGenericInterface(Type type, Type interfaceType)
     at System.Text.Json.Serialization.IAsyncEnumerableConverterFactory.CanConvert(Type typeToConvert)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetBuiltInConverter(Type typeToConvert)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetConverterForType(Type typeToConvert, JsonSerializerOptions options, Boolean resolveJsonConverterAttribute)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreateJsonTypeInfo(Type type, JsonSerializerOptions options)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options)
     at System.Text.Json.JsonSerializerOptions.GetTypeInfoNoCaching(Type type)
     at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
     at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Boolean resolveIfMutable)
     at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions options, Type inputType)
     at System.Text.Json.JsonSerializer.GetTypeInfo[T](JsonSerializerOptions options)
     at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
     at Microsoft.DotNet.ToolPackage.LocalToolsResolverCache.GetCacheTable(String packageCacheFile)
     at Microsoft.DotNet.ToolPackage.LocalToolsResolverCache.TryLoad(RestoredCommandIdentifier restoredCommandIdentifier, RestoredCommand& restoredCommand)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.PackageHasBeenRestored(ToolManifestPackage package, String targetFramework)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.InstallPackages(ToolManifestPackage package, Nullable`1 configFile)
     at System.Linq.Parallel.SelectQueryOperator`2.SelectQueryOperatorResults.GetElement(Int32 index)
     at System.Linq.Parallel.QueryResults`1.get_Item(Int32 index)
     at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
     at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()
     at System.Linq.Parallel.SpoolingTaskBase.Work()
     at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location ---
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
     --- End of inner exception stack trace ---
     at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
     at System.Linq.Parallel.SpoolingTask.SpoolStopAndGo[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, SynchronousChannel`1[] channels, TaskScheduler taskScheduler)
     at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
     at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
     at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
     at System.Linq.Parallel.QueryOperator`1.ExecuteAndGetResultsAsArray()
     at System.Linq.ParallelEnumerable.ToArray[TSource](ParallelQuery`1 source)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.Execute()
     at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at Microsoft.DotNet.Cli.Parser.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseHelp>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__17_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<<UseParseDirective>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass7_0.<<UseExceptionHandler>b__0>d.MoveNext()

Version: 8.0.100-alpha.1.23068.1

Author: marek-safar
Assignees: -
Labels:

area-System.Text.Json

Milestone: -

@marek-safar marek-safar added area-System.Reflection and removed untriaged New issue has not been triaged by the area owner area-System.Text.Json labels Feb 9, 2023
@ghost
Copy link

ghost commented Feb 9, 2023

Tagging subscribers to this area: @dotnet/area-system-reflection
See info in area-owners.md if you want to be subscribed.

Issue Details

It happened to me twice but could not reproduce it reliably.

__DistroRid: osx-arm64
  Determining projects to restore...
  Unhandled exception: System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
   ---> System.NullReferenceException: Object reference not set to an instance of an object.
     at System.RuntimeType.get_IsGenericTypeDefinition()
     at System.RuntimeType.MakeGenericType(Type[] instantiation)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.PopulateInterfaces(Filter filter)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.GetListByName(Char* pName, Int32 cNameLen, Byte* pUtf8Name, Int32 cUtf8Name, MemberListType listType, CacheType cacheType)
     at System.RuntimeType.RuntimeTypeCache.MemberInfoCache`1.Populate(String name, MemberListType listType, CacheType cacheType)
     at System.RuntimeType.GetInterfaces()
     at System.Text.Json.Reflection.ReflectionExtensions.GetCompatibleGenericInterface(Type type, Type interfaceType)
     at System.Text.Json.Serialization.IAsyncEnumerableConverterFactory.CanConvert(Type typeToConvert)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetBuiltInConverter(Type typeToConvert)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetConverterForType(Type typeToConvert, JsonSerializerOptions options, Boolean resolveJsonConverterAttribute)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.CreateJsonTypeInfo(Type type, JsonSerializerOptions options)
     at System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver.GetTypeInfo(Type type, JsonSerializerOptions options)
     at System.Text.Json.JsonSerializerOptions.GetTypeInfoNoCaching(Type type)
     at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
     at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(Type type, Boolean ensureConfigured, Boolean resolveIfMutable)
     at System.Text.Json.JsonSerializer.GetTypeInfo(JsonSerializerOptions options, Type inputType)
     at System.Text.Json.JsonSerializer.GetTypeInfo[T](JsonSerializerOptions options)
     at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
     at Microsoft.DotNet.ToolPackage.LocalToolsResolverCache.GetCacheTable(String packageCacheFile)
     at Microsoft.DotNet.ToolPackage.LocalToolsResolverCache.TryLoad(RestoredCommandIdentifier restoredCommandIdentifier, RestoredCommand& restoredCommand)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.PackageHasBeenRestored(ToolManifestPackage package, String targetFramework)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.InstallPackages(ToolManifestPackage package, Nullable`1 configFile)
     at System.Linq.Parallel.SelectQueryOperator`2.SelectQueryOperatorResults.GetElement(Int32 index)
     at System.Linq.Parallel.QueryResults`1.get_Item(Int32 index)
     at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)
     at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()
     at System.Linq.Parallel.SpoolingTaskBase.Work()
     at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
  --- End of stack trace from previous location ---
     at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
     --- End of inner exception stack trace ---
     at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)
     at System.Linq.Parallel.SpoolingTask.SpoolStopAndGo[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, SynchronousChannel`1[] channels, TaskScheduler taskScheduler)
     at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute()
     at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 queryId)
     at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)
     at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)
     at System.Linq.Parallel.QueryOperator`1.ExecuteAndGetResultsAsArray()
     at System.Linq.ParallelEnumerable.ToArray[TSource](ParallelQuery`1 source)
     at Microsoft.DotNet.Tools.Tool.Restore.ToolRestoreCommand.Execute()
     at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at Microsoft.DotNet.Cli.Parser.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseHelp>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__17_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<<UseParseDirective>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass7_0.<<UseExceptionHandler>b__0>d.MoveNext()

Version: 8.0.100-alpha.1.23068.1

Author: marek-safar
Assignees: -
Labels:

area-System.Reflection, area-System.Text.Json

Milestone: -

@marek-safar marek-safar added the untriaged New issue has not been triaged by the area owner label Feb 9, 2023
@jkotas
Copy link
Member

jkotas commented Feb 9, 2023

@EgorBo This is likely caused by missing memory barrier when allocation RuntimeType objects on FOD. Could you please take a look?

@jkotas jkotas added this to the 8.0.0 milestone Feb 9, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Feb 9, 2023
@EgorBo
Copy link
Member

EgorBo commented Feb 9, 2023

@EgorBo This is likely caused by missing memory barrier when allocation RuntimeType objects on FOD. Could you please take a look?

hm.. in general, RtunimeType objects are expected to be allocated and cached under a lock here https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/typehandle.cpp#L366

@jkotas
Copy link
Member

jkotas commented Feb 9, 2023

Here is what is likely happening:

Thread 1: Acquires the lock
Thread 1: Allocates, initializes and publishes the instance (without taking any memory barriers)

Thread 2: Observes the pointer to the published instance, but does not observe the initialized instance (the writes that are initializing the instance are still pending)
-> NullReferenceException caused by attempt to access null TypeHandle

Thread 1: Releases the lock that flushes pending memory writes

@jkotas
Copy link
Member

jkotas commented Feb 9, 2023

BTW: We avoid this sort of problem for regular managed objects and managed references by issuing memory barrier as part of the GC write barrier.

@jkotas
Copy link
Member

jkotas commented Feb 9, 2023

The fix should be to change this https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/typehandle.cpp#L380 and other similar places to use VolatileStore.

@EgorBo
Copy link
Member

EgorBo commented Feb 9, 2023

The fix should be to change this https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/typehandle.cpp#L380 and other similar places to use VolatileStore.

Ok, I'll go check all the places I touched related to FOH for that, thanks for the analysis!

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Feb 10, 2023
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Feb 10, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Mar 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants