Deadlock on EventSource.EventListenersLock between EventPipeEventDispatcher.StopDispatchTask() and EventSource.Initialize()

[sandersaares]

Description

I have encountered a deadlock on EventSource.EventListenersLock.

The thread holding the lock is in EventPipeEventDispatcher.StopDispatchTask() doing m_dispatchTask.Wait():

>	System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 575	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3021	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2956	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2752	C#
 	[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]	
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.StopDispatchTask() Line 153	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() Line 88	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords) Line 57	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string> arguments) Line 4076	C#
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.EventListening.DotNetEventListener.OnEventSourceCreated(object sender, System.Diagnostics.Tracing.EventSourceCreatedEventArgs e)	Unknown
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallBackForExistingEventSources(bool addToListenersList, System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs> callback) Line 4412	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.EventSourceCreated.add(System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs> value) Line 3951	C#
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.EventListening.DotNetEventListener.DotNetEventListener(Prometheus.DotNetRuntime.EventListening.IEventListener eventListener, System.Diagnostics.Tracing.EventLevel level, Prometheus.DotNetRuntime.EventListening.DotNetEventListener.GlobalOptions globalOptions)	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.CreateEventListeners.AnonymousMethod__9_0(Prometheus.DotNetRuntime.ListenerRegistration r)	Unknown
 	System.Linq.dll!System.Linq.Enumerable.SelectEnumerableIterator<Prometheus.DotNetRuntime.ListenerRegistration, Prometheus.DotNetRuntime.EventListening.DotNetEventListener>.ToArray()	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.CreateEventListeners()	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.DotNetRuntimeStatsCollector(Microsoft.Extensions.DependencyInjection.ServiceProvider serviceProvider, Prometheus.CollectorRegistry metricRegistry, Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.Options options)	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsBuilder.Builder.StartCollecting(Prometheus.CollectorRegistry registry)	Unknown
[snip]

m_dispatchTask is meanwhile busy dispatching events and one of these dispatches has triggered code that wants to take the same lock:

Reproduction Steps

I do not have an easy way to consistently reproduce the issue but I have a process dump of the deadlocked state, if it helps.

Expected behavior

No deadlocks when doing EventSource related opterations (in this case when accessing Activity.Current, which appears to use EventSource internally, though the expectation is general for all EventSource related operations).

Actual behavior

Deadlock occurs.

Regression?

No response

Known Workarounds

No response

Configuration

Windows 11 x64.

Reproduced both with .NET 7 and .NET 8 preview 3.

Other information

I have a process memory dump, if it helps. Ping me on Teams (sasaares).

[ghost]

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have encountered a deadlock on EventSource.EventListenersLock.

The thread holding the lock is in EventPipeEventDispatcher.StopDispatchTask() doing m_dispatchTask.Wait():

>	System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 575	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 3021	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2956	C#
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2752	C#
 	[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]	
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.StopDispatchTask() Line 153	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.CommitDispatchConfiguration() Line 88	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventDispatcher.SendCommand(System.Diagnostics.Tracing.EventListener eventListener, System.Diagnostics.Tracing.EventCommand command, bool enable, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeywords) Line 57	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.EnableEvents(System.Diagnostics.Tracing.EventSource eventSource, System.Diagnostics.Tracing.EventLevel level, System.Diagnostics.Tracing.EventKeywords matchAnyKeyword, System.Collections.Generic.IDictionary<string, string> arguments) Line 4076	C#
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.EventListening.DotNetEventListener.OnEventSourceCreated(object sender, System.Diagnostics.Tracing.EventSourceCreatedEventArgs e)	Unknown
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.CallBackForExistingEventSources(bool addToListenersList, System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs> callback) Line 4412	C#
 	System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventListener.EventSourceCreated.add(System.EventHandler<System.Diagnostics.Tracing.EventSourceCreatedEventArgs> value) Line 3951	C#
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.EventListening.DotNetEventListener.DotNetEventListener(Prometheus.DotNetRuntime.EventListening.IEventListener eventListener, System.Diagnostics.Tracing.EventLevel level, Prometheus.DotNetRuntime.EventListening.DotNetEventListener.GlobalOptions globalOptions)	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.CreateEventListeners.AnonymousMethod__9_0(Prometheus.DotNetRuntime.ListenerRegistration r)	Unknown
 	System.Linq.dll!System.Linq.Enumerable.SelectEnumerableIterator<Prometheus.DotNetRuntime.ListenerRegistration, Prometheus.DotNetRuntime.EventListening.DotNetEventListener>.ToArray()	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.CreateEventListeners()	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.DotNetRuntimeStatsCollector(Microsoft.Extensions.DependencyInjection.ServiceProvider serviceProvider, Prometheus.CollectorRegistry metricRegistry, Prometheus.DotNetRuntime.DotNetRuntimeStatsCollector.Options options)	Unknown
 	prometheus-net.DotNetRuntime.dll!Prometheus.DotNetRuntime.DotNetRuntimeStatsBuilder.Builder.StartCollecting(Prometheus.CollectorRegistry registry)	Unknown
[snip]

m_dispatchTask is meanwhile busy dispatching events and one of these dispatches has triggered code that wants to take the same lock:

Reproduction Steps

I do not have an easy way to consistently reproduce the issue but I have a process dump of the deadlocked state, if it helps.

Expected behavior

No deadlocks when doing EventSource related opterations (in this case when accessing Activity.Current, which appears to use EventSource internally, though the expectation is general for all EventSource related operations).

Actual behavior

Deadlock occurs.

Regression?

No response

Known Workarounds

No response

Configuration

Windows 11 x64.

Reproduced both with .NET 7 and .NET 8 preview 3.

Other information

I have a process memory dump, if it helps. Ping me on Teams (sasaares).

Author:	sandersaares
Assignees:	-
Labels:	area-System.Diagnostics.Tracing, area-Tracing-coreclr, untriaged
Milestone:	-

[tommcdon]

@davmason possibly related to #86233

[davmason]

Thanks for reporting this, @sandersaares. I was hoping to get to this sooner but it will be at least another two weeks before I will have a fix. I am still hoping to get a fix in for 8.