Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EventPipe sessions can leak the EventPipeThreadSessionState* associated with them #76430

Closed
davmason opened this issue Sep 30, 2022 · 2 comments

Comments

@davmason
Copy link
Member

davmason commented Sep 30, 2022

Description

As reported by a partner team, we have a latent bug in EventPipe where the session can leak the associated thread session state. The EventPipeThreadSessionStates are freed when the buffer for that thread is exhausted, so if a buffer is never created the state is never freed.

The sequence of events:

  • Enough threads are in use that the buffers are at maximum capacity
  • A thread tries to write an event and it creates the thread local session state
  • We see that too many buffers are in use and drop the event
  • The session ends before an event is successfully written to that thread, i.e. all are dropped and a buffer is never created
  • The session is disabled, and the code that deletes the thread local session state never sees this thread because a buffer was never created
  • At some later time another session is created, an event is written and we do not recreate the thread local session state since it still exists from previously
  • This time the thread successfully writes an event and a buffer is created
  • When this second session shuts down it tries to clean up the thread local session state, but it is still pointing to the old session which has been deleted and now is being used by other native code for other purposes, so it contains garbage
  • We try to use an index that causes us to access invalid memory and AV
@davmason davmason added this to the 8.0.0 milestone Sep 30, 2022
@davmason davmason self-assigned this Sep 30, 2022
@davmason
Copy link
Member Author

Sample app that will hit this reliably:

using System.Reflection.Emit;
using System.Reflection;
using System.Diagnostics.Tracing;
using System.Diagnostics;
using System;
using System.Threading.Tasks;
using System.Collections.Generic;
using System.Threading;

class MyEventListener : EventListener
{
    private static object s_consoleLock = new object();

    protected override void OnEventSourceCreated(EventSource eventSource)
    {
        base.OnEventSourceCreated(eventSource);

        if (eventSource.Name.Equals("Microsoft-Windows-DotNETRuntime"))
        {
            // Keyword 0x10 is JIT events, 0x1 is GC
            EventKeywords keywords = (EventKeywords)(0x10 | 0x1);
            EnableEvents(eventSource, EventLevel.Verbose, keywords);
        }
    }

    protected override void OnEventWritten(EventWrittenEventArgs eventData)
    {
        base.OnEventWritten(eventData);
    }
}

class Program
{
    delegate long SquareItInvoker(int input);

    delegate TReturn OneParameter<TReturn, TParameter0>
        (TParameter0 p0);

    static void Main(string[] args)
    {
        Console.WriteLine("Starting");

        // Need to declare this here to turn on activity tracking
        using MyEventListener listener = new MyEventListener();

        int eventListenerThreads = 5;
        int jitThreads = 1000;

        List<Thread> threads = new List<Thread>();
        // Threads that constantly start and stop sessions
        for (int i = 0; i < eventListenerThreads; ++i)
        {
            Thread t = new Thread(() =>
            {
                while (true)
                {
                    using (MyEventListener listener = new MyEventListener())
                    {
                        Thread.Sleep(1);
                    }
                }
            });
            t.Start();
            threads.Add(t);
        }

        // A bunch of threads doin stuff
        for (int i = 0; i < jitThreads; ++i)
        {
            Thread t = new Thread(() =>
                {
                    while (true)
                    {
                        // Generate some JIT activity
                        MakeDynamicMethod();
                    }
                });
            t.Start();
            threads.Add(t);
        }

        foreach (Thread t in threads)
        {
            t.Join();
        }
    }

    static void GenerateJITEvents()
    {
    }

    static void MakeDynamicMethod()
    {
        AssemblyName name = new AssemblyName(GetRandomName());
        AssemblyBuilder dynamicAssembly = AssemblyBuilder.DefineDynamicAssembly(name, AssemblyBuilderAccess.RunAndCollect);
        ModuleBuilder dynamicModule = dynamicAssembly.DefineDynamicModule(GetRandomName());

        Type[] methodArgs = { typeof(int) };

        DynamicMethod squareIt = new DynamicMethod(
            "SquareIt",
            typeof(long),
            methodArgs,
            dynamicModule);

        ILGenerator il = squareIt.GetILGenerator();
        il.Emit(OpCodes.Ldarg_0);
        il.Emit(OpCodes.Conv_I8);
        il.Emit(OpCodes.Dup);
        il.Emit(OpCodes.Mul);
        il.Emit(OpCodes.Ret);

        OneParameter<long, int> invokeSquareIt =
            (OneParameter<long, int>)
            squareIt.CreateDelegate(typeof(OneParameter<long, int>));

        Random random = new Random();
        invokeSquareIt(random.Next());
    }

    static string GetRandomName()
    {
        return Guid.NewGuid().ToString();
    }
}

@davmason
Copy link
Member Author

Fixed in main and 6.0, closing

@ghost ghost locked as resolved and limited conversation to collaborators Dec 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant