Benchmark Reporting - Deadlock/Getting Stuck Reading From Process.StandardOutput (NetCore 2.1) #828
Comments
|
Starting to see some correlation, the dotnet host never finishes / terminates. CodeGen is throwing errors for files being kept open... Exact same test now fails to build... very curious. Error |
|
When in doubt... run simple tests again. Sleep benchmark fails. public class Sleep
{
[Benchmark]
public void Time50() => Thread.Sleep(50);
[Benchmark(Baseline = true)]
public void Time100() => Thread.Sleep(100);
[Benchmark]
public void Time150() => Thread.Sleep(150);
} |
houseofcat
changed the title
houseofcat
changed the title
|
@adamsitnik could you take a look? |
Hi @thyams . The proposal looks good to me, could you send a PR and include a link to sample StackOverflow issue? So far I have seen such behavior when |
|
@adamsitnik Done! |
* Minor fix Potential fix for SynchronousProcessOutputLoggerWithDiagnoser infinite loop. Added time in Execute, detected freeze during UnitTest. * Adding timeout to basestream read. * Stream doesn't support timeout. * Enhancing ProcessInput * Minor tweak. Disabling Executor timeout in Toolchains. Adjusting While loop and comments on Process
|
@adamsitnik No dice! I will keep stabbing at it. Poking CoreFx team (https://github.com/dotnet/corefx/issues/31138) |
|
@adamsitnik How do I force BenchmarkDotNet to run in Debug mode, I am detecting an error writing/reading to files during simple tests. This isn't working... public class DebugConfig : ManualConfig
{
public DebugConfig()
{
Add(JitOptimizationsValidator.DontFailOnError); // ALLOW NON-OPTIMIZED DLLS
}
} |
public class DebugConfig : ManualConfig
{
public DebugConfig()
{
Add(JitOptimizationsValidator.DontFailOnError); // ALLOW NON-OPTIMIZED DLLS
Add(Job.Default
.WithCustomBuildConfiguration("Debug") // to compile in DEBUG
.With(InProcessToolchain.Instance)); // to debug the benchmark in-process
// add columns, exporters, loggers, diagnosers etc
}
} |
…, dotnet#828 * Minor fix Potential fix for SynchronousProcessOutputLoggerWithDiagnoser infinite loop. Added time in Execute, detected freeze during UnitTest. * Adding timeout to basestream read. * Stream doesn't support timeout. * Enhancing ProcessInput * Minor tweak. Disabling Executor timeout in Toolchains. Adjusting While loop and comments on Process
|
I am back @adamsitnik! I have identified the code that causes issues in a simple Solution file! It all started when I added delays / sleeps to compare runtimes in parallel. In my own personal library, I have a ThrottleSmallBodyLoops (with has an await Task.Delay(random 0 through 2)). This prevents TPL from choking with mass parallel processes. The code that causes the benchmark to freeze is This is what I believe hangs your CI jobs too! Solution is attached! I maybe able to do a PR later for you guys again :) |
|
Hi @houseofcat ! I talked with @stephentoub about your code sample. Here is what Stephen wrote:
|
|
I will double check but the issue occurred for me even on 1 string iterations. Or when the only thing the test does is Being frozen on the //After All portion of Benchmark DotNet (reading/writing to file) would be something else than my test still running - at least how it looks like to me. |
You're referring to the AsParallel? That's part of the LINQ query, which is operating on only 12 items as far as it's concerned (the 12 partitions), regardless of how many elements are in the input. And that AsParallel is largely useless, anyway, as the only thing it's parallelizing is the invocation of the Select, which processes at most 12 items, and for each of those 12, it's basically just invoking the async method (which yields immediately due to the Delay) and immediately returning a Task for it, so there's really nothing there to parallelize. Regardless of that AsParallel, you are going to end up with 12 asynchronous processing loops running concurrently, and so with 1,000,000 items and 12 concurrent processors (each of which spends most of its time waiting for timers to fire), you'd expect it to take somewhere around 20 minutes. |
What I was trying to test doesn't super translate well to the demo Benchmark code I wrote. So I will have to write a more thoughtful test to seem less "stupid" but suffice to say the partitioning code is very performant and utilizes very little system memory. For good reason too, I took it from your article from back in time: So enough butt kissing! The goal was to limit degrees of parallelism while iterating. I modified it to use PLINQ instead of The real production code is asynchronously publish to RabbitMQ and/or saving to the DB. As of right now, our devs have a terrible design pattern. Put everything in a I was trying to prove what I was seeing that when there was a Parallel.ForEach (with max degrees of parallelism) or slower iterator as designed, the overall system performance was far greater. I analogously compared it to the tuning of the pistons of an engine. What's happening here is that the tests run, and finish, but when generating the report it freezes (but only when there is a Sleep or Delay.) It froze previously in my CookedRabbit library and this code wasn't called but in my get Channels I have an EDIT EDIT 2 https://user-images.githubusercontent.com/1414138/42740075-e8a4afe0-886f-11e8-9178-7e21727d28f9.png Being called from here: EndOfStream, sub call Get. The Process StandardOutput or StandardError have a thread deadlock situation and WaitForExit() runs infinitely due to the Dotnet Processes are not terminating. Related issue #885 So I think a part of this is => what mechanism does a dotnet host project determine it's finished and properly terminate? Because I do have a keep heartbeat alive in the background when this happens... could this prevent the dotnet host from terminating? |
|
Based on my latest step through, it is 100% caused by Benchmark Processes not terminating under dotnet host thus not releasing the EndOfStream. Since my test code produces no infinite loops, I have to assume a deadlock on StandardOutput, StandardError, or both. Manually terminating the process "unfreezes" the lock, however, no results are passed through to the main logger host thus a report can't be generated. My stackoverflow research indicates that a full buffer could cause a deadlock scenario as described. I will move code to an event driven paradigm on data received to test a buffer overflow to see if it works. |
houseofcat
changed the title
|
I converted the entire process to event driven paradigm on processing StandardOutput and StandardError. Freezes at the same spot. I pulled everything into my library to step through every component available to me. Does BenchmarkDotNet not support C# 7.1+? I have my code set to latest but when everything is integrated together, I get an error stating my C# feature set is too high specifically the use of |
|
@houseofcat if your project file (.csproj/vbproj/fsproj) with benchmarks defines a but it does not work when |
|
I will have a look at it after work! Thanks! |
|
Today i have some problem after update VS2019 to 16.3.4. Benchmark frozen at: |
|
The task manager shows that the benchmark is doing something |
|
Hi @Nim Did you select some part of the printed text in console? By default Windows OS let's the app run then, but blocks the printing to standard output. Also, you can always attach debugger to the benchmark process and see what is going on:
You can read more about troubleshooting BDN in our docs: https://benchmarkdotnet.org/articles/guides/troubleshooting.html |
|
@adamsitnik I apologize for disturbing you. Freezing was my fault. But I improved the helper class for multi-threaded tests. Later I can share it so that you add an example to the documentation. If it is interesting. |
Update (7/15/2018 @ 7:25 PM EST)
Decoupled my Benchmark project from a local project and instead added reference to the NuGet it corresponds to.
In addition to that, I installed 2.302 SDK (
NetCore 2.1.2).Standard Sleep benchmark is working again.
Moving to BenchmarkDotNet nightlies (BDN v0.10.14.685, Core v0.10.14.508).
Update (7/15/2018 @ 8:07 PM EST)
Nothing seems to work.
Thread stuck here:
Found the exception in background:
Not sure how I can help but the file it's stuck on seems to be the .log.
I allowed continuation found the code frozen in one spot - continuously frozen in one spot:
Guessing this needs a timeout?
BenchmarkDotNet.Loggers.SynchronousProcessOutputLoggerWithDiagnoser Line 38There appears to be a couple of Stackoverflow open issues which indicate that they experienced infinite loops in this same manner.
Code Change Proposal:
Asynchronous would be considered a better choice and adding a cancellation token on a timer.
Original Post
The text was updated successfully, but these errors were encountered: