UDP tests fail with timeout sending/receiving data over loopback

[mellinoe]

Hit a timeout in this test:

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_debug_prtest/565/testReport/junit/System.Net.Sockets.Tests/DualMode/ReceiveFromAsyncV6BoundToSpecificV6_Success/

Stacktrace

MESSAGE:
System.TimeoutException : The operation has timed out.
+++++++++++++++++++
STACK TRACE:
at System.Net.Sockets.Tests.DualMode.ReceiveFromAsync_Helper(IPAddress listenOn, IPAddress connectTo) in d:\j\workspace\dotnet_corefx\windows_nt_debug_prtest\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs:line 599 at System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV6BoundToSpecificV6_Success() in d:\j\workspace\dotnet_corefx\windows_nt_debug_prtest\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs:line 536

[mellinoe]

@CIPop , @pgavlin

[SidharthNabar]

Need to investigate this test - this seems to be a test setup/infra issue, hence moving to RTM milestone.

[CIPop]

Similar to the UDP test: dotnet/corefx#5411

Test timed out. This is indicative of CPU/mem/tcp pressure on the CI machine and since it's intermittent I'll tentatively mark it as a "test bug".

To confirm we'll requires a repro with ETW traces enabled either in the CI (we need #15570 for this) or in a private environment.

[stephentoub]

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_debug_prtest/1569/console

10:06:54      System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV6BoundToSpecificV6_Success [FAIL]
10:06:54         System.TimeoutException : The operation has timed out.
10:06:55         Stack Trace:
10:06:55            d:\j\workspace\dotnet_corefx\windows_nt_debug_prtest\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(601,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsync_Helper(IPAddress listenOn, IPAddress connectTo)
10:06:55            d:\j\workspace\dotnet_corefx\windows_nt_debug_prtest\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(538,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV6BoundToSpecificV6_Success()

[ericeil]

The common theme here seems to be failures in ReceiveFromAsync_Helper, which is using UDP, not TCP. I've changed the title accordingly.

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_release_prtest/3095/consoleFull

06:05:27      System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV4BoundToAnyV4_Success [FAIL]
06:05:28         System.TimeoutException : The operation has timed out.
06:05:28         Stack Trace:
06:05:28            d:\j\workspace\windows_nt_re---5067c899\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(590,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsync_Helper(IPAddress listenOn, IPAddress connectTo)

[stephentoub]

More here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_debug_prtest/3144/consoleFull

18:14:30      System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV4BoundToAnyV4_Success [FAIL]
18:14:30         System.TimeoutException : The operation has timed out.
18:14:30         Stack Trace:
18:14:30            d:\j\workspace\windows_nt_de---06b7984d\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(603,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsync_Helper(IPAddress listenOn, IPAddress connectTo)
18:14:30            d:\j\workspace\windows_nt_de---06b7984d\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(528,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV4BoundToAnyV4_Success()
18:14:30      System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV4BoundToSpecificV4_Success [FAIL]
18:14:30         System.TimeoutException : The operation has timed out.
18:14:30         Stack Trace:
18:14:30            d:\j\workspace\windows_nt_de---06b7984d\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(603,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsync_Helper(IPAddress listenOn, IPAddress connectTo)
18:14:30            d:\j\workspace\windows_nt_de---06b7984d\src\System.Net.Sockets\tests\FunctionalTests\DualModeSocketTest.cs(521,0): at System.Net.Sockets.Tests.DualMode.ReceiveFromAsyncV4BoundToSpecificV4_Success()

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_release_prtest/3307/consoleFull

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_debug/838/consoleFull

[stephentoub]

Again here:
http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/windows_nt_release_prtest/3367/consoleFull

[ericeil]

Looks like another instance last night: http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/outerloop_windows_nt_release/124/

I'll work on a fix.

[ericeil]

With dotnet/corefx#7112, these should be much less likely. There are three test cases left to address, but we haven't seen these failures in any of those cases so far. We can clean up the remaining cases in RTM.

[ericeil]

The remaining tests don't seem to have failed in the last month, so hardening those isn't looking particularly urgent. Moving this issue out of this release for now.

[stephentoub]

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/release_1.0.0/job/windows_nt_release_prtest/44/consoleText

[ericeil]

The overall problem here seems basically solved. Let's file separate issues for any similar failures that come up in the future.