Test: System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1) failed in CI

[KristinXie1]

Failed test: System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1)
Configuration: OuterLoop_CentOS7.1_debug (build#123)

Message:

System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1) [FAIL]
        Assert.True() Failure
        Expected: True
        Actual:   False

Stack Trace:

/mnt/resource/j/workspace/dotnet_corefx/master/outerloop_centos7.1_debug/src/System.Net.Sockets/tests/FunctionalTests/SendReceive.cs(148,0): at System.Net.Sockets.Tests.SendReceive.<SendToRecvFrom_Datagram_UDP>d__17.MoveNext()
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           --- End of stack trace from previous location where exception was thrown ---
              at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
              at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_debug/123/consoleText

[KristinXie1]

This issue is repro on OuterLoop_CentOS7.1_release (build#123), detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_release/123/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[karelz]

cc @Priya91 @ianhays @steveharter

[KristinXie1]

Failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_debug/134/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[steveharter]

@stephentoub this test was added about three weeks ago. Any thoughts? We should probably disable it.

  int sent = await SendToAsync(right, new ArraySegment<byte>(sendBuffer), leftEndpoint);
>>Assert.True(await receiverAck.WaitAsync(AckTimeout));
  senderAck.Release();

Also there are two Assert.True's here without any userMessage param so that should be added.

[stephentoub]

this test was added about three weeks ago

The test itself has actually been in the repo since 2015:
dotnet/corefx@09932a4#diff-5d77ee23eb2ea596e218f9c1ef09d793R22

What changed a few weeks ago was allowing the test to work with the various Async APIs on Socket, e.g. allowing it to work with the EAP, Task, and APM methods (and the sync ones), rather than just the APM ones. It's possible an issue was introduced as part of that conversion.

[KristinXie1]

Failed again on build 20170322.02: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170322.02/workItem/System.Net.Sockets.Tests/analysis/xunit/System.Net.Sockets.Tests.SendReceiveApm~2FSendToRecvFrom_Datagram_UDP(loopbackAddress:%20::1)

[KristinXie1]

Failed again on build 20170324.01

[KristinXie1]

Failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_centos7.1_release/138/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[stephentoub]

What changed a few weeks ago was allowing the test to work with the various Async APIs on Socket

@steveharter, actually, FYI, it looks like the test has failed not just on EAP but also on APM, which was the one that previously existed.

[stephentoub]

@steveharter, this looks like the same problem as was fixed for some other tests in https://github.com/dotnet/corefx/issues/5185. The test is sending 10 packets over UDP and expecting all 10 to be received, which isn't guaranteed. There's even a TODO in the test stating that it needs to be hardened against such loss:
https://github.com/dotnet/corefx/blob/ed823ad9470f2ecf412d5089fe36cd6958fc5834/src/System.Net.Sockets/tests/FunctionalTests/SendReceive.cs#L71

[steveharter]

What changed a few weeks ago was allowing the test to work with the various Async APIs on Socket

@steveharter, actually, FYI, it looks like the test has failed not just on EAP but also on APM, which was the one that previously existed.

FWIW according to jdash, the tests started failing on 3/9/2017. The refactoring work was on 2/26/2017 - dotnet/corefx@ca392ca#diff-5d77ee23eb2ea596e218f9c1ef09d793

Test Case: System.Net.Sockets.Tests.SendReceiveEap.SendToRecvFrom_Datagram_UDP(loopbackAddress: ::1)
Failed Jenkins Jobs
Build Number 	Machine Name 	Date
dotnet_corefx/master/outerloop_centos7.1_debug 123 	centos71-20170216-outer1fbe70 	03/09 07:52 AM
dotnet_corefx/master/outerloop_centos7.1_debug 125 	centos71-20170216-outer125790 	03/11 07:52 AM
dotnet_corefx/master/outerloop_centos7.1_release 123 	centos71-20170216-outer6ffb00 	03/12 03:32 PM

[steveharter]

I believe many timeouts or proposed UDP packet loss are interference from other tests that happen to listen on another tests port, where tests do a receivefrom on a port that they did not bind to (IPV4\IPV4 dual mode tests), so the 'bad' test receives the data causing the 'good' test to timeout or miss some data. This is mostly a Linux issue due to the randomness in port assignment (vs Windows which is incremental).

So if the refactoring isn't to blame, perhaps a new or other modified test is (that was added\modified on or a few days before 3/9/2017).

[KristinXie1]

Failed again here: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_centos7.1_release/7/testReport/System.Net.Sockets.Tests/SendReceiveEap/SendToRecvFrom_Datagram_UDP_loopbackAddress____1_/

[KristinXie1]

This issue is repro on portable core tests: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fportable~2Fcli~2F/build/20170407.01/workItem/System.Net.Sockets.Tests/analysis/xunit/System.Net.Sockets.Tests.SendReceiveApm~2FSendToRecvFrom_Datagram_UDP(loopbackAddress:%20::1)

[steveharter]

Stress testing Windows 10 resulted in this test starting but not finishing (so not considered a failure in CI reports), unlike Linux which does fail in CI. For the Windows 10 repro, a background exception was reported that may have originated from that test, or from the other test that was currently running: System.Net.Sockets.Tests.DualModeConnectToIPAddressArray.DualModeConnect_IPAddressListToHost_Throws

System.Net.Sockets.Tests.SendReceiveApm.SendToRecvFrom_Datagram_UDP(loopbackAddress: 127.0.0.1) [STARTING]
...
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

For Linux, possible interference from the DualMode test which mixes\matches IPv4\v6 addresses expecting failures, which may randomly have port collisions on linux so those should be disabled.

[karelz]

Overall it looks like 1/week failure rate ... borderline for addressing it in 2.0. Given that this is most likely just bad test and we're tight on workforce for 2.0, we will keep it in Future - I marked it as 'wishlist' as it should be on top of our Future backlog.

[wfurt]

I could not find any recent failure. Please reopen if this fails again.

[krwq]

No failures because test case is disabled:

System.Net.Sockets/tests/FunctionalTests/SendReceive.cs:41

[antonfirsov]

This is a major blocker for implementing #938, since we need a robust way to test UDP, if we want to cover those changes.

[antonfirsov]

This failure is likely not about port stealing or any other direct interference with other tests.

The test tends to fail when CPU load is high. I can reproduce the failure by running the Theory's cases on a 2 VCPU Linux system in parallel with one single other test case that calculates PI.