[Bug]: MSSQL (primarily Linux) pipeline flakey

[seantleonard]

What happened?

MSSQL pipeline is flakely , typically in Linux tests, but sometimes windows, where all integration tests pass, but there is either a timeout or connection forcibly closed error. There may be a memory leak or inefficient use of resources occuring in test fixture which should be identified and remediated.

Why this is impactful:

1. Affects iteration velocity -> may allow quicker pace of merges because pipeline execution speed may be increased due to this and multiple runs don't need to be triggered

Version

latest main

What database are you using?

Azure SQL

What hosting model are you using?

Local (including CLI)

Which API approach are you accessing DAB through?

REST

Relevant log output

Windows:

Passed!  - Failed:     0, Passed:  1010, Skipped:     2, Total:  1012, Duration: 11 m 55 sTest Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Exception: An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.
 - Azure.DataApiBuilder.Service.Tests.dll (net6.0)
##[error]Error: The process 'C:\hostedtoolcache\windows\dotnet\dotnet.exe' failed with exit code 1

Code of Conduct

• I agree to follow this project's Code of Conduct

[seantleonard]

Working on using preview feature of AppService to run profiler (apparently different perf measurements/and suggestions than what can be gathered from Visual Studio).
Requires publishing DAB to app service, executing requests and checking profiler output.

[seantleonard]

[current] I'm adding some more diagnostics (via dotnet test --diagto try and identify the issue in addition to having :
Public GitHub issues(1, 2, 3) range from incorrect async function declarations, test machine OOM, and/or other code defects.
Issue is not consistent and pipeline takes 20-30 moinutes to execute. ( i briefly looked at what tests could be modified to improve test execution times, but that is a larger effor that may not fit within GA timeframe, so i stopped that line of investigation currently.)