NotSupportedException & InvalidOperationException for unknown reasons

[joshlang]

I'm lumping these two bugs into one, because I suspect they have the same root cause.

.NET Standard 2.1
Running in Azure AKS (linux)
System.Data.SqlClient version 4.8.1
Connecting to a tiny little sql azure database instance

We've run into two issues:

Issue 1

InvalidOperationException: Invalid operation. The connection is closed.

^-- FOREVER!

Our code is exceedingly simple (flattened):

using var connection = new SqlConnection(ConnectionString);
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
using var cmd = connection.CreateCommand();
cmd.CommandText = command;
using var r = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
// consume reader results
// end of method

Stack trace:

System.InvalidOperationException:
   at System.Data.SqlClient.TdsParserStateObject.WritePacket (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParserStateObject.WriteBytes (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParser.WriteString (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.SqlCommand.BeginExecuteReader (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Data.SqlClient.SqlCommand.ExecuteReaderAsync (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [our code]

Sometimes, it gets stuck in an infinite loop of doing that. In other words, every attempt to execute this code would throw that same exception.

We fixed it by wrapping it like this:

try
{
...
}
catch (InvalidOperationException)
{
    SqlConnection.ClearAllPools();
    throw;
}

To me, it looks like a SqlConnection gets into an unstable state, gets put back into the pool and then explodes every time it's taken from the pool.

I'd say it looks like a race condition with a forgotten using statement or something... but... no. There aren't even any concurrent calls happening.

Issue 2

NotSupportedException: A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 35 - An internal exception was caught) The WriteAsync method cannot be called when another write operation is pending.

Stack trace:

System.Data.SqlClient.SqlException:
   at System.Data.SqlClient.TdsParserStateObject.SNIWritePacket (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParserStateObject.WriteSni (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParserStateObject.WritePacket (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParserStateObject.ExecuteFlush (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.TdsParser.TdsExecuteSQLBatch (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Data.SqlClient.SqlCommand.BeginExecuteReader (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Data.SqlClient.SqlCommand.ExecuteReaderAsync (System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Symetria.Sql.Core.SqlBase+<ExecuteQueryAsync>d__9.MoveNext (Symetria.Sql.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullSymetria.Sql.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /src/Symetria.Sql.Core/SqlBase.csSymetria.Sql.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 96)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [our code ... same as above]

Our application insights shows 183 exceptions of Issue 1 and only 4 exceptions of this Issue 2 over the past 24 hours.

I'm somewhat at a loss on how to debug this or provide more useful information. Open to suggestions.

[joshlang]

Ah, and I see now you've ported to Microsoft.Data.SqlClient, so I'll try that instead.

[joshlang]

The issue also occurs in Microsoft.Data.SqlClient, so re-opening this issue.

Updated stack traces:

Issue 1

System.InvalidOperationException:
   at Microsoft.Data.SqlClient.TdsParserStateObject.WritePacket (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ExecuteFlush (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParser.TdsExecuteSQLBatch (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.BeginExecuteReaderInternal (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand+<>c__DisplayClass171_0.<ExecuteReaderAsync>b__0 (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteReaderAsync (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [our code]

Issue 2

Microsoft.Data.SqlClient.SqlException:
   at Microsoft.Data.SqlClient.TdsParserStateObject.SNIWritePacket (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParserStateObject.WriteSni (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParserStateObject.WritePacket (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ExecuteFlush (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParser.TdsExecuteSQLBatch (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.BeginExecuteReaderInternal (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand+<>c__DisplayClass171_0.<ExecuteReaderAsync>b__0 (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteReaderAsync (Microsoft.Data.SqlClient, Version=2.0.20021.1, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at [our code]

[cheenamalhotra]

Hi @joshlang

Could you please upload your repro app that we can take a look at to understand dependencies?

[joshlang]

Hi Cheena,

Sorry, I'm not going to be able to do that for you any time soon.

[DavoudEshtehari]

According to investigations, I wrote a sample application to test the issue. You can change the connection string and the number of records and number of run it on your own.
UnknownException459.zip

[DavoudEshtehari]

The ExecuteScalarAsync and ExecuteReaderAsync have issue by Azure SQL Server and enabled managed SNI. The Netfx does not produce this error.

[DavoudEshtehari]

PR #579 is prepared to fix this issue and is depending to fix the issue #38185 at dotnet/runtime.