Encoding.UTF8.GetString: The output char buffer is too small...

[joshlang]

NET 3.1

Feeding some garbage data into Encoding.UTF8.GetString, I occasionally get this exception:
System.ArgumentException: 'The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'. (Parameter 'chars')'

Stack trace:

   at System.Text.Encoding.ThrowCharsOverflow()
   at System.Text.Encoding.ThrowCharsOverflow(DecoderNLS decoder, Boolean nothingDecoded)
   at System.Text.Encoding.GetCharsWithFallback(ReadOnlySpan`1 bytes, Int32 originalBytesLength, Span`1 chars, Int32 originalCharsLength, DecoderNLS decoder)
   at System.Text.UTF8Encoding.GetCharsWithFallback(ReadOnlySpan`1 bytes, Int32 originalBytesLength, Span`1 chars, Int32 originalCharsLength, DecoderNLS decoder)
   at System.Text.Encoding.GetCharsWithFallback(Byte* pOriginalBytes, Int32 originalByteCount, Char* pOriginalChars, Int32 originalCharCount, Int32 bytesConsumedSoFar, Int32 charsWrittenSoFar)
   at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount)
   at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding)
   at System.Text.Encoding.GetString(ReadOnlySpan`1 bytes)
   at ...Our code...

I was accidentally using it on binary data while doing some debugging.

It happened using the ReadOnlySpan overload, but these byte arrays are the data:

new byte[] { 1,6,104,97,115,104,116,120,1,32,125,34,109,16,93,67,238,247,111,164,124,222,214,204,168,122,3,194,48,190,150,196,26,251,175,111,65,1,97,0,30,248,0,4,207,225,40,0 } ;
new byte[] { 1,6,104,97,115,104,116,120,1,32,146,49,170,24,95,32,149,96,115,203,170,163,193,27,202,95,134,122,121,85,104,162,76,212,216,15,177,166,148,80,204,57,0,4,222,80,41,0 } ;

Neither of these is reproducible though.

Even when visual studio has the "Exception Thrown!" exception window open and highlighted, re-executing the same statement doesn't throw an exception the second time.

My suspicion would be something to do with ArrayPool, if that's even used by this method.

Not sure how to be more useful with this one...

[GrabYourPitchforks]

Normally this means that the input buffer's contents have changed in the middle of the operation. The Encoding.GetString(ReadOnlySpan<char>) API performs two passes: the first pass to calculate the necessary string length, and the second pass to perform the actual conversion. If the data within the byte buffer changes between these two operations, the conversion routine might fail because the number of resulting output chars could have increased.

[GrabYourPitchforks]

@joshlang - double-check that another thread in your application isn't mutating the buffers passed to this API while the operation is in progress. If you're seeing that this exception still occurs with static data, please let me know so that we can investigate further.

[joshlang]

I just did. I've confirmed that it's probably what happened. I was calling TrySetResult on a TaskCompletionSource before doing the ghetto debug outputting statements that caused the exception.

You're a genius. I'll close this issue. Thanks!