System.Memory.Tests Work Item fails without reports, long running Base64EncoderUnitTests.EncodeTooLargeSpan

[antonfirsov]

System.Buffers.Text.Tests.Base64EncoderUnitTests.EncodeTooLargeSpan test hang in in:

https://dev.azure.com/dnceng/public/_build/results?buildId=877008&view=ms.vss-test-web.build-test-results-tab
(net6.0-windows-Debug-x64-CoreCLR_release-Windows.10.Amd64.Server19H1.ES.Open)

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-44246-merge-d0435e106c844e1b8e/System.Memory.Tests/console.1eb45cee.log?sv=2019-07-07&se=2020-11-25T21%3A20%3A41Z&sr=c&sp=rl&sig=U3vXcd%2B3cmRt8pldKzKKdpi4Q0ADmohzWT9WnwwEFm0%3D

[stephentoub]

I suspect it's not actually a hang and that it was just taking a really long time encoding 2GB in the Base64.EncodeToUtf8 call. Unfortunately there's no dump included to confirm or deny that.

[antonfirsov]

In any case the "System.Memory.Tests Work Item" failed in OuterLoop, and not 100% sure, but it's probably not the first time I've seen this happening. If it occurs regularly we probably want to do something about it.

[GrabYourPitchforks]

Is there anything special about the machine this was running on? I wouldn't expect this test to take that long unless system memory was so tight that we were hitting the pagefile over and over and over again. Maybe we can use VirtualLock to play tricks here?

[danmoseley]

https://engsrvprod.kusto.windows.net/engineeringdata?query=H4sIAAAAAAAEAGWRQU%2fCQBCF7036H0YuQFJoqSQaknoQNcFEE0IN56U72gV2h8xugzX%2beLeAUOMedg%2fz5r1vZpfEm5lDbSEM4vgb9iUywpMyypYoIctACodOaeylSZoMRulgdAPJeJKmk3EyTK%2fHiT%2f9MPjXegeG9r3B6DaRffjrzgqN3NavQiNYJ9jZvXIldBa19SjDTsutLfUwv5IX1MT1MEfrbKft%2ffip3JQkwlUGycVn4YSrbOPQzf0sVLluU1yTMrBRRmbKGGR4ppUFMs07k5fuexamKA%2f5jO82LlFIG2vhUfiUvmNaY%2bFgXmGFDW10Rokgr3f%2bbs8SnfcUwZSMpS2%2bsYqOwREsT79yhLDEDlb1ZbUSbQGHSqW1YPWFUFBlXM8v2utaDMfkByo2yLn4CIMw%2bAFf8FUS8gEAAA%3d%3d&web=0

WorkItems 
| where Finished > now(-180d)  
| where FriendlyName == "System.Memory.Tests"
| where Status == 'Timeout'
| join kind=inner Jobs on JobId
| where Branch == "refs/heads/master"
| sort by Finished desc 
| summarize count()  by QueueName, Type, DockerTag

QueueName	Type	DockerTag	count_
windows.10.amd64.server20h1.open	test/functional/cli/outerloop/	mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-2004-helix-amd64-20200904200251-272704c	43
windows.10.amd64.serverrs5.open	test/functional/cli/outerloop/	mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504	14
windows.10.amd64.serverrs5.open	test/functional/cli/outerloop/		10
windows.10.amd64.server19h1.open	test/functional/cli/outerloop/		7
windows.81.amd64.open	test/functional/cli/outerloop/		4
windows.10.amd64.serverrs5.open.rt	test/functional/cli/outerloop/	mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504	4
windows.10.amd64.server19h1.open.rt	test/functional/cli/outerloop/		3
windows.10.amd64.open	test/functional/cli/innerloop/		2
ubuntu.1804.amd64.open	test/functional/cli/outerloop/		2
windows.10.amd64.serverrs5.open.rt	test/functional/cli/outerloop/		2
ubuntu.1604.amd64.open	test/functional/cli/innerloop/	mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7	1
windows.10.amd64.serverrs5.open	test/functional/cli/innerloop/		1
ubuntu.1604.amd64.open	test/functional/cli/innerloop/		1
windows.81.amd64.open.rt	test/functional/cli/outerloop/		1
windows.10.amd64.server20h1.open.rt	test/functional/cli/outerloop/	mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-2004-helix-amd64-20200904200251-272704c	1

Seems almost exclusively a Windows problem. I don't know how to get info on the memory though. Will have to ask engineering folks after holidays.

[danmoseley]

Also, I looked at 5-10 console logs and it was always System.Buffers.Text.Tests.Base64EncoderUnitTests.EncodeTooLargeSpan