Device unauthorized errors during Android device CI runs

[steveisok]

Example build: https://dev.azure.com/dnceng/public/_build/results?buildId=956492&view=results

Example log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-47057-merge-86b88422cf374fa882/Microsoft.Extensions.Configuration.Json.Tests/console.d1b0ae97.log?sv=2019-07-07&se=2021-02-08T18%3A02%3A12Z&sr=c&sp=rl&sig=DtoVG0NJIQ9Th%2BgD5IONKr3jkRUFCMvsp6U61UZ9QtE%3D

Adb seems to be claiming that the device is unauthorized, which causes a bunch of tests to not run and fail. I'm not sure if this is machine specific or device specific. I'm seeing numerous BadExit's on DNCENGWIN-036 and 30.

[10:02:24] fail: Error trying to get device architecture: error: device unauthorized.
                 This adb server's $ADB_VENDOR_KEYS is not set
                 Try 'adb kill-server' if that seems wrong.
                 Otherwise check for a confirmation dialog on your device.
                 
[10:02:24] crit: Failure to run test package: Object reference not set to an instance of an object.
                 System.NullReferenceException: Object reference not set to an instance of an object.
                    at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.<>c__DisplayClass17_0.<GetDeviceToUse>b__0(KeyValuePair`2 kvp) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 259
                    at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
                    at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.GetDeviceToUse(ILogger logger, AdbRunner runner, String apkRequiredArchitecture) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 259
                    at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.InvokeInternal(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 99
XHarness exit code: 71

C:\h\w\A9ED09B0\w\B03109F7\e>set _commandExitCode=71 

[ghost]

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Example build: https://dev.azure.com/dnceng/public/_build/results?buildId=956492&view=results

Example log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-47057-merge-86b88422cf374fa882/Microsoft.Extensions.Configuration.Json.Tests/console.d1b0ae97.log?sv=2019-07-07&se=2021-02-08T18%3A02%3A12Z&sr=c&sp=rl&sig=DtoVG0NJIQ9Th%2BgD5IONKr3jkRUFCMvsp6U61UZ9QtE%3D

Adb seems to be claiming that the device is unauthorized, which causes a bunch of tests to not run and fail. I'm not sure if this is machine specific or device specific. I'm seeing numerous BadExit's on DNCENGWIN-036 and 30.

[10:02:24] fail: Error trying to get device architecture: error: device unauthorized.
                 This adb server's $ADB_VENDOR_KEYS is not set
                 Try 'adb kill-server' if that seems wrong.
                 Otherwise check for a confirmation dialog on your device.
                 
[10:02:24] crit: Failure to run test package: Object reference not set to an instance of an object.
                 System.NullReferenceException: Object reference not set to an instance of an object.
                    at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.<>c__DisplayClass17_0.<GetDeviceToUse>b__0(KeyValuePair`2 kvp) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 259
                    at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
                    at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.GetDeviceToUse(ILogger logger, AdbRunner runner, String apkRequiredArchitecture) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 259
                    at Microsoft.DotNet.XHarness.CLI.Commands.Android.AndroidTestCommand.InvokeInternal(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/Android/AndroidTestCommand.cs:line 99
XHarness exit code: 71

C:\h\w\A9ED09B0\w\B03109F7\e>set _commandExitCode=71 

Author:	steveisok
Assignees:	-
Labels:	area-Infrastructure-mono, os-android
Milestone:	-

[steveisok]

/cc @MattGal

[MattGal]

Sure seems device-specific ... @akoeplinger any suggestions? We can just teach it to try the suggested workaround of kill-server in this case too.

[akoeplinger]

Hm we should try if killing the adb server works, but it's possible that the device lost authentication and someone needs to re-auth it on the phone.

[ViktorHofer]

Most likely mitigated with fixes mentioned in https://github.com/dotnet/core-eng/issues/11891.

[mdh1418]

I'm seeing this happening for the following test suites

System.ComponentModel.Primitives.Tests.csproj
System.Diagnostics.StackTrace.Tests.csproj
System.Linq.Parallel.Tests.csproj
System.Linq.Queryable.Tests.csproj
System.Linq.Tests.csproj
System.Memory.Data.Tests.csproj
System.Memory.Tests.csproj
System.Net.Connections.Tests.csproj
System.Private.Uri.ExtendedFunctional.Tests.csproj
System.Resources.Extensions.Tests.csproj
System.Runtime.Serialization.Json.ReflectionOnly.Tests.csproj
System.Runtime.Serialization.Xml.ReflectionOnly.Tests.csproj
System.Security.Claims.Tests.csproj
System.Security.Cryptography.Encoding.Tests.csproj
System.Security.Cryptography.X509Certificates.Tests.csproj
System.Security.Cryptography.Xml.Tests.csproj
System.Xml.Linq.SDMSample.Tests.csproj
System.Xml.RW.SubtreeReader.Tests.csproj

based on https://dev.azure.com/dnceng/public/_build/results?buildId=1093022&view=logs&j=c693b1a1-52cc-5241-304f-e4b4c2d5f73e&t=ca6cd8a5-c31d-5a98-8419-f200a8c4128d from test runs in #50095

[MattGal]

@mdh1418 it's more likely related to specific devices versus the tests but we'll take a look and get an issue created if so.

[akoeplinger]

@MattGal this ended up being the devices right: https://github.com/dotnet/core-eng/issues/12841 ?

[MattGal]

@MattGal this ended up being the devices right: dotnet/core-eng#12841 ?

I think so... but they seem to be going bad more often. Is it possible that a sufficiently weird APK could cause the problem? (@parose1 would be interested in the answer)

[akoeplinger]

Hard to say, we'd need to know in what state the device is then we could maybe track down how it got there by correlating which workitems ran around the time it went bad.