Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Text.Json.SourceGeneration intermittent failures #72101

Closed
greenEkatherine opened this issue Jul 13, 2022 · 21 comments · Fixed by #94466
Closed

System.Text.Json.SourceGeneration intermittent failures #72101

greenEkatherine opened this issue Jul 13, 2022 · 21 comments · Fixed by #94466
Labels
area-Codegen-meta-mono disabled-test The test is disabled in source code against the issue os-android runtime-mono specific to the Mono runtime test-failure
Milestone

Comments

@greenEkatherine
Copy link
Contributor

greenEkatherine commented Jul 13, 2022

Frequency:

  • 4/1-7/22 ... 618 failures, first one starting on 7/5 ... 36 failures per day. Many of them in Official runs
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('System.Text.Json.SourceGeneration', '', true, 'NaN', true);

Assertions failed in several builds for tests:

Example failure:
Assert.Equal() Failure Expected: 1.10000002 Actual: NaN log results

cc @jtschuster

Runfo Tracking Issue: System.Text.Json failures on Mono_Release

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900783 runtime-extra-platforms PR 67861 net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900783 runtime-extra-platforms PR 67861 net7.0-iOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900774 runtime-extra-platforms PR 72545 net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900774 runtime-extra-platforms PR 72545 net7.0-tvOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900571 runtime-extra-platforms Rolling net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900571 runtime-extra-platforms Rolling net7.0-tvOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900134 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-tvOS-Release-arm64-Mono_Release-OSX.1015.Amd64.AppleTV.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899117 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899117 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1100.Arm64.Open console.log runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py

Displaying 100 of 1291 results

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
5 59 97
@greenEkatherine greenEkatherine added area-System.Text.Json blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' untriaged New issue has not been triaged by the area owner test-failure labels Jul 13, 2022
@ghost
Copy link

ghost commented Jul 13, 2022

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

Assertions failed in several builds for tests:

Example failure:
Assert.Equal() Failure Expected: 1.10000002 Actual: NaN log results

cc @jtschuster

Author: greenEkatherine
Assignees: -
Labels:

area-System.Text.Json, blocking-clean-ci, untriaged, test-failure

Milestone: -

@eiriktsarpalis
Copy link
Member

The failing assertion according to the build logs is this one:

Seems related to #69800. The output does not match the assertion inputs so wondering if this might be a runtime issue.

@eiriktsarpalis eiriktsarpalis added the runtime-mono specific to the Mono runtime label Jul 14, 2022
@jtschuster
Copy link
Member

Also hitting similar assert failures in System.Text.Serialization.Test.ContinuationTests
https://runfo.azurewebsites.net/view/build/?number=1869899

@danmoseley
Copy link
Member

Always and only Android and WASM. Result is NaN. Seems likely a Mono issue.
@vargaz @SamMonoRT thoughts?

@danmoseley
Copy link
Member

@marek-safar @steveisok thoughts about the right area path for this? It seems that it's only Android and WASM. It would be unlikely that there is JSON bug specific to those, especially relating to numerical operations.

@danmoseley danmoseley added arch-wasm WebAssembly architecture os-android labels Jul 20, 2022
@ghost
Copy link

ghost commented Jul 20, 2022

Tagging subscribers to 'arch-android': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

Assertions failed in several builds for tests:

Example failure:
Assert.Equal() Failure Expected: 1.10000002 Actual: NaN log results

cc @jtschuster

Runfo Tracking Issue: System.Text.Json failures on Mono_Release

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893157 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1893044 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892796 runtime-extra-platforms PR 72445 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892338 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892209 runtime-extra-platforms PR 72442 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892187 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892187 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892187 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892187 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892187 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1892187 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py

Displaying 100 of 941 results

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
8 47 62
Author: greenEkatherine
Assignees: -
Labels:

arch-wasm, area-System.Text.Json, blocking-clean-ci, os-android, untriaged, runtime-mono, test-failure

Milestone: -

@steveisok
Copy link
Member

@danmoseley For android, there is a known floating point issue that would be fixed by #65723. This only impacts x86 and is not very high on the priority list.

I think where we hit this on Android x86 we should skip.

@steveisok
Copy link
Member

/cc @lewing as he may have a better idea for wasm.

@marek-safar
Copy link
Contributor

This looks like codegen bug in MonoVM but it's strange that it's NaN instead of 1.1.

@lewing @SamMonoRT could you assign this?

@danmoseley
Copy link
Member

danmoseley commented Jul 21, 2022

It'd be great if someone could add a skip for x86 Android as you suggest.

@vargaz
Copy link
Contributor

vargaz commented Jul 21, 2022

The wasm failures look different from the android ones, i.e. the test runner seems to time out.

@lewing
Copy link
Member

lewing commented Jul 21, 2022

most if not all of the wasm failures are in the threaded runtimes so those failures are not something to worry about at the moment

@danmoseley
Copy link
Member

Yeah, the WASM ones are clearly totally different - they are hangs. @lewing are you saying we should disable these tests for wasm too? that sounds good to me.

@lewing
Copy link
Member

lewing commented Jul 21, 2022

@danmoseley we do want to test json on wasm but the threaded runtime lanes are just being brought up so we expect breakage there.

@simonrozsival can you work with @radical to disable the tests for threaded lanes?

@radical
Copy link
Member

radical commented Jul 22, 2022

@simonrozsival can you extract the wasm part into a separate issue? A tracking issue for all the threaded lane failures will be fine too.

@karelz
Copy link
Member

karelz commented Jul 22, 2022

@lewing can someone on your team disable the tests ASAP please? In avergae it is 36 failures per day across PRs and Official runs -- lots of noise in Test monitoring.

@lewing
Copy link
Member

lewing commented Jul 22, 2022

@karelz #72664

@karelz
Copy link
Member

karelz commented Jul 22, 2022

Great, thanks @lewing!

@simonrozsival
Copy link
Member

@radical I'll look into it on Monday

@tlakollo
Copy link
Contributor

The latest runtime-extra-platforms rolling build doesn't have any System.Text.Json failures, so I will remove the blocking-clean-ci tag

@tlakollo tlakollo removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Jul 23, 2022
@simonrozsival simonrozsival added the disabled-test The test is disabled in source code against the issue label Jul 23, 2022
@lewing lewing removed arch-wasm WebAssembly architecture untriaged New issue has not been triaged by the area owner labels Jul 25, 2022
@lewing lewing added this to the 8.0.0 milestone Jul 25, 2022
@lewing
Copy link
Member

lewing commented Jul 25, 2022

The failing tests have been disabled and there are tracking issues for the actual failures, closing this

@lewing lewing closed this as completed Jul 25, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Aug 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Codegen-meta-mono disabled-test The test is disabled in source code against the issue os-android runtime-mono specific to the Mono runtime test-failure
Projects
None yet
Development

Successfully merging a pull request may close this issue.