Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure: baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll #103547

Closed
v-wenyuxu opened this issue Jun 17, 2024 · 7 comments · Fixed by #106043
Closed

Test failure: baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll #103547

v-wenyuxu opened this issue Jun 17, 2024 · 7 comments · Fixed by #106043
Assignees
@cshung
Labels
arch-arm64 area-VM-coreclr blocking-clean-ci-optional Blocking optional rolling runs GCStress in-pr There is an active PR which will close this issue when it is merged os-linux Linux OS (any supported distro)
Milestone
9.0.0

Comments

@v-wenyuxu
Copy link

Failed in: runtime-coreclr gcstress-extra 20240615.1

Failed tests:

coreclr linux arm64 Checked gcstress0xc_disabler2r_heapverify1 @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux arm64 Checked gcstress0xc_disabler2r_jitstress2 @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux arm64 Checked gcstress0xc_disabler2r @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows arm64 Checked gcstress0xc_disabler2r_jitstress2 @ Windows.11.Arm64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows arm64 Checked gcstress0xc_disabler2r @ Windows.11.Arm64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux x64 Checked gcstress0xc_disabler2r_heapverify1 @ Ubuntu.2204.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows arm64 Checked gcstress0xc_disabler2r_heapverify1 @ Windows.11.Arm64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows x64 Checked gcstress0xc_disabler2r_heapverify1 @ Windows.10.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux x64 Checked gcstress0xc_disabler2r_jitstress2 @ Ubuntu.2204.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows x64 Checked gcstress0xc_disabler2r_jitstress2 @ Windows.10.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux x64 Checked gcstress0xc_disabler2r @ Ubuntu.2204.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows x64 Checked gcstress0xc_disabler2r @ Windows.10.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll

Error message:

 Failed: Verify_ServerGC_Env_Enable: exit code = 134

Stack trace:

   at TestConfigTester.RunTests()
   at Program.<<Main>$>g__TestExecutor4|0_5(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)
@v-wenyuxu v-wenyuxu added arch-arm64 os-linux Linux OS (any supported distro) GCStress blocking-clean-ci-optional Blocking optional rolling runs labels Jun 17, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 17, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Jun 17, 2024
@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

@vcsjones vcsjones removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 18, 2024
@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr gcstress-extra 20240616.3

Failed tests:

coreclr windows arm64 Checked gcstress0xc_disabler2r_heapverify1 @ Windows.11.Arm64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux arm64 Checked gcstress0xc_disabler2r_heapverify1 @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows arm64 Checked gcstress0xc_disabler2r @ Windows.11.Arm64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows arm64 Checked gcstress0xc_disabler2r_jitstress2 @ Windows.11.Arm64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux arm64 Checked gcstress0xc_disabler2r @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux arm64 Checked gcstress0xc_disabler2r_jitstress2 @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows x64 Checked gcstress0xc_disabler2r_heapverify1 @ Windows.10.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux x64 Checked gcstress0xc_disabler2r_jitstress2 @ Ubuntu.2204.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux x64 Checked gcstress0xc_disabler2r_heapverify1 @ Ubuntu.2204.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows x64 Checked gcstress0xc_disabler2r_jitstress2 @ Windows.10.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr linux x64 Checked gcstress0xc_disabler2r @ Ubuntu.2204.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
coreclr windows x64 Checked gcstress0xc_disabler2r @ Windows.10.Amd64.Open
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll

Error message:

 Failed: Verify_ServerGC_Env_Enable: exit code = -1073740286

Stack trace:

   at TestConfigTester.RunTests()
   at Program.<<Main>$>g__TestExecutor4|0_5(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-06-17T07:33:21.772Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_disabler2r windows.11.arm64.open arm64 baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll Failed {"System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","BuildId":"709719","BuildNumber":"20240616.1","System.JobName":"__default","System.PhaseAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","AzurePipelinesTestRunId":"17727754","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_disabler2r","DefinitionId":"141","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64"} Schedule
2024-06-17T07:33:21.772Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_disabler2r windows.11.arm64.open arm64 baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll Failed {"System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","BuildId":"709719","BuildNumber":"20240616.1","System.JobName":"__default","System.PhaseAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","AzurePipelinesTestRunId":"17727754","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_disabler2r","DefinitionId":"141","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64"} Schedule
2024-06-17T07:33:21.772Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_disabler2r windows.11.arm64.open arm64 baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll Failed {"System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","BuildId":"709719","BuildNumber":"20240616.1","System.JobName":"__default","System.PhaseAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","AzurePipelinesTestRunId":"17727754","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_disabler2r","DefinitionId":"141","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64"} Schedule

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Jun 26, 2024
@mangod9 mangod9 added this to the 9.0.0 milestone Jun 26, 2024
@mangod9
Copy link
Member

mangod9 commented Jun 26, 2024

adding @VSadov to check if this can be reproed.

@VSadov
Copy link
Member

VSadov commented Jun 27, 2024

The testcase launches a separate small program while enabling Server GC and that program hits an assert.

Looks like a GC bug. Possibly just an assert that is too strict. @dotnet/gc

22:35:51.533 Running test: baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
Running: "C:\h\w\9FEF08EE\w\B0DA09B1\e\baseservices\baseservices\TestConfig.dll" Verify_ServerGC_Env_Enable
    set DOTNET_gcServer=1

Assert failure(PID 5332 [0x000014d4], Thread: 8404 [0x20d4]): total_soh_stable_size > 0

CORECLR! SVR::gc_heap::dynamic_heap_count_data_t::compute_gen0_budget_per_heap + 0x40 (0x00007ff9`98052cf0)
CORECLR! SVR::gc_heap::update_end_gc_time_per_heap + 0x294 (0x00007ff9`98082b0c)
CORECLR! SVR::gc_heap::gc1 + 0x130C (0x00007ff9`9805e7dc)
CORECLR! SVR::gc_heap::garbage_collect + 0x898 (0x00007ff9`9805d4a0)
CORECLR! SVR::gc_heap::gc_thread_function + 0x498 (0x00007ff9`9805ed98)
CORECLR! SVR::gc_heap::gc_thread_stub + 0x78 (0x00007ff9`9805f268)
CORECLR! <lambda_e7b69842a3dfd51332f5bcb927a9be70>::<lambda_invoker_cdecl> + 0x7C (0x00007ff9`97ee3dbc)
KERNEL32! BaseThreadInitThunk + 0x30 (0x00007ff9`c7952310)
NTDLL! RtlUserThreadStart + 0x3C (0x00007ff9`cad9578c)
    File: D:\a\_work\1\s\src\coreclr\gc\gcpriv.h:4882
    Image: C:\h\w\9FEF08EE\p\corerun.exe

System.Exception: Failed: Verify_ServerGC_Env_Enable: exit code = -1073740286
   at TestConfigTester.RunTests()
   at Program.<<Main>$>g__TestExecutor4|0_5(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)
22:36:03.800 Failed test: baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll
22:36:03.918 Running test: baseservices/callconvs/TestCallingConventions/TestCallingConventions.cmd

@v-wenyuxu
Copy link
Author

Failed in: runtime-coreclr outerloop 20240630.3

Failed tests:

coreclr linux arm64 Checked @ (Ubuntu.1804.Arm64.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
    - baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll

Error message:

 Failed: Verify_ServerGC_Env_Enable: exit code = 134

Stack trace:

   at TestConfigTester.RunTests()
   at Program.<<Main>$>g__TestExecutor4|0_5(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-06-17T07:33:21.772Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_disabler2r windows.11.arm64.open arm64 baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll Failed {"System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","BuildId":"709719","BuildNumber":"20240616.1","System.JobName":"__default","System.PhaseAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","AzurePipelinesTestRunId":"17727754","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_disabler2r","DefinitionId":"141","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64"} Schedule
2024-06-17T07:33:21.772Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_disabler2r windows.11.arm64.open arm64 baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll Failed {"System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","BuildId":"709719","BuildNumber":"20240616.1","System.JobName":"__default","System.PhaseAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","AzurePipelinesTestRunId":"17727754","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_disabler2r","DefinitionId":"141","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64"} Schedule
2024-06-17T07:33:21.772Z runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_disabler2r windows.11.arm64.open arm64 baseservices/RuntimeConfiguration/TestConfigTester/TestConfigTester.dll Failed {"System.StageAttempt":"1","DefinitionName":"runtime-coreclr crossgen2-composite gcstress","BuildId":"709719","BuildNumber":"20240616.1","System.JobName":"__default","System.PhaseAttempt":"1","System.JobId":"c55e12b8-4209-535c-d94d-51b07fc6ab63","AzurePipelinesTestRunId":"17727754","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"Windows.11.Arm64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1_Composite_windows_arm64_checked","configuration":"Checked-gcstress0xc_disabler2r","DefinitionId":"141","Reason":"Schedule","System.JobAttempt":"1","architecture":"arm64"} Schedule

@cshung
Copy link
Member

cshung commented Aug 5, 2024

The bug is understood.

With gcstress=c, the very first GC is triggered before anything is allocated, therefore get_total_soh_stable_size() returned 0 and upset the logic at dynamic_heap_count_data_t::get_total_soh_stable_size().

We can't just relax the assert because this will lead to division by 0.

@Maoni0
Copy link
Member

Maoni0 commented Aug 6, 2024

we can just give it a temp value like min gen1 size, the result from compute_total_gen0_budget will be adjusted to min gen0 budget anyway, ie, in get_total_soh_stable_size, add the if (!total_stable_size) block.

size_t gc_heap::get_total_soh_stable_size()
{
    if (current_total_soh_stable_size)
    {
        return current_total_soh_stable_size;
    }
    else
    {
        size_t total_stable_size = 0;
        for (int i = 0; i < gc_heap::n_heaps; i++)
        {
            gc_heap* hp = g_heaps[i];
            total_stable_size += hp->generation_size (max_generation - 1) / 2;
        }

        if (!total_stable_size)
        {
            // Setting a temp value before a GC naturally happens (ie, due to allocation).
            total_stable_size = dd_min_size (g_heaps[0]->dynamic_data_of (max_generation - 1));
        }

        return total_stable_size;
    }
}

@cshung cshung mentioned this issue Aug 6, 2024
Merged
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Aug 6, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Sep 7, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@cshung cshung

Labels
arch-arm64 area-VM-coreclr blocking-clean-ci-optional Blocking optional rolling runs GCStress in-pr There is an active PR which will close this issue when it is merged os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
9.0.0
Development

Successfully merging a pull request may close this issue.

Make sure get_total_soh_stable_size never return zero cshung/runtime
7 participants
@vcsjones @cshung @VSadov @jeffschwMSFT @Maoni0 @mangod9 @v-wenyuxu