[browser] Increase timeout and clean up after failed attempts of launching browser when testing

[ilonatommy]

This is a playground for fixing #107771.
The issue is hit only on Windows, where we are running WBT sequentially.

Changes:

• Upgrade Playwright to "1.47.0" (newest available)
• After upgrading Playwright, it is creating node dir in linux-x64 dir, not in linux. Node requires elevated execution rights, so chmod +x path was changed accordingly.
• Killing all chrome processes before we call the problematic method that launches the browser.
• Making sure we clean up after each run by closing Browser and disposing of Playwright.

[ilonatommy]

It does not help:

        Attempt 3 failed with TimeoutException: System.TimeoutException: Timeout 15000ms exceeded.
=========================== logs ===========================
<launching> C:\helix\work\correlation\chrome-win\chrome.exe --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --headless --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --explicitly-allowed-ports=49238 --ignore-certificate-errors --user-data-dir=C:\Users\ContainerAdministrator\AppData\Local\Temp\playwright_chromiumdev_profile-j7hftC --remote-debugging-pipe --no-startup-window
<launched> pid=9100
============================================================
           at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](String guid, String method, Object args) in /_/src/Playwright/Transport/Connection.cs:line 164
           at Microsoft.Playwright.Transport.Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal) in /_/src/Playwright/Transport/Connection.cs:line 475
           at Microsoft.Playwright.Core.BrowserType.LaunchAsync(BrowserTypeLaunchOptions options) in /_/src/Playwright/Core/BrowserType.cs:line 61
           at Wasm.Build.Tests.BrowserRunner.SpawnBrowserAsync(String browserUrl, Boolean headless, Int32 timeout, Int32 maxRetries) in /_/src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs:line 120

log

[lewing]

Can we check if there are other running chrome instances or something to help diagnose the issue?

[ilonatommy]

If we keep the last commit about killing all other chromes, running WBT locally can be a nuisance. If it works, we should check for running on CI and only then call TerminateExistingChromeInstances. I did not register local timeouts similar to these on CI, so it should be fine.

[ilonatommy]

It's not about chrome instances, Found 0 old Chrome processes. in log when it still timeouts.

[ilonatommy]

The chrome bump will be ready soon: #107504, maybe that might help?

[ilonatommy]

Failures are on Windows with Workloads-NoFingerprint-ST-Wasm.Build.Tests:

 Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations(forConsole: False, runOutsideProjectDirectory: True, extraProperties: "") [FAIL]
      System.Exception : Timed out after 120s waiting for 'WASM EXIT' message
      Stack Trace:
        /_/src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs(242,0): at Wasm.Build.Tests.BrowserRunner.WaitForExitMessageAsync(TimeSpan timeout)
        /_/src/mono/wasm/Wasm.Build.Tests/Templates/WasmTemplateTests.cs(324,0): at Wasm.Build.Tests.WasmTemplateTests.BrowserRunTwiceWithAndThenWithoutBuildAsync(String config, String extraProperties, Boolean runOutsideProjectDirectory)
        /_/src/mono/wasm/Wasm.Build.Tests/Templates/WasmTemplateTests.cs(325,0): at Wasm.Build.Tests.WasmTemplateTests.BrowserRunTwiceWithAndThenWithoutBuildAsync(String config, String extraProperties, Boolean runOutsideProjectDirectory)
        /_/src/mono/wasm/Wasm.Build.Tests/Templates/WasmTemplateTests.cs(301,0): at Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations(Boolean forConsole, Boolean runOutsideProjectDirectory, String extraProperties)

[ilonatommy]

Log:

System.ComponentModel.Win32Exception : An error occurred trying to start process '/root/helix/work/workitem/e/.playwright/node/linux-x64/node' with working directory '/root/helix/work/workitem/e'. Permission denied
      Stack Trace:
           at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
           at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
           at System.Diagnostics.Process.Start()
        /_/src/Playwright/Transport/StdIOTransport.cs(168,0): at Microsoft.Playwright.Transport.StdIOTransport.StartProcessWithUTF8IOEncoding(Process process)
        /_/src/Playwright/Transport/StdIOTransport.cs(47,0): at Microsoft.Playwright.Transport.StdIOTransport..ctor()
        /_/src/Playwright/Playwright.cs(43,0): at Microsoft.Playwright.Playwright.CreateAsync()
        /_/src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs(128,0): at Wasm.Build.Tests.BrowserRunner.SpawnBrowserAsync(String browserUrl, Boolean headless, Int32 timeout, Int32 maxRetries, String language)
        /_/src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs(201,0): at Wasm.Build.Tests.BrowserRunner.RunAsync(ToolCommand cmd, String args, Boolean headless, String language, Action`2 onConsoleMessage, Action`1 onServerMessage, Action`1 onError, Func`2 modifyBrowserUrl)
        /_/src/mono/wasm/Wasm.Build.Tests/Blazor/BlazorWasmTestBase.cs(202,0): at Wasm.Build.Tests.BlazorWasmTestBase.BlazorRunTest(String runArgs, String workingDirectory, BlazorRunOptions runOptions)
        /_/src/mono/wasm/Wasm.Build.Tests/Blazor/BlazorWasmTestBase.cs(229,0): at Wasm.Build.Tests.BlazorWasmTestBase.BlazorRunTest(String runArgs, String workingDirectory, BlazorRunOptions runOptions)
        /_/src/mono/wasm/Wasm.Build.Tests/Blazor/WorkloadRequiredTests.cs(119,0): at Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization(String config, Boolean invariant, Boolean publish)
        --- End of stack trace from previous location ---

It's a problem only on linux, so I will add a task to yml that adds exec permission to node.

@akoeplinger, do you know why it did not help? It's still failing with same message

[ilonatommy]

The issue is still existing after the Playwright bump, see: https://helix.dot.net/api/2019-06-17/jobs/8d53b01d-113d-4e65-a893-6a5d825a52b6/workitems/Workloads-NoWebcil-ST-Wasm.Build.Tests.TestAppScenarios.LibraryInitializerTests/console.
I created an issue in Playwright repo: microsoft/playwright#32897, maybe we can get some suggestions about what could be going on.

[ilonatommy]

Using default chromium from Playwright is broken on Linux:

Microsoft.Playwright.PlaywrightException : Executable doesn't exist at /home/helixbot/.cache/ms-playwright/chromium-1134/chrome-linux/chrome
      ╔════════════════════════════════════════════════════════════╗
      ║ Looks like Playwright was just installed or updated.       ║
      ║ Please run the following command to download new browsers: ║
      ║                                                            ║
      ║     pwsh bin/Debug/netX/playwright.ps1 install             ║
      ║                                                            ║
      ║ <3 Playwright Team                                         ║
      ╚════════════════════════════════════════════════════════════╝
      Stack Trace:
        /_/src/Playwright/Transport/Connection.cs(206,0): at Microsoft.Playwright.Transport.Connection.InnerSendMessageToServerAsync[T](ChannelOwner object, String method, Dictionary`2 dictionary, Boolean keepNulls)
        /_/src/Playwright/Transport/Connection.cs(532,0): at Microsoft.Playwright.Transport.Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal)
        /_/src/Playwright/Core/BrowserType.cs(56,0): at Microsoft.Playwright.Core.BrowserType.LaunchAsync(BrowserTypeLaunchOptions options)
        /_/src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs(129,0): at Wasm.Build.Tests.BrowserRunner.SpawnBrowserAsync(String browserUrl, Boolean headless, Nullable`1 timeout, Int32 maxRetries, String language)
        /_/src/mono/wasm/Wasm.Build.Tests/BrowserRunner.cs(201,0): at Wasm.Build.Tests.BrowserRunner.RunAsync(ToolCommand cmd, String args, Boolean headless, String language, Action`2 onConsoleMessage, Action`1 onServerMessage, Action`1 onError, Func`2 modifyBrowserUrl)
        /_/src/mono/wasm/Wasm.Build.Tests/Blazor/BlazorWasmTestBase.cs(202,0): at Wasm.Build.Tests.BlazorWasmTestBase.BlazorRunTest(String runArgs, String workingDirectory, BlazorRunOptions runOptions)
        /_/src/mono/wasm/Wasm.Build.Tests/Blazor/BlazorWasmTestBase.cs(229,0): at Wasm.Build.Tests.BlazorWasmTestBase.BlazorRunTest(String runArgs, String workingDirectory, BlazorRunOptions runOptions)
        /_/src/mono/wasm/Wasm.Build.Tests/Blazor/WorkloadRequiredTests.cs(119,0): at Wasm.Build.Tests.Blazor.WorkloadRequiredTests.WorkloadNotRequiredForInvariantGlobalization(String config, Boolean invariant, Boolean publish)
        --- End of stack trace from previous location ---

https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-107865-merge-50c72d5eddc049108b/NoWorkload-NoWebcil-ST-Wasm.Build.Tests/1/console.47c37d30.log?helixlogtype=result

[ilonatommy]

1st and 2nd run with default chrome + higher timeout: success.
3rd: timeout on Linux, Windows passes.
4th: passes.