test: [M3-8478] - Fix Cypress StackScript Linode deploy test flake

[jdamore-linode]

Description 📝

This resolves some causes of test flakiness in the StackScript Linode deploy test in create-stackscript.spec.ts.

There seem to be two main causes of flake for this test in CI, and this PR attempts to fix both:

• Attempting to find a region in the region drop-down without scrolling to it first
• Issues selecting a region quickly upon page load

I also changed the way that StackScripts get entered into the create form. Using cy.type() to input the script is slow and adds several seconds to our test runtime, so I've circumvented it by setting the field's value directly.

I ran the test on repeat 30 times (with the other test skipped) and it passed each time.

(cc @mjac0bs -- I'm curious if you continue seeing failures with these changes? No worries if you don't have time to take a look!)

Changes 🔄

• Scroll to desired region before attempting to select it from drop-down
• Wait for request to account availability endpoint to resolve before interacting with region select
  • ℹ️ It's not clear to me if interacting with the region select before this request resolves is what causes the region selection issue, or if waiting for this request to resolve only fixes it incidentally

How to test 🧪

Because this test doesn't fail every time, CI may not be the most reliable way to gauge the effectiveness of this change.

You can run the test locally with this command (feel free to skip the other test since it's a slow one and isn't related to these changes):

yarn cy:run -s "cypress/e2e/core/stackscripts/create-stackscripts.spec.ts"

As an Author I have considered 🤔

Check all that apply

• 👀 Doing a self review
• ❔ Our contribution guidelines
• 🤏 Splitting feature into small PRs
• ➕ Adding a changeset
• 🧪 Providing/Improving test coverage
• 🔐 Removing all sensitive information from the code and PR description
• 🚩 Using a feature flag to protect the release
• 👣 Providing comprehensive reproduction steps
• 📑 Providing or updating our documentation
• 🕛 Scheduling a pair reviewing session
• 📱 Providing mobile support
• ♿ Providing accessibility support

[jdamore-linode]

We'll try to merge this after @AzureLatte's PR #10788 just in case there are merge conflicts

[jaalah-akamai]

I'm still seeing the failure when I checkout this branch :(

yarn cy:run -s "cypress/e2e/core/stackscripts/create-stackscripts.spec.ts"

[jdamore-linode]

@jaalah-akamai I'm assuming it fails consistently, too? There's another issue related to this test (but which also impacts others) that's captured by M3-8471 but I believe is unrelated to the flake I'm trying to address here.

You could help me narrow it down a lot:

• Are you running Cloud via yarn up or yarn dev?
• If so, does the test still fail if you run Cloud via yarn build && yarn start:manager:ci?

[github-actions]

Coverage Report: ❌
Base Coverage: 82.63%
Current Coverage: 82.61%

[jaalah-akamai]

• Are you running Cloud via yarn up or yarn dev?
• If so, does the test still fail if you run Cloud via yarn build && yarn start:manager:ci?

1. I ran it with yarn install && yarn build && yarn dev and it fails consistently
2. Running it with yarn build && yarn start:manager:ci failed too (see report)

Report

yarn run v1.22.21
$ yarn workspace linode-manager cy:run -s cypress/e2e/core/stackscripts/create-stackscripts.spec.ts
$ cypress run -b chrome -s cypress/e2e/core/stackscripts/create-stackscripts.spec.ts

DevTools listening on ws://127.0.0.1:62511/devtools/browser/5bc5edf4-b1d8-4777-9f9c-7c8f62c7ec65
2024-08-23 13:25:34.958 Cypress[75754:128427111] WARNING: Secure coding is not enabled for restorable state! Enable secure coding by implementing NSApplicationDelegate.applicationSupportsSecureRestorableState: and returning YES.
(node:75782) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
(node:75782) ExperimentalWarning: The Node.js specifier resolution flag is experimental. It could change or be removed at any time.
┌──────────────────────┬─────────────────────────────┐
│       (index)        │           Values            │
├──────────────────────┼─────────────────────────────┤
│ API Request Root URL │ 'https://api.linode.com/v4' │
└──────────────────────┴─────────────────────────────┘

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.11.0                                                                        │
  │ Browser:        Chrome 127 (headless)                                                          │
  │ Node Version:   v18.14.1 (.../node/v18.14.1/bin/node)                 │
  │ Specs:          1 found (create-stackscripts.spec.ts)                                          │
  │ Searched:       cypress/e2e/core/stackscripts/create-stackscripts.spec.ts                      │
  │ Experiments:    experimentalRunAllSpecs=true,experimentalMemoryManagement=true                 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  create-stackscripts.spec.ts                                                     (1 of 1)
Browser information:
┌─────────────────┬──────────────────────────┐
│     (index)     │          Values          │
├─────────────────┼──────────────────────────┤
│     Browser     │ 'Chrome v127.0.6533.120' │
│     Family      │        'chromium'        │
│    Headless     │          'Yes'           │
│ Chrome Headless │          'Old'           │
└─────────────────┴──────────────────────────┘


  Create stackscripts
    1) creates a StackScript and deploys a Linode with it
    2) creates a StackScript with Any/All target image


  0 passing (1m)
  2 failing

  1) Create stackscripts
       creates a StackScript and deploys a Linode with it:
     AssertionError: Timed out retrying after 80000ms: Unable to find an element with the text: US, Chicago, IL (us-ord). This could be because the text is broken up by multiple elements. In this case, you can provide a function for your text matcher to make your matcher more flexible.

Ignored nodes: comments, script, style
<div
  class="base-Popper-root MuiAutocomplete-popper MuiAutocomplete-popperDisablePortal css-1khgx4r"
  data-popper-placement="bottom"
  data-qa-autocomplete-popper="true"
  data-testid="autocomplete-popper"
  role="presentation"
  style="width: 416px; position: absolute; inset: 0px auto auto 0px; margin: 0px; transform: translate(24px, 1195px);"
>
  <div
    class="MuiPaper-root MuiPaper-elevation MuiPaper-rounded MuiPaper-elevation1 MuiAutocomplete-paper css-1ftvvpo"
  >
    <ul
      aria-labelledby=":r15:-label"
      class="MuiAutocomplete-listbox css-zttxxl"
      id=":r15:-listbox"
      role="listbox"
    >
      <li>
        <div
          class="MuiListSubheader-root MuiListSubheader-gutters MuiListSubheader-sticky MuiAutocomplete-groupLabel css-oodkzr"
        >
          North America
        </div>
        <ul
          class="MuiAutocomplete-groupUl css-15s1ek9"
        >
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem Mui-focused"
            data-option-index="0"
            data-qa-option="us-southeast"
            data-qa-tooltip=""
            data-testid="us-southeast"
            id=":r15:-option-0"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Atlanta, GA
            </div>
            (us-southeast)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="1"
            data-qa-option="us-ord"
            data-qa-tooltip=""
            data-testid="us-ord"
            id=":r15:-option-1"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Chicago, IL
            </div>
            (us-ord)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="2"
            data-qa-option="us-central"
            data-qa-tooltip=""
            data-testid="us-central"
            id=":r15:-option-2"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Dallas, TX
            </div>
            (us-central)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="3"
            data-qa-option="us-west"
            data-qa-tooltip=""
            data-testid="us-west"
            id=":r15:-option-3"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Fremont, CA
            </div>
            (us-west)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="4"
            data-qa-option="us-hou"
            data-qa-tooltip=""
            data-testid="us-hou"
            id=":r15:-option-4"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Houston, TX
            </div>
            (us-hou)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="5"
            data-qa-option="us-lax"
            data-qa-tooltip=""
            data-testid="us-lax"
            id=":r15:-option-5"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Los Angeles, CA
            </div>
            (us-lax)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="6"
            data-qa-option="us-mia"
            data-qa-tooltip=""
            data-testid="us-mia"
            id=":r15:-option-6"
            role="option"
            tabindex="-1"
          >
            <div
              class="MuiBox-root css-b5zaes"
            >
              <div
                class="css-6ymqa8-RegionSelectFlagContainer"
              >
                <div
                  class="fi fi-us fi-xx css-194iuv8-StyledFlag"
                />
              </div>
              US, Miami, FL
            </div>
            (us-mia)
          </li>
          <li
            aria-label=""
            aria-selected="false"
            class="MuiListItem-root MuiListItem-gutters MuiListItem-padding MuiAutocomplete-option css-1iwvvk7-RegionSelectListItem"
            data-option-index="7"
            data-qa-option="us-east"
            data-qa-tooltip=""
     ...
      at Object.findByTitle (http://localhost:3000/__cypress/tests?p=cypress/e2e/core/stackscripts/create-stackscripts.spec.ts:46731:78)
      at Object.findItemByRegionLabel (http://localhost:3000/__cypress/tests?p=cypress/e2e/core/stackscripts/create-stackscripts.spec.ts:46768:33)
      at fillOutLinodeForm (http://localhost:3000/__cypress/tests?p=cypress/e2e/core/stackscripts/create-stackscripts.spec.ts:52988:21)
      at Context.eval (http://localhost:3000/__cypress/tests?p=cypress/e2e/core/stackscripts/create-stackscripts.spec.ts:53074:7)

  2) Create stackscripts
       creates a StackScript with Any/All target image:
     Error: Linode APIv4 request failed with status code 400
- This Linode's region does not support VLANs at this time. (field 'interfaces[0].purpose')
Request: POST https://api.linode.com/v4/linode/instances
      at enhanceError (http://localhost:3000/__cypress/tests?p=cypress/support/e2e.ts:13268:14)
      at wrapPromise (http://localhost:3000/__cypress/tests?p=cypress/support/e2e.ts:13332:17)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      2                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  2                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 28 seconds                                                             │
  │ Spec Ran:     create-stackscripts.spec.ts                                                      │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

[jaalah-akamai]

I bet this has to do with the region changes again?

[jdamore-linode]

I bet this has to do with the region changes again?

Thanks @jaalah-akamai, I'm intrigued.

The DOM printout makes it pretty clear why the test is failing: it expects the label and ID text to be inside the same element, e.g. <div>US, Chicago, IL (us-ord)</div> and in your case it's more like <div>US, Chicago, IL</div> (us-ord)

But I'm baffled where this is coming from, and haven't been able to reproduce this at all on my end, even messing around with the Gecko and Linode Create v2 feature flags. Any idea where this is coming from?

[jaalah-akamai]

@hana-linode Can we double check this? This appears to be causing the issue. It's outside the <div> in that one case cc @jdamore-linode

[jdamore-linode]

@hana-linode Can we double check this? This appears to be causing the issue. It's outside the <div> in that one case cc @jdamore-linode

Thanks for the find, @jaalah-akamai! I can update the Cypress ui.regionSelect.findItemByRegionId() and ui.regionSelect.findItemByRegionLabel() utils to account for both states so no worries as far as getting tests passing! I'll do that in a separate PR for M3-8471.

One thing that @abailly-akamai noticed here, however, is that this change prevents users from searching for regions by ID in the drop-down. That breaks a couple tests which we can fix on a case-by-case basis, but I'm wondering if that ought to be treated as a bug and fixed in the component instead?

(cc @hana-linode)

[mjac0bs]

@jdamore-linode This branch does seem to fix the issue I was seeing previously for me. 🤔 It passed all 12 times with yarn up and another 12 with yarn build && yarn start:manager:ci.

[coliu-akamai]

This could be just a me problem, but I got this reason for a test failure (used yarn up) 😅
I'd used yarn build && yarn start:manager:ci right before, and didn't have any issues - will try rerunning too

[jdamore-linode]

This could be just a me problem, but I got this reason for a test failure (used yarn up) 😅 I'd used yarn build && yarn start:manager:ci right before, and didn't have any issues - will try rerunning too

(screenshot)

Thanks @coliu-akamai! Re-running it might let it pass -- that failure indicates Cypress tried to create a Linode in a region without availability, and the region gets randomly selected based on what's returned by the /regions endpoint.

Since that test tries to create a Linode of type g6-dedicated-2, I'm guessing it might be some non-generally-available region associated with your test account since I'd expect that plan to be available in most Prod regions (and we haven't seen that particular failure in CI as far as I know).