[CI] Lighthouse updates

[andrewetchen]

Reminders for me to look into the following:

Lighthouse CI

We are currently a deprecated authentication config for our Lighthouse CI job: app_id and app_password are deprecated.

dawn/.github/workflows/ci.yml

Lines 4 to 16 in 3591cdf

	lhci:
	name: Lighthouse
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v2
	- name: Lighthouse
	uses: shopify/lighthouse-ci-action@1.0
	with:
	app_id: ${{ secrets.SHOP_APP_ID }}
	app_password: ${{ secrets.SHOP_APP_PASSWORD }}
	store: ${{ secrets.SHOP_STORE }}
	password: ${{ secrets.SHOP_PASSWORD }}
	lhci_github_app_token: ${{ secrets.LHCI_GITHUB_TOKEN }}

https://github.com/Shopify/lighthouse-ci-action/blob/13cf9a620e4d97fb36b2e5a87dae2eefc2670f21/action.yml#L43-L50

...
  app_id:
    description: '(deprecated) Shopify private app ID'
    depreciationMessage: Shopify private apps are deprecated. Use a Custom App `access_token` instead.
    required: false
  app_password:
    description: '(deprecated) Shopify private app password'
    depreciationMessage: Shopify private apps are deprecated. Use a Custom App `access_token` instead.
    required: false
...

We should use the Custom app authentication method.

---

Add leaving v by changing shopify/lighthouse-ci-action@1.0 to shopify/lighthouse-ci-action@v1: this might not be too critical as Github doesn't currently apply any versioning semantics for actions but they might in the future and its good practice.

dawn/.github/workflows/ci.yml

Line 10 in 3591cdf

uses: shopify/lighthouse-ci-action@1.0

This was also recently changed on the lighthouse-ci-action repo.

I tested this by pushing a test commit and let the CI do its thing:

• https://github.com/Shopify/dawn/runs/6070248681?check_suite_focus=true

  

[andrewetchen]

Update April 25 2022

To use the Custom app method for authentication, I added a custom app to our store.

Next steps

Add SHOP_ACCESS_TOKEN as a new secret and update the following section of our ci.yml:

From this:

...
  with:
    app_id: ${{ secrets.SHOP_APP_ID }}
    app_password: ${{ secrets.SHOP_APP_PASSWORD }}
    store: ${{ secrets.SHOP_STORE }}
    password: ${{ secrets.SHOP_PASSWORD }}
    lhci_github_app_token: ${{ secrets.LHCI_GITHUB_TOKEN }}

To this:

...
  with:
    store: ${{ secrets.SHOP_STORE }}
    password: ${{ secrets.SHOP_PASSWORD }}
    access_token: ${{ secrets.SHOP_ACCESS_TOKEN }}
    lhci_github_app_token: ${{ secrets.LHCI_GITHUB_TOKEN }}

[andrewetchen]

Update

I coordinated with Tyler to get the new secrets added to the repo and everything seems to be good to go in terms of using the non-deprecated authentication method.

I'll have to do some more digging but I wanted to quickly point out that we might've been getting skewed Lighthouse scores using the deprecated authentication method (how it is currently):

This was an action run from a recent commit:

• https://github.com/Shopify/dawn/runs/6233179949?check_suite_focus=true
• Click on the Lighthouse run
• Click on the first Lighthouse report link
• Notice the https://dawn-dev-ci.myshopify.com domain at the top?
  • screenshot
• This is the same for all of our previous CI runs.

Here’s an action run using the new Custom app authentication method:

• https://github.com/Shopify/dawn/runs/6233576615?check_suite_focus=true
• Click on the Lighthouse run
• Click on the first Lighthouse report link
• Notice the https://os2-demo.myshopify.com domain? I believe this is what it should be since any changes made to this PR should be checking against our demo store for performance and accessibility.
  • screenshot

[andrewetchen]

Update

I ran 2 more tests, pushing 2 separate commits:

• dawn-dev-ci
  • Commit: 4a4c115
  • Workflow: https://github.com/Shopify/dawn/runs/6275130447?check_suite_focus=true#step:4:57
  • Passed ✅
• os2-demo:
  • Commit: 5e5bca3
  • Workflow: https://github.com/Shopify/dawn/runs/6275223405?check_suite_focus=true#step:4:60
  • Accessibility failed ❌

The os2-demo accessibility report shows why it’s failing:

• I've confirmed that a stores' live or unpublished themes do not factor into Lighthouse CI tests.
• What does play a part is how each store (dawn-dev-ci and os2-demo) is setup differently (Products, Navigation, Collections, etc.).
• When the Lighthouse CI starts, it runs on the code of the PR and "pushes" it to a non-published/unlisted development theme (not shown in Admin > Themes) and is tested against this.

[andrewetchen]

Update - May 16 2022

Adding some notes. I'm hoping this will also help establish a Lighthouse CI standard for our other themes, not just Dawn.

Our current Lighthouse CI runs proposed changes against the dawn-dev-ci store: if you create a PR with changes to the header.liquid, the Lighthouse CI will take a copy of this version of Dawn and test against the store setup/config of dawn-dev-ci.

Navigation / Header

Running the Lighthouse CI on os2-demo will result in failed accessibility audit (noted in above comment).

• Upon further troubleshooting, one of the main differences between dawn-dev-ci and os2-demo is the navigation menus
  • Screenshot of dawn-dev-ci navigation
  • The os2-demo navigation has a lot more menu items but the thing to highlight here is that when anchor links are used, the ARIA IDs are no longer unique. This is what causes the Lighthouse CI audit to fail accessibility: screenshot
  • Tyler mentioned that we'd need to update the header.liquid by assigning a parent index:
    • 5e7c1b7
    • These changes resulted in a passed Lighthouse accessibility audit.
    • Another way to think of about this is that all current and previous Lighthouse CI runs (especially PRs with changes to the header, navigation, etc.) weren't being checked effectively since dawn-dev-ci only has 2 menu items: we weren't checking for nested menu items and anchors etc.

Lighthouse CI / ci.yml improvements

When we do update the Shopify/dawn repo to use the os2-demo store instead of dawn-dev-ci, we should take advantage of the following options (note: the Lighthouse CI runs 3 times on the homepage, product page, and collection page):

• Update the ci.yml to use the Puppy product by including the optional product_handle option (the Puppy product is a good test 🐶): https://github.com/Shopify/lighthouse-ci-action#configuration
• Update the ci.yml to use a collection that has a collection image and collection description by including the optional collection_handle option: https://github.com/Shopify/lighthouse-ci-action#configuration

We can also look into Apps to see if any of them affect the Lighthouse CI runs (installed and not enabled: think Shopify Product Reviews).