[Azure Pipelines] Upgrade macOS VMs to Mojave

[foolip]

No description provided.

[foolip]

I opened #17217 to be able to see if any failures are pre-existing. I'll also push that commit to this branch to this PR to see if everything passes. Additionally, I'll trigger a full run of the same type as runs every 6 hours.

[foolip]

The manual build is https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18422.

[foolip]

https://wpt.fyi/results/?diff&filter=ADC&run_id=266520006&run_id=242570013 compares results from 10.13 and 10.14. Looks OK except for a huge regression in css-grid results. The exact same commit wasn't tested, so I'll try again after the next Safari TP run on master.

[gsnedders]

FWIW, those grid results match what I get locally in STP 84.

[foolip]

New build on top of commit 38bf3be in progress:
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18475

Sent #17225 to be able to do the same for stable.

[foolip]

I've rebased and triggered new full runs of both Safaris:
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18765
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18766

[foolip]

Diff of results compared to the parent commit for those runs:
https://wpt.fyi/results/?diff&filter=ADC&run_id=255090001&run_id=242660007
https://wpt.fyi/results/?diff&filter=ADC&run_id=252540003&run_id=260590004

[foolip]

Aha! For both Safaris /infrastructure/assumptions/ahem.html is failing because Ahem appears to not be loaded. This is also the cause of some other failures such as /css/CSS2/text/letter-spacing-004.xht.

Rather than debugging system font installation on Mojave I'll wait for web-platform-tests/rfcs#22 to be resolved, we're pinned to Safari TP 82 anyway so there's no urgency to upgrade.

[gsnedders]

Pretty sure 10.14 is the point at which they stopped loading non-system fonts (by default?).

com.apple.Safari.ContentPageGroupIdentifier.WebKit2ShouldAllowUserInstalledFonts is the pref to configure it, but I think changing that requires disabling the sandboxing?

[foolip]

Oh, I didn't know that changes was coming / has come. Good thing that we went ahead with the Ahem change then! I won't try the setting to get the old behavior, @LukeZielinski's work on Ahem will probably resolve this in a matter of weeks anyway.

[foolip]

Since #17173 was merged yesterday I've rebased and triggered another full run of the Safaris to see if the differences are fewer now:
https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18914

[foolip]

New diffs:
https://wpt.fyi/results/?diff&filter=ADC&run_id=255120001&run_id=240740006
https://wpt.fyi/results/?diff&filter=ADC&run_id=231010007&run_id=252590016

First looks about the same, but results for Safari TP seem incomplete :(

[gsnedders]

A bunch of the regressions in CSS are just more things needed Ahem

[foolip]

It looks like "all tests (Safari Technology Preview) 1" in https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18914 silently failed to run all of the tests it should have run. Comparing it to https://dev.azure.com/web-platform-tests/wpt/_build/results?buildId=18874 where the last test was in /xhr/, that job stopped at /payment-request/payment-is-showing.https.html:

  ▶ ERROR [expected OK] /payment-request/payment-is-showing.https.html
  └   → Traceback (most recent call last):
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 282, in _run
    self.result = True, self.func(self.protocol, self.url, self.timeout)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 373, in do_testharness
    done, rv = handler(result)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/wptrunner/wptrunner/executors/base.py", line 604, in __call__
    return callback(url, payload)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/wptrunner/wptrunner/executors/base.py", line 622, in process_action
    self._send_message("complete", "error")
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/wptrunner/wptrunner/executors/base.py", line 631, in _send_message
    self.protocol.testdriver.send_message(message_type, status, message=message)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 189, in send_message
    self.webdriver.execute_script("window.postMessage(%s, '*')" % json.dumps(obj))
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/webdriver/webdriver/client.py", line 20, in inner
    return func(self, *args, **kwargs)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/webdriver/webdriver/client.py", line 631, in execute_script
    return self.send_session_command("POST", "execute/sync", body)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/webdriver/webdriver/client.py", line 508, in send_session_command
    return self.send_command(method, url, body)
  File "/Users/vsts/agent/2.152.1/work/1/s/tools/webdriver/webdriver/client.py", line 472, in send_command
    raise err
InvalidSessionIdException: invalid session id (404): 



##[section]Finishing: Run tests

@gsnedders is "invalid session id" something from SafariDriver? Is this another case of #13073?

[gsnedders]

@foolip that should be recoverable; the irrecoverable error should be when we repeatedly fail to restart.

[gsnedders]

(Needs a rebase to pass CI, due to #17286. Sorry!)

[foolip]

@gsnedders is there a bug for making this case restart properly? We could potentially be getting bad runs already because of this.

[gsnedders]

@foolip we should already handle this; I think we really need logs with debug info in them to figure out what's going on (i.e., --log-mach - --log-mach-level debug).

[foolip]

I've sent a minimized test in #18489. However, I think it'll pass in Safari on 10.13 where we still have the system font installed.

[foolip]

I think https://bugs.webkit.org/show_bug.cgi?id=174030 is the right WebKit bug for this, commented there.

[foolip]

https://bugs.webkit.org/show_bug.cgi?id=174030 has been fixed, but upgrading is still not possible. It's likely going to be necessary to upgrade both the OS and STP at the same time as I've attempted in #18925 but still isn't working. This PR won't be merged at any rate, so I'm closing it.

Will try to summarize the state of things in #17186.