Monitoring ticket for manual testing of Cabal library 3.6.3 (for GHC 9.2.2) candidate.

[Mikolaj]

[Edit: the context is that we don't have the GHC in CI yet, so we need to test manually our release dedicated to this GHC] Let's note down here what runs from CI have been manually performed (and on what OSes) on the 3.6 branch (CI is not backported to 3.6), which haven't been, what tests failed.

[jneira]

From #8025 (comment)

Just in case i am adding a 9.2.1 job against 3.6 here: jneira#6
Not sure if it worths to add it upstream though

I agree, local tests should be enough

I suppose local tests has been done only in linux so there is some risk of unexpected bugs in windows/macos. I hope the risk is low enough to go forward.

I will try to give a try to the Cabal specific test suites locally in windows but afaics ci is not (and neved did) running them for windows

[jneira]

Ok, the ci job for 9.2.1 and 3.6:

• passed sh validate.sh -j 2 -w ghc-9.2.1 -v --lib-only -s lib-tests 💯
• failed in sh validate.sh -j 2 -w ghc-9.2.1 -v --lib-only -s lib-suite:

UNEXPECTED FAIL: PackageTests/Backpack/Includes3/setup-external-ok.test.hs PackageTests/Backpack/Includes2/setup-per-component.test.hs PackageTests/Backpack/Includes2/setup-external.test.hs

[jneira]

Not sure if it worths to add it upstream though

Opened a pr upstream: #8027

[Mikolaj]

Thank you. How do I interpret the 'passed' and then 'failed' for things that look the same?

[jneira]

Thank you. How do I interpret the 'passed' and then 'failed' for things that look the same?

it is lib-tests vs lib-suite and the steps are defined in validate.hs

• lib-tests runs the Cabal package tests
• lib-suite runs the Cabal specific tests of cabal-testsuite (not passing any cabal-intall executable in --with-cabal)
  • so you can see lot lot of skipped tests (specific cabal-install ones)

[Mikolaj]

Doh, I missed the last words. :)

So these are the same failures we see in 3.7 and we agreed to ignore, right?

[jneira]

yeah, both 3.6 and master fails in the lib-suite step with

UNEXPECTED FAIL: PackageTests/Backpack/Includes3/setup-external-ok.test.hs PackageTests/Backpack/Includes2/setup-per-component.test.hs PackageTests/Backpack/Includes2/setup-external.test.hs

but not sure if that failing test new in 9.2.1 could affect ghc in any way

[Mikolaj]

Yeah, I think we can take that risk.

So what matrix of tests with GHC 9.2 will be run for 3.6.3 compared to tests for 3.7? Only Linux? Why not Windows? Apart of that, are all tests run? How about cabal-install tests that exercise the Cabal library (and almost all of them do, right?)? What's our plan informed by cost/benefit?

[jneira]

So what matrix of tests with GHC 9.2 will be run for 3.6.3 compared to tests for 3.7? Only Linux? Why not Windows?

• Windows test coverage has been misrepresented historically, it only tests 8.6.5 and 8.10.4 in both master and 3.6 (it only has the plus of dogfooding tests in master over other os's)
  • I am trying to improve it here: Merge workflows jneira/cabal#5, will try to do a pr asap against upstream
  • so not having tests for 9.2.1 in master i am not sure if it worths create a new one for 3.6
• master is testing 9.2.1 against macos and 3.7 does not, macos is failing in master with the same error than linux so i think we can discard the backport to 3.6

[Mikolaj]

OK, agreed, Linux-only is sensible cost/benefit, especially relative to 3.7 (by which I mean master branch).

[Mikolaj]

Well, at least it seems it wasn't a problem for anybody in 9.2.1, so let's hope it won't be in 9.2.2 either.

Is there anything else left to test? If not, I guess we can bump version (in cabal-install as well, even though we probably don't intend to release it) to 3.6.3 and wait for @emilypi's decision whether/when to release or a release checklist from @emilypi or a volunteer hero that would attempt the release without a checklist (and then we'd all handle the fallout in the coming months).

[Mikolaj]

BTW, GHC devs don't need to wait for the release. Just bumping the version and, I guess, a tag, should be enough for them.

[Mikolaj]

It seems we bumped version already, so I hope the answer to "Is there anything else left to test?" is negative. Closing. Thank you @jneira.