Mark failing tests as expected failures, or fix them

[robx]

This marks tests that are currently failing on CI as "expected broken", to move back towards "green".

My understanding is that these were broken all along, but only started visibly failing recently with an extended test matrix through #7952.

Also

• raises the limit on the memory test suit to make that pass on 9.2 (memory-usage-tests fail with GHC 9.2 #8029)
• update some old travis-related test skips

---

Please include the following checklist in your PR:

• Patches conform to the coding conventions.
• no changelog, since no user-visible changes
• no documentation update necessary

[jneira]

I dont like see commits and prs with red crosses neither. It has a point though: make us eager to fix failing tests.
Skipping them is somewhat like hiding it and then the fix is delayed until the end of the times.
Is ther a way to make them broken, so at least they will fail if they are fixed? I think i've seen messages similar to "expected broken" in test logs.
That way we can avoid skipped tests which should not be, like https://github.com/haskell/cabal/pull/8022/files#diff-33a6bc12843418321803ea88d60a1bc8f76f352619a287ae4b91e1c299e77b87L8

[jneira]

Also, could be a comment be attached mentioning the issue?

[robx]

I mean, when we get a bug report, we'd never first merge a failing test either while the fix is (maybe or not) incoming sooner or later, right?

Expected failures would be nice, but they're not available at this point. I'm happy to file an issue so that can be done as a follow-up.

And yeah I absolutely mean to relevant tickets, meant to already, will fix if I didn't.

[jneira]

And yeah I absolutely mean to relevant tickets, meant to already, will fix if I didn't.

I mean mentioning it in the test itself, to be shown in test logs, if possible

[andreabedini]

As long as we capture the information and link to the relevant tickets (which @Mikolaj has done), I am ok with this.

[jneira]

Expected failures would be nice, but they're not available at this point. I'm happy to file an issue so that can be done as a follow-up

could we not use

cabal/cabal-testsuite/src/Test/Cabal/Prelude.hs

Line 827 in 00a2351

expectBrokenIf :: Bool -> Int -> TestM a -> TestM ()

?

It has the ticket and can be guarded with os and ghc version iiuic

[jneira]

This will have the nice effect of removing experimental from the linux/macos workflow and the matrix will be lot simpler

[andreabedini]

IMHO using expectBrokenIf would be a clever solution since it declares our expectations more clearly than just skipping the test. 👍 from me.

[jneira]

Great work, many thanks, some reason to not handle PackageTests/NewBuild/T3827?

[jneira]

oops

[robx]

Great work, many thanks, some reason to not handle PackageTests/NewBuild/T3827?

You're welcome!

I just pushed an amended commit to handle T3827 more generally (previously I had skipped just the macos 8.10.7 failure).

[jneira]

maybe we could move this skipIfWindows to expectBrokenIf ((osx || win) && ghc)

maybe it should be in other pr, revising all skipIfWindows and transform as much of them as possible in knownBrokenIfWindows

[robx]

Yeah, I think I'd prefer to leave this out. Skipping might also be more appropriate (in some of those ifWindows cases at least), where we test non-Windows functionality, instead of something that we'd usually expect to work but that's broken for some reason.

(There's a second difference -- expectBrokenIf tests will be run, while skipIf won't. So e.g. for tests that are too slow, we can't use expectBrokenIf.)

[jneira]

Many thanks again, eager to see green checks everywhere again 😺

[Mikolaj]

Thanks a lot!

[jneira]

Ok we can add all jobs to master protection rules!