Migrate Visual Snapshot Tests to Percy

[Sherlouk]

As discussed on Discord.

https://percy.io/

Work with Dave to set everything up.

[daveverwer]

A quick update on this. I heard back from BrowserStack support team this morning and they do not currently include a Percy plan as part of their open-source sponsored account. I've asked if they have plans to add it, but I think I'm speaking to the wrong person about it.

The idea is good, but we wouldn't be able to implement this on their free plan.

[Sherlouk]

Alternatively: consider other third parties other than Percy who may have open-source sponsored accounts!

[daveverwer]

Alternatively: consider other third parties other than Percy who may have open-source sponsored accounts!

Yes, of course. I had a look before posting this and couldn't see any that offer anything obvious. Are there any you'd recommend?

[Sherlouk]

I can't explicitly recommend any others as I've not tried them, but I have a few others which we've done some light investigations on so I'll do some digging and update you on my findings 👍

[finestructure]

Are we still pursuing this or should we close the issue?

[daveverwer]

I chased quite hard and got back two responses that said they are not currently supporting open-source projects with Percy and the cost of the product on its own is way out of our reach.

That said, it would be good to fix this problem. It's not really that great that all snapshot tests need to come through my machine.

[finestructure]

What we could do is set up a hosted runner using one of our Macs at MacStadium to run the snapshots and be the canonical source.

The only difficulty is to make sure whenever we change Xcode versions that machine can and does follow.

[daveverwer]

The other advantage of Percy was that it took the snapshots out of the repo. They are making it quite large already.

Given BrowserStack's track record with open-source, I'm sure they will add Percy to their open-source programme (which is very generous) at some point. Their marketing page for the programme says that it covers "All products" and Percy is listed in the footer of that same page.

I think we keep this open for a little while and I'll add a reminder to chase again in a month or so.

[Sherlouk]

Have we considered other third parties such as Applitools?
Should be same benefits, and has open-source pricing?

[finestructure]

I'm not super excited in general about tying this to a 3rd party service, to be honest. We don't have that many screens and the PR diffs have worked fine so far.

The other advantage of Percy was that it took the snapshots out of the repo. They are making it quite large already.

Unfortunately, this would only affect future growth because the images are in the history already. (Also, at 18M Tests isn't super big I'd say.)