FillBlankFormTest#app_ShouldNotCrash is flakey

[seadowg]

We've seen a this test fail a few times now even though it looks like everything is working correctly in the video (https://console.firebase.google.com/project/api-project-322300403941/testlab/histories/bh.f6f8331e5ae6e371/matrices/8855972944306261818/details?stepId=bs.de451e242e6def7a&testCaseId=5&tabId=video).

[seadowg]

This is in regression, so we should discuss whether we should fix this or not. That's given recent discussion that QA might take these on in a separate project.

[grzesiek2010]

It failes in:

at org.odk.collect.android.support.pages.Page.checkIsSnackbarWithMessageDisplayed(Page.kt:175)
at org.odk.collect.android.regression.FillBlankFormTest.app_ShouldNotCrash(FillBlankFormTest.java:386)

it's because of checking the new snackbar so I don't think there is something wrong with that particular test but maybe generally with testing snackbars?

[seadowg]

Yeah this definitely backs up that theory: https://console.firebase.google.com/project/api-project-322300403941/testlab/histories/bh.f6f8331e5ae6e371/matrices/7884108888395309184/details?stepId=bs.6c6af3002b159be0&testCaseId=14.

I'm wondering if we should move to a similar approach as we have toasts. I guess snackers are inherently hard to UI test as they disappear. We might be able to do something as simple as keeping them on screen indefinitely, another approach would be to wrap snackbar creation in a component that can be switched out during tests for a recording version that we can make assertions on.

[grzesiek2010]

Ok I've spent some time investigating the issue and here are my thoughts:

• watching the videos from failing tests I see that the Snackbar is visible for a short moment and disappears
• I think it happens not because we dismiss it but because it's an old Snackbar (in app_ShouldNotCrash we finalize a few forms so it's possible that you do that for one then starts another one and finalize it too and it takes less than 5s so you still see the old Snaskbar when you are back in the main menu)
• theoretically when an old Snackbar is displayed replacing it should not be a problem but it looks like sometimes it doesn't work
• I have found a similar issue so I guess it's an Android bug https://stackoverflow.com/questions/43680655/snackbar-sometimes-doesnt-show-up-when-it-replaces-another-one

So the solution would be to programmatically dismiss the old Snackbar when a new one is about to be displayed. We already do that with Toasts so it looks like we had a similar issue there.

[grzesiek2010]

I would try something like this #5682 what do you think @seadowg

[seadowg]

theoretically when an old Snackbar is displayed replacing it should not be a problem but it looks like sometimes it doesn't work

That makes sense to me. I'd been trying to work out if Snackbars get hidden when another is shown for something else (unrelated to this issue), and I'd seen that it does look that's handled in SnackbarManager, but it could be that that code is buggy.