Implement high-level integration tests for the reference application #118

hbatagelo · 2024-04-29T19:26:13Z

Adds integration tests for the reference application.

This requires mir-flutter-app to be bundled into the mir-test-tools snap (related PR).

hbatagelo · 2024-04-29T19:39:44Z

mir-ci/mir_ci/tests/robot/platforms/wayland/Screencopy.py

This is a bug fix. self.frame_count accessed in match() could contain a frame number out of sync with the grabbed screenshot. This fix makes sure that the frame count checked in match() is the actual frame count when grab_screenshot() was called.

Would this be better in another PR?

Moved to #119.

mattkae · 2024-04-30T14:50:59Z

Tests are failing by the way @hbatagelo , but I am reviewing!

mattkae · 2024-04-30T14:51:55Z

Oh wait. Is that because canonical/mir-test-tools#76 isn't in yet?

mattkae

I have some initial questions and minor feedback!

mattkae · 2024-04-30T15:16:08Z

mir-ci/mir_ci/tests/robot/platforms/wayland/Screencopy.py

Would this be better in another PR?

mir-ci/mir_ci/tests/test_mir_flutter_app.py

RAOF · 2024-05-01T09:03:47Z

mir-ci/mir_ci/fixtures/servers.py

@@ -22,7 +22,8 @@ class ServerCap(Flag):
    DISPLAY_CONFIG = auto()
    SCREENCOPY = auto()
    INPUT_METHOD = auto()
-    ALL = FLOATING_WINDOWS | DRAG_AND_DROP | DISPLAY_CONFIG | SCREENCOPY | INPUT_METHOD
+    MIR_FLUTTER_APP = auto()
+    ALL = FLOATING_WINDOWS | DRAG_AND_DROP | DISPLAY_CONFIG | SCREENCOPY | INPUT_METHOD | MIR_FLUTTER_APP


It's kinda weird that the rest of these are server capabilities, and then there's MIR_FLUTTER_APP. Could this instead be a MIR_SHELL_PROTO capability (denoting the support for the mir_shell protocol), and then have the test case run the flutter app against the server selected?

That would fit in better with how the rest of these tests work, and also allow us to test different servers that support mir_shell? (ie: maybe we want to test both miriway and miral-shell?)

Yes, mir_demo_server.mir_flutter_app is just the demo server running an app. It doesn't seem to fit here, but do I have another option? Using MIR_SHELL_PROTO would work great if, instead of bundling mir_flutter_app into mir-test-tools, mir_flutter_app had its own snap. This was actually what I did in my first commit, running a mir_flutter_app snap in the servers that support mir_shell, like miriway and miral-shell (IIRC, I used MIR_SHELL to tag the compatible servers, but then changed to MIR_FLUTTER_APP).

In the current state, MIR_SHELL_PROTO can only be used with mir_demo_server.mir_flutter_app because test_mir_flutter_app.py won't run any application as a subprocess. If we tag miriway or confined_server with MIR_SHELL_PROTO, the test will just time out because there will be no application to test.

@hbatagelo why do we have the reference app "bundled" with a server? Why not just run a server like we usually do, and run the app itself separately, like we do in other tests?

@hbatagelo why do we have the reference app "bundled" with a server? Why not just run a server like we usually do, and run the app itself separately, like we do in other tests?

Oh, I thought MIRENG-479 was about that. I used the other mir-test-tools apps (sdl22-test, qt-test, etc) as a reference, as they are all invoked by mir_demo_server. I'll change the mir-flutter-app snap "app" so it calls mir_flutter_app directly.

hbatagelo · 2024-05-02T21:24:58Z

The Match Any Of keyword uses the fix proposed in #119. I'll change this PR to a draft to prevent it from merging before #119.

Saviq

I would also rather this follow the OSK test MO, where the robot file is a static one stored in the suites and assets are collected into a single folder.

A little copy-pasta now, but will be easier to maintain going forward.

Another thing I'd consider is applying the same "proportional geometry" approach I used in OSKMap.py to reduce the amount of templates that need updating for different variants etc. Maybe even a MirApp.resource makes sense here?

hbatagelo · 2024-05-23T14:06:58Z

The test cases are now contained in a separate robot file. Upon testing, the relevant files (robot file + templates) are collected into a single folder, similar to how test_osk works.
Several template files have been removed because the test cases now use relative positioning whenever possible.
MIR_SHELL_PROTO enum is now used to tag servers that can run the app.
The Match Any Of keyword has been removed because it is not used with relative positioning.

mattkae

One small nit, but this looks good to me otherwsie

mir-ci/mir_ci/tests/test_mir_flutter_app.py

RAOF

This looks good to me, now.

mattkae

With my final nit out of the way, I would be happy to see this merge (maybe if we can make CI a little more green if possible)

Saviq · 2024-05-28T12:44:16Z

Need to resolve the servers.py conflict.

Rebasing on main should improve the test results a bunch.

codecov · 2024-05-28T15:40:18Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.68%. Comparing base (cac5744) to head (7b3444b).
Report is 13 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #118      +/-   ##
==========================================
- Coverage   68.92%   68.68%   -0.24%     
==========================================
  Files          16       16              
  Lines         872      875       +3     
  Branches      126      126              
==========================================
  Hits          601      601              
- Misses        247      249       +2     
- Partials       24       25       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Saviq · 2024-05-29T09:14:31Z

The FlutterApp test fails on mir-test-tools, you can find the failure log in the test-results artifact:

https://github.com/canonical/mir-ci/actions/runs/9272416938?pr=118

It's there in test-results-mir_test_tools/tmp/pytest-of-runner/pytest-current/test_mir_flutter_app_mir_test_current/log.html

Stale

hbatagelo added 2 commits April 23, 2024 17:52

Add high-level integration tests for the reference application

619ed41

Use mir-test-tools.mir-flutter-app

4dd97db

hbatagelo requested a review from a team as a code owner April 29, 2024 19:26

Pre-commit formatting

7181f2b

hbatagelo commented Apr 29, 2024

View reviewed changes

mattkae requested changes Apr 30, 2024

View reviewed changes

hbatagelo added 2 commits April 30, 2024 17:29

Capitalize keywords

2618457

Revert screencopy fix

3f673f1

RAOF reviewed May 1, 2024

View reviewed changes

Add keyword Match Any Of, renamed files and other minor changes

66eb911

hbatagelo marked this pull request as draft May 2, 2024 21:25

Saviq previously requested changes May 8, 2024

View reviewed changes

hbatagelo added 2 commits May 13, 2024 16:31

Remove Click And Sleep

a8286cc

Use external robot file and give preference to relative positioning

ef74b90

hbatagelo marked this pull request as ready for review May 23, 2024 14:12

hbatagelo requested a review from Saviq May 23, 2024 20:50

mattkae requested changes May 24, 2024

View reviewed changes

mir-ci/mir_ci/tests/test_mir_flutter_app.py Show resolved Hide resolved

hbatagelo mentioned this pull request May 24, 2024

Add VNC backend to the Robot kvm resource #120

Merged

RAOF approved these changes May 27, 2024

View reviewed changes

mattkae approved these changes May 28, 2024

View reviewed changes

hbatagelo force-pushed the MIRENG-103/mir-flutter-app branch from 207c8cb to ef74b90 Compare May 28, 2024 15:41

Merge branch 'main' into MIRENG-103/mir-flutter-app

92a3a15

Fix selection of positioner preset

7b3444b

hbatagelo force-pushed the MIRENG-103/mir-flutter-app branch from a6f2868 to 7b3444b Compare May 29, 2024 12:31

Saviq enabled auto-merge June 3, 2024 09:57

Saviq added this pull request to the merge queue Jun 3, 2024

Merged via the queue into main with commit 95c6b45 Jun 3, 2024
10 of 16 checks passed

Saviq deleted the MIRENG-103/mir-flutter-app branch June 3, 2024 09:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement high-level integration tests for the reference application #118

Implement high-level integration tests for the reference application #118

hbatagelo commented Apr 29, 2024

hbatagelo Apr 29, 2024

mattkae Apr 30, 2024

hbatagelo Apr 30, 2024

mattkae commented Apr 30, 2024

mattkae commented Apr 30, 2024

mattkae left a comment

mattkae Apr 30, 2024

RAOF May 1, 2024

hbatagelo May 2, 2024 •

edited

Loading

Saviq May 7, 2024

hbatagelo May 7, 2024

hbatagelo commented May 2, 2024

Saviq left a comment •

edited

Loading

hbatagelo commented May 23, 2024

mattkae left a comment

RAOF left a comment

mattkae left a comment

Saviq commented May 28, 2024

codecov bot commented May 28, 2024 •

edited

Loading

Saviq commented May 29, 2024

Implement high-level integration tests for the reference application #118

Implement high-level integration tests for the reference application #118

Conversation

hbatagelo commented Apr 29, 2024

hbatagelo Apr 29, 2024

Choose a reason for hiding this comment

mattkae Apr 30, 2024

Choose a reason for hiding this comment

hbatagelo Apr 30, 2024

Choose a reason for hiding this comment

mattkae commented Apr 30, 2024

mattkae commented Apr 30, 2024

mattkae left a comment

Choose a reason for hiding this comment

mattkae Apr 30, 2024

Choose a reason for hiding this comment

RAOF May 1, 2024

Choose a reason for hiding this comment

hbatagelo May 2, 2024 • edited Loading

Choose a reason for hiding this comment

Saviq May 7, 2024

Choose a reason for hiding this comment

hbatagelo May 7, 2024

Choose a reason for hiding this comment

hbatagelo commented May 2, 2024

Saviq left a comment • edited Loading

Choose a reason for hiding this comment

hbatagelo commented May 23, 2024

mattkae left a comment

Choose a reason for hiding this comment

RAOF left a comment

Choose a reason for hiding this comment

mattkae left a comment

Choose a reason for hiding this comment

Saviq commented May 28, 2024

codecov bot commented May 28, 2024 • edited Loading

Codecov Report

Saviq commented May 29, 2024

hbatagelo May 2, 2024 •

edited

Loading

Saviq left a comment •

edited

Loading

codecov bot commented May 28, 2024 •

edited

Loading