checkbox remote needs a way to resume a test session #22
Comments
|
This thread was migrated from launchpad.net https://launchpad.net/~pieq wrote on 2021-11-03 07:54:15:The following just happened to me using Checkbox remote through the cdts snap: cdts 0.8 180 20.04/stable ce-certification-qa classic I installed a new image to test on the DUT, installed checkbox20 and cdts snaps, then ran At the end of the testing session, I saw the Re-Run screen. I selected a few jobs that I wanted to re-run, then pressed R. While this happened, my host froze and I had to reboot it. When I tried to reconnect to the DUT using In the sessions directory, I can see a lot of sessions, even though I only selected a test plan once: $ ls /var/tmp/checkbox-ng/sessions/ I attached all the sessions into sessions.tar.xz. How can I wrap up the testing session, and generate the submission.tar.xz? https://launchpad.net/~pieq wrote on 2021-11-15 03:22:57:Changing importance to Critical, because it potentially leads to a lot of time wasted by QA:
In both cases, a lot of time is spent understanding what's going on, trying to salvage whatever can be salvaged, and re-organizing the test runs to isolate the job that will mess up the entire session, so it can be run separately – or not at all. It is therefore very important that Checkbox remote gets this "resume session" feature that local Checkbox run already has. It could be presented as a list of available, not-yet-finished sessions, and the user would select the one (s)he wants to resume, then press Enter to resume. P.S.: A full, wrapped-up submission is mandatory in order to be able to generate an HTML report that will be shared with project managers and customers. https://launchpad.net/~pieq wrote on 2021-12-01 08:10:09:It happened to me again today, with another customer project, this time when running the firmware/fwts_desktop_diagnosis job. The job crashes the DUT with this error: [13337.339760] Call Trace: On the Host, I see: Reconnecting... I manually shut down and restart the DUT. Then the Host presents me... the list of available test plans. I have the following sessions: $ ll -latrh but checkbox remote seems to be ignoring them. https://launchpad.net/~pieq wrote on 2022-03-18 09:06:49:Annnnnd it happened again while running the RTC battery test on a project device. On my remote, I'm using cdts 0.9 278 20.04/stable ce-certification-qa classic And on the project (conroe), I'm using a checkbox snap made for the project with the same checkbox20 snap. The RTC battery test stops the DUT, then the DUT wakes up on its own after 30 seconds. While it is booting, I can see "Reconnecting..." on my remote device, and once DUT finishes booting... the "Select test plan" screen appears on my remote device. $ ls -latrh /var/tmp/checkbox-ng/sessions/
|
pieqq
changed the title
pieqq
changed the title
* Add connection test for kuiper Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Update format and name of rules Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Move the rule out of the if statement to make the condition shorter Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Set `SHARED` to share source instance across rules Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Print error logs for failed tests - add error logs - change log level to debug - improve log words - redirect stdout to stderr when test fails Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Add `print_error_log()` Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Add `print_error_log()` in utils Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Update Vault's status check Signed-off-by: Mengyi <mengyi.wang@canonical.com> * Capture service errors on checkbox test failure (#24) * Capture service errors on checkbox test failure Signed-off-by: Mengyi Wang <mengyi.wang@canonical.com>
Fix: wrong name of after-suspend-eeprom-automated
Fix: wrong name of after-suspend-eeprom-automated
This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1936477
Summary
Description
It is far too easy for the CDTS master to lose connection with the SUT. When this happens, you end up back at the main selection screen, and the previous test session is completely lost.
Even trying to select tests to rerun them can cause the session to be lost.
Running Checkbox on the SUT locally can detect interrupted sessions and allow you to resume.
Steps to reproduce
systemctl status checkbox-ng.service
● checkbox-ng.service - Checkbox Remote Service
Loaded: loaded (/lib/systemd/system/checkbox-ng.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-11-10 10:01:37 CST; 5 days ago
Main PID: 1455 (checkbox-cli)
Tasks: 1 (limit: 19051)
Memory: 28.1M
CGroup: /system.slice/checkbox-ng.service
└─1455 /usr/bin/python3 /usr/bin/checkbox-cli service
(Please note: the service name might be different if you use CDTS or Checkbox as a snap. For CDTS for instance, it's
snap.cdts.service.service)$ checkbox-cli remote <DUT_IP>
Select a test plan and start the test run (for instance, an automated test plan or a stress test plan such as client-cert-iot-server-20-04-[automated|stress]).
While testing is ongoing, restart your host computer.
After restart, try connecting back to the DUT:
$ checkbox-cli remote <DUT_IP>
Expected result
Checkbox prompts to ask if you want to resume the session.
Actual result
Checkbox starts afresh, showing you the list of available test plans in order to start a new test run.
Attachments
sessions.tar.xz
202110-29555-sessions.tar.gz
Tags:
['checkbox', 'checkbox-session-resume']
The text was updated successfully, but these errors were encountered: