Set up automated testing on hardware

[jessebraham]

Over the coming weeks we will come up with a plan for adding automated testing on hardware for this repository, likely as a scheduled workflow.

As the path forward becomes more clear I will continue to update this issue with information and tasks as needed.

• Set up a self-hosted runner with the required devices physically connected
  • Come up with a full list of required devices and ensure all are connected
• Create a workflow for Github Actions to invoke the testing process

Currently blocked on #466

[jessebraham]

A self-hosted runner has been set up with a number of devices connected to it.

[SergioGasquez]

For reference, here is the test suite that esptool is currently using:

• Source
• Docs

[jessebraham]

I think this will need to slip into next quarter, unfortunately. Once HIL testing in esp-hal has been sorted I can shift my focus here (unless somebody else feels like taking over).

[achxkloel]

New updates for HIL testing

Test branch - https://github.com/esp-rs/espflash/tree/test/hil

Problem with standard input

Github Actions are designed for non-interactive use and
workflows will not be able to provide input to running commands.
TTY is not allocated for the remote session created by Github Actions,
so there isn't any terminal / device connected to standard input.

When espflash monitor or espflash flash --monitor <IMAGE> commands are executed,
they start listening to the key input from the keyboard:

Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

If there is no input connected, it fails immediately with next error:

Error:   × Failed to initialize input reader

You can easily reproduce this error in your terminal by running one of the following commands:

exec 0< /dev/null
exec 0<&-
exec 0>&-

And then run espflash monitor on some port.

script workaround

There is workaround which prepares TTY for Github Actions workflow steps - https://github.com/gfx/example-github-actions-with-tty?tab=readme-ov-file
Related issue - actions/runner#241

Example:

  - name: monitor test
    env:
      ESPFLASH_PORT: /dev/serial_ports/${{ matrix.board }}
    shell: 'script -qec "bash {0}"'
    run: timeout 10s espflash_app/espflash monitor

It works fine, but it's still not what we need.
We must reset chip to start monitoring, but it's imposible, because there is no interaction.

* Important note from script command man page:

It is not recommended to run script in non-interactive shells.  The inner shell of script
is always interactive, and this could lead to unexpected results.

Reset solution

New feature with before and after arguments allows to monitor a target without reseting it - #561

Updated workflow step:

  - name: monitor test
    env:
      ESPFLASH_PORT: /dev/serial_ports/${{ matrix.board }}
    shell: 'script -qec "bash {0}"'
    run: timeout 10s espflash_app/espflash monitor -b no-reset-no-sync -c ${{ matrix.board }}

Now espflash monitor may give some output after flashing to serial port.

Serial port output is missing

Github Actions doesn't show output from serial port.
So the program is actually running, but there's no way to verify that.
This only happens when running through the commands: espflash monitor and espflash flash --monitor <IMAGE>

Here are some examples:

espflash flash

espflash_app/espflash flash --port /dev/ttyUSB0 espflash/esp32s3_app/hello_world

Expected output:

[2024-02-01T11:10:44Z INFO ] Serial port: '/dev/ttyUSB0'
[2024-02-01T11:10:44Z INFO ] Connecting...
[2024-02-01T11:10:44Z INFO ] Using flash stub
Chip type:         esp32s3 (revision v0.1)
Crystal frequency: 40 MHz
Flash size:        8MB
Features:          WiFi, BLE
MAC address:       f4:12:fa:40:56:ac
App/part. size:    92,432/1,048,576 bytes, 8.82%
[00:00:00] [========================================]      13/13      0x0
[00:00:00] [========================================]       1/1       0x8000
[00:00:00] [========================================]      22/22      0x10000
[2024-02-01T11:10:46Z INFO ] Flashing has completed!

Result:

[2024-02-01T11:10:44Z INFO ] Serial port: '/dev/ttyUSB0'
[2024-02-01T11:10:44Z INFO ] Connecting...
[2024-02-01T11:10:44Z INFO ] Using flash stub
Chip type:         esp32s3 (revision v0.1)
Crystal frequency: 40 MHz
Flash size:        8MB
Features:          WiFi, BLE
MAC address:       f4:12:fa:40:56:ac
App/part. size:    92,432/1,048,576 bytes, 8.82%
[2024-02-01T11:10:46Z INFO ] Flashing has completed!

espflash monitor

espflash_app/espflash monitor -b no-reset-no-sync -c esp32c3 --port /dev/ttyUSB0

Expected output:

[2024-02-01T11:12:56Z WARN ] Pre-connection option 'NoResetNoSync' was selected. Connection may fail if the chip is not in bootloader or flasher stub mode.
[2024-02-01T11:12:56Z INFO ] Serial port: '/dev/ttyUSB0'
[2024-02-01T11:12:56Z INFO ] Connecting...
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

Hello world!
Hello world!
Hello world!
Hello world!
Hello world!

Result:

[2024-02-01T11:12:56Z WARN ] Pre-connection option 'NoResetNoSync' was selected. Connection may fail if the chip is not in bootloader or flasher stub mode.
[2024-02-01T11:12:56Z INFO ] Serial port: '/dev/ttyUSB0'
[2024-02-01T11:12:56Z INFO ] Connecting...
Commands:
    CTRL+R    Reset chip
    CTRL+C    Exit

cat alternative

However, if we read the contents of the port using the cat command, it will work.

Example:

  - name: cat serial port
    env:
      ESPFLASH_PORT: /dev/serial_ports/${{ matrix.board }}
    run: timeout 5s cat ${{ env.ESPFLASH_PORT }}

But this way we can't verify espflash subcommands.