[IO-1332][internal] E2E Test harness

[owencjones]

Problem

We lack a simple way to run darwin commands and take output for assertions

Solution

Write a simple wrapping function that runs darwin commands, and returns exit code, and STDOUT and STDERR streams outputs

Changelog

Added run_cli_command function, which wraps CLI commands
Added tests for run_cli_command

[linear]

IO-1332 Create CLI Testing Harness

Pytest has poor support/no support for test CLI commands, so we need a testing harness for the CLI, that is capable of:

• Running a shell command
• Handling the output and returning:
  • Output to STDOUT, STDERR as applicable
  • The shell return code, a signed 8-bit integer in POSIX systems
• Making assertions based on this

E.g.

def test_it_creates_a_dataset():
    stdout, stderr, output_code = run_cli_command("darwin dataset create test-dataset")

    assert stdout.contains("test-dataset created")
    assert output_code == 0
    assert not stderr

This repo that I started work on some time ago may be of help, but is maybe a bit more than we need, but the file https://github.com/owencjones/cli_test/blob/main/cli_test/types.py contains a lot of what we need. It creates an expectation class, which we could use, but the functional approach above is just as good.

[Nathanjp91]

this shouldn't be necessary, patch resets automatically after exiting a decorator

[owencjones]

That's true, originally, I was wrapping several tests in a with mock.patch but removed it because it's very messy. I'll correct in the next ticket because it's in the same area