Skip to content

Commit

Permalink
modify cli-test logic : ignore stderr message by default
Browse files Browse the repository at this point in the history
Previously, cli-test would, by default, check that a stderr output is strictly identical to a saved outcome.
When there was no instructions on how to interpret stderr, it would default to requiring it to be empty.

There are many tests cases though where stderr content doesn't matter, and we are mainly interested in the return code of the cli.
For these cases, it was possible to set a .ignore document, which would instruct to ignore stderr content.

This PR update the logic, to make .ignore the default.
When willing to check that stderr content is empty, one must now add an empty .strict file.

This will allow status message to evolve without triggering many cli-tests errors.
This is especially important when some of these status include compression results, which may change as a result of compression optimizations.
It also makes it easier to add new tests which only care about the CLI's return code.
  • Loading branch information
Cyan4973 committed Jan 26, 2023
1 parent 7b3f03b commit 3c21522
Showing 1 changed file with 1 addition and 4 deletions.
5 changes: 1 addition & 4 deletions tests/cli-tests/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -395,12 +395,10 @@ def _check_output(self, out_name: str) -> None:
return self._check_output_exact(out_name, read_file(exact_name), exact_name)
elif os.path.exists(glob_name):
return self._check_output_glob(out_name, read_file(glob_name))
elif os.path.exists(ignore_name):
else:
check_name = f"check_{out_name}"
self._success[check_name] = True
self._message[check_name] = f"{out_name} ignored!"
else:
return self._check_output_exact(out_name, bytes(), exact_name)

def _check_stderr(self) -> None:
"""Checks the stderr output against the expectation."""
Expand Down Expand Up @@ -738,4 +736,3 @@ def setup_zstd_symlink_dir(zstd_symlink_dir: str, zstd: str) -> None:
sys.exit(0)
else:
sys.exit(1)

0 comments on commit 3c21522

Please sign in to comment.