test_threaded_bad_options not expected to raise AustinError?

[delroth]

Description

On my system test_threaded_bad_options does not pass, instead failing due to an AustinError. I don't really understand how this test is supposed to work right now, it's by design trying to make austin fail to start, but doesn't assert that an exception was raised.

This is unlike test_simple_bad_options and test_async_bad_options which both have a with raises(AustinError).

IMO this should also have a with raises(AustinError), but that raises the question of why the test is currently passing in your CI environment and not on my local system... My local setup is slightly weird (I'm working on packaging austin, austin-python and austin-tui for NixOS) but looking at the code I don't understand how this could not fail.

Steps to Reproduce

1. (probably some unknown preconditions which are met on my system but not on CI)
2. pytest

Expected behavior: All tests pass.

Actual behavior: test_threaded_bad_options is the only failure, with the following stack trace:

============================= test session starts ==============================
platform linux -- Python 3.10.7, pytest-7.1.3, pluggy-1.0.0
rootdir: /build/source
collected 61 items / 3 deselected / 58 selected

test/test_aio.py .....                                                   [  8%]
test/test_cli.py ........                                                [ 22%]
test/test_config.py ..                                                   [ 25%]
test/test_semver.py ......                                               [ 36%]
test/test_simple.py ......                                               [ 46%]
test/test_threads.py ...F                                                [ 53%]
test/format/test_compress.py ..                                          [ 56%]
test/format/test_mojo.py ...                                             [ 62%]
test/format/test_pprof.py .                                              [ 63%]
test/format/test_speedscope.py ...                                       [ 68%]
test/stats/test_austin_file_reader.py .                                  [ 70%]
test/stats/test_austin_stats.py ....                                     [ 77%]
test/stats/test_frame.py ....                                            [ 84%]
test/stats/test_hierarchical_stats.py ...                                [ 89%]
test/stats/test_sample.py ....                                           [ 96%]
test/tools/test_diff.py s                                                [ 98%]
test/tools/test_resolve.py x                                             [100%]

=================================== FAILURES ===================================
__________________________ test_threaded_bad_options ___________________________

    def test_threaded_bad_options():
        austin = TestThreadedAustin(terminate_callback=lambda *args: None)
        austin.start(["-I", "1000", "python", "-c", "for i in range(1000000): print(i)"])
>       austin.join()

test/test_threads.py:124:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
austin/threads.py:106: in join
    raise self._exc
austin/threads.py:75: in _thread_bootstrap
    SimpleAustin.start(self, args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test.test_threads.TestThreadedAustin object at 0x7ffff620d8a0>
args = ['-I', '1000', 'python', '-c', 'for i in range(1000000): print(i)']

    def start(self, args: List[str] = None) -> None:
        """Start the Austin process."""
        try:
            self.proc = subprocess.Popen(
                [self.binary_path] + ["-P"] + (args or sys.argv[1:]),
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )
        except FileNotFoundError:
            raise AustinError("Austin executable not found.") from None

        if not self.proc.stdout:
            raise AustinError("Standard output stream is unexpectedly missing")
        if not self.proc.stderr:
            raise AustinError("Standard error stream is unexpectedly missing")

        try:
            if not self._read_meta():
>               raise AustinError("Austin did not start properly")
E               austin.AustinError: Austin did not start properly

austin/simple.py:96: AustinError
=============================== warnings summary ===============================
test/test_aio.py::test_async_time
  /build/source/test/test_aio.py:76: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(

test/test_aio.py::test_async_memory
  /build/source/test/test_aio.py:103: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(

test/test_aio.py::test_async_terminate
  /build/source/test/test_aio.py:133: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(

test/test_aio.py::test_async_bad_options
  /build/source/test/test_aio.py:155: DeprecationWarning: There is no current event loop
    asyncio.get_event_loop().run_until_complete(

test/format/test_mojo.py: 200000 warnings
  /nix/store/fkcl1wzq3106qqgl84bhgk1lp56q6bzg-python3-3.10.7/lib/python3.10/random.py:370: DeprecationWarning: non-integer arguments to randrange() have been deprecated since Python 3.10 and will be removed in a subsequent version
    return self.randrange(a, b+1)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test/test_threads.py::test_threaded_bad_options - austin.AustinError: ...
= 1 failed, 55 passed, 1 skipped, 3 deselected, 1 xfailed, 200004 warnings in 15.64s =

Reproduces how often: 100% on my system, but that's not particularly helpful to you :)

Versions

Python 3.10, austin 3.4.1, austin-python 1.4.1.

Additional Information

n/a

[delroth]

Oh, I think I found something which might repro outside of my system:

pytest -k 'not test_threaded_invalid_binary'

This causes test_threaded_bad_options to fail. There must be some state leakage between the two tests.

(In my Nix package test_threaded_invalid_binary is disabled because we don't look up austin from PATH, the way Nix works is to instead reference the executable dependency directly. I should have mentioned it in the initial issue message but didn't imagine it would be relevant... woops.)

[delroth]

Even better repro: running just test_threaded_bad_options in isolation fails:

[nix-shell:/tmp/source]$ pytest -k 'test_threaded_bad_options'
=============================================== test session starts ===============================================
platform linux -- Python 3.10.7, pytest-7.1.3, pluggy-1.0.0
rootdir: /tmp/source
collected 61 items / 60 deselected / 1 selected

test/test_threads.py F                                                                                      [100%]

==================================================== FAILURES =====================================================
____________________________________________ test_threaded_bad_options ____________________________________________

    def test_threaded_bad_options():
        austin = TestThreadedAustin(terminate_callback=lambda *args: None)
        austin.start(["-I", "1000", "python", "-c", "for i in range(1000000): print(i)"])
>       austin.join()

test/test_threads.py:124:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
austin/threads.py:106: in join
    raise self._exc
austin/threads.py:75: in _thread_bootstrap
    SimpleAustin.start(self, args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test.test_threads.TestThreadedAustin object at 0x6447ac7ac880>
args = ['-I', '1000', 'python', '-c', 'for i in range(1000000): print(i)']

    def start(self, args: List[str] = None) -> None:
        """Start the Austin process."""
        try:
            self.proc = subprocess.Popen(
                [self.binary_path] + ["-P"] + (args or sys.argv[1:]),
                stdout=subprocess.PIPE,
                stderr=subprocess.PIPE,
            )
        except FileNotFoundError:
            raise AustinError("Austin executable not found.") from None

        if not self.proc.stdout:
            raise AustinError("Standard output stream is unexpectedly missing")
        if not self.proc.stderr:
            raise AustinError("Standard error stream is unexpectedly missing")

        try:
            if not self._read_meta():
>               raise AustinError("Austin did not start properly")
E               austin.AustinError: Austin did not start properly

austin/simple.py:96: AustinError
============================================= short test summary info =============================================
FAILED test/test_threads.py::test_threaded_bad_options - austin.AustinError: Austin did not start properly
======================================== 1 failed, 60 deselected in 0.18s =========================================