Support passing arguments to the test runner

[bkhouri]

I want to create my own macro around using the rules_bazel_integration_test and I created a custom test runner in python that uses pytest. I want to be able to specify a default set of "tests" required to pass on each target type, eg.: "bazel info" and "bazel query //..."

I would also like to have each caller of the custom macro specify unique tests. My macro is essentially this

def __get_label(label):
    """
    Return a Label object representation.

    Args:
        label: a string label"""
    if label.startswith("@") or label.startswith("//"):
        return label
    else:
        return Label("//{}:{}".format(native.package_name(), label.lstrip(":")))

def __bazel_py_intetration_test(
        name,
        srcs,
        workspace_path,
        run_files,
        main = None,
        tags = None,
        deps = [],
        env = {},
        target_compatible_with = None):
    test_scenario_name = "{}_scenario".format(name)
    tags = tags or []
    srcs = [__get_label(src) for src in srcs] + [
        "@//bazel/integration_tests:bit_python_runner.py",  ## The test runner py_library
        "@//bazel/integration_tests/py:common_py_bit_tests",  # common tests
    ]
    python.binary(
        name = test_scenario_name,
        main = ":bit_python_runner.py",
        srcs = srcs,
        args = [
            "--fixtures-per-test",
            "-rpa",
            "-vvvv",
        ] + ["$(locations {})".format(src) for src in srcs],
        deps = deps+[
           # any dependencies required for :bit_python_runner.py
        ],
        testonly = True,
    )

    workspace_files = integration_test_utils.glob_workspace_files(workspace_path)

    workspace_files = [
        "@//:local_repository_files",  # all files in the main workspace
    ] + workspace_files + run_files

    # Declare the integration tests.
    bazel_integration_test(
        name = name,
        bazel_binary = "@//tools:bazel",
        test_runner = ":{}".format(test_scenario_name),
        env = env,
        tags = ["exclusive", "bit"] + tags,
        workspace_files = workspace_files,
        workspace_path = workspace_path,
        target_compatible_with = target_compatible_with,
    )

The args value of the python.binary does not get called when invoking the bazel_integration_test.

My end goal is to be able to define a test once, and have said test execute against all "bazel_integration_test" targets. One option is to allow passing arguments to the test runner

[cgrindel]

The issue with why the args is not called is subtle. rules_bazel_integration_test passes the test_runner to the underlying test using the data attribute. This makes the binary and its runfiles available to the test. However, it does not serialize/preserve arguments that are passed to the test runner via the args attribute. In other words, the test runner binary needs to have the arguments serialized in the binary or the runfiles.

There are several ways to tackle this:

1. Update your macro to write a script that includes the arguments that you would like preserved. Then, pass that script as the test runner to bazel_integration_test. However, if you are passing file paths, you need to ensure that those files are included in the runfiles for the binary. See the next two options before going down this path.
2. Use the execute_binary rule to create a binary target that includes the provided binary and the arguments. This rule collects arguments and their related files, puts them into a package so that it can be copied and executed elsewhere.
3. Use the execute_binary_utils Starlark module to write your own binary rule that serializes the arguments. See default_test_runner as an example.