Re-write flag values to place generated files in undeclared outputs dir

[amartani]

Some common pytest plugins, such as pytest-reportlog, pytest-json-report and pytest-html, allow the user to generate reports, accepting a file path as a flag. In Bazel, these generated files can only be accessed if placed inside the test undeclared outputs dir. However, bazel doesn't provide any way to expand environment variable values provided via args.

This commit rewrites flag values for the plugins mentioned above, prepend the test undeclared output dir folder to the value of the corresponding flags. This allows passing relative paths to these flags (via args = [] or via --test_arg), and the shim fixes the flag values.

[caseyduquettesc]

Great feature!

[amartani]

@caseyduquettesc Thanks for accepting the PR!

Small question - when trying out this with pytest-html, I noticed that the html report didn't contain captured output, and this happened due to the rule setting --capture=no.

So, in my repo, I override it back (args = ["--capture=fd"]). But I am curious on why --capture=no was added, since --capture=fd seems to work fine for me. Is there any gotchas that you found with it?

[caseyduquettesc]

@amartani Yeah there is a gotcha. If output is captured, --test_output=streamed won't work, or any of the test output options because bazel no longer controls the output.

However, I was just refreshing myself on what capture does and I see that there's an option to both capture and stream the output (--capture=tee-sys), which might address your issue without regressing anything. Would you mind trying out that flag and letting me know if the output shows up in your report?

[amartani]

That does work, but I think I would still personally prefer the default pytest behavior by default, and tell people to use --test_output=streamed --test_arg=--capture=no when needed (possibly with a .bazelrc alias for that). I think pytest's default is useful, as it only prints stdout / logs of failing tests (similar to bazel's --test_output=errors but at a test case level), and in a more "organized" way (instead of mixed with pytest's own output).

That said, it's easy to override, both when using the rule from BUILD files and also on the command line, so feel free to keep the current default if it is useful for you.