BUG: can't run tests after JSON reporting addition

[tylerjereddy]

Following the merge of gh-131 a few days ago, our testing with the suite here started to fail as described at kokkos/pykokkos#63

I can reproduce in act and locally as well:

platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/tyler/github_projects/array-api-tests
plugins: forked-1.4.0, hypothesis-6.54.3, metadata-2.0.2, json-report-1.5.0, xdist-2.5.0
collected 9 items                                                                                                                                                                                                                                                                    
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/_pytest/main.py", line 659, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/pluggy/_manager.py", line 262, in check_pending
INTERNALERROR>     raise PluginValidationError(
INTERNALERROR> pluggy._manager.PluginValidationError: unknown hook 'pytest_json_modifyreport' in plugin <module 'conftest' from '/home/tyler/github_projects/array-api-tests/conftest.py'>

From a quick check, this diff gets rid of the problem, but I'm not sufficiently familiar with the hook side of pytest to say if this is what you'd want (and how you'd adjust your CI to catch the original issue).

diff --git a/reporting.py b/reporting.py
index b15b336..f7c7d6b 100644
--- a/reporting.py
+++ b/reporting.py
@@ -91,6 +91,7 @@ def add_extra_json_metadata(request, json_metadata):
 
     request.addfinalizer(finalizer)
 
+@mark.optionalhook
 def pytest_json_modifyreport(json_report):
     # Deduplicate warnings. These duplicate warnings can cause the file size
     # to become huge. For instance, a warning from np.bool which is emitted

In fact, looking more closely at the merged PR, your CI did fail with the same error we are seeing in pykokkos! Should I just pin to a commit for a while or was that accidental?

[asmeurer]

The pytest-json-report plugin should be required. Are you getting this error with it installed? It's in the requirements.txt

With that being said, if this diff makes it optional, I'm fine with that.

Also you should only test against tagged releases. I wonder if there's an easy for people to test the "latest" tag without having to manually update it, but also not just always testing master. Maybe we should make a latest branch which always points to the latest tag, and make that the default branch. CC @honno

[tylerjereddy]

Yes, the error shows up with the plug-in installed. I see it in the CI for this project as well. The diff is needed even with plugin.

…

On Thu, Aug 25, 2022 at 16:32 Aaron Meurer ***@***.***> wrote: The pytest-json-report plugin should be required. Are you getting this error with it installed? It's in the requirements.txt With that being said, if this diff makes it optional, I'm fine with that. Also you should only test against tagged releases. I wonder if there's an easy for people to test the "latest" tag without having to manually update it, but also not just always testing master. Maybe we should make a latest branch which always points to the latest tag, and make that the default branch. CC @honno <https://github.com/honno> — Reply to this email directly, view it on GitHub <#138 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4JOZTBXVMYKSJE2CXW3TTV27X7NANCNFSM57UOEHMQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>