Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skipping slow tests #8

Open
alexmojaki opened this issue Sep 22, 2022 · 1 comment
Open

Skipping slow tests #8

alexmojaki opened this issue Sep 22, 2022 · 1 comment

Comments

@alexmojaki
Copy link
Contributor

alexmojaki commented Sep 22, 2022

When I run ./scripts/run-tests.sh, all tests show PASSED except two:

tests/test_metrics.py::test_nubia FAILED
tests/test_metrics.py::test_gold_paragraphs SKIPPED (Parses all PDFs - very slow)
  1. test_nubia fails with an obscure pydantic validation error, I had to add a print to understand the problem. How about adding @mark.skipif(not settings.OUGHT_INFERENCE_API_KEY, reason='...')?
  2. Both tests have @mark.slow but this doesn't do anything by default. I assume this is why test_gold_paragraphs has a hard @mark.skip. This doesn't seem great - what do you do when you actually want to run the test? Should a way to skip slow tests by default (with or without using @mark.slow) be investigated?
  3. -m 'not slow' correctly deselects the slow tests, but only if I directly run docker compose and pytest. ./scripts/run-tests.sh -m 'not slow' doesn't work, and the shell (tested both bash and zsh) shows it running with -m 'not\' slow. Looks like this isn't working:

# https://superuser.com/questions/403263/how-to-pass-bash-script-arguments-to-a-subshell
extra_pytest_args="$(printf "${1+ %q}" "$@")" # Note: this will have a leading space before the first arg

@lslunis
Copy link
Collaborator

lslunis commented Sep 23, 2022

This definitely seems worth fixing as part of an overhaul of all the scripts: #7 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants