Run non dask TPC-H benchmarks on schedule or label

[milesgranger]

#1044 (comment)

@crusaderky do you have a preference if these are ran as frequently as existing tests, or shall I add/extend a workflow to run once a day/ect?

[crusaderky]

I think it makes sense to run them only on nightly or when the PR contains a specific tag, exactly like with workflows?

benchmarks/.github/workflows/tests.yml

Lines 99 to 108 in cc4a6e5

	- name: Determine if workflows should be run
	# Run workflows on PRs with `workflows` label and nightly cron job
	if: |
	github.event_name == 'schedule'
	|| (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'workflows'))
	run: |
	# Put EXTRA_OPTIONS into $GITHUB_ENV so it can be used in subsequent workflow steps
	export EXTRA_OPTIONS="--run-workflows"
	echo $EXTRA_OPTIONS
	echo EXTRA_OPTIONS=$EXTRA_OPTIONS >> $GITHUB_ENV

[milesgranger]

@crusaderky see what you think about the added block. No rush though, thanks! 🙏

[mrocklin]

No rush though, thanks!

I'll actually add a bit of a rush to this. I'd like to remove the need to add this keyword to run TPCH tests. Ideally I'd like to do this well before Thursday's demo day (so that I can write and update a readme)

[milesgranger]

Gentle ping @crusaderky

I believe @mrocklin is advocating we ought to just run them all the time, in that case maybe the block added in c44c916 ought to go in favor of just removing the flag done originally in 550c245

[crusaderky]

LGTM; retesting in both configurations before merge

[crusaderky]

No rush though, thanks!

I'll actually add a bit of a rush to this. I'd like to remove the need to add this keyword to run TPCH tests. Ideally I'd like to do this well before Thursday's demo day (so that I can write and update a readme)

+1 for this. I assume @mrocklin is experiencing the same kind of frustration that I face daily when running @slow tests in distributed locally. I would rather have a flag that is ON by default and is added to ci:

on the command line: pytest -m 'not nondask_tpch'
in the header of the test files: pytestmark = pytest.mark.nondask_tpch

This follows the same design of avoid_ci in distributed.

[mrocklin]

I'm actually fine with these never being run on CI. Mostly I do everything here manually. I want to target this directory

py.test tests/tpch

And have it run. I don't want to have to specify --tpch-non-dask every time I do this.

I think that my personal preference here is for us to not flag things on or off, but instead be more precise with our py.test foo commands about which directories to run.

[milesgranger]

That makes sense!
I've added some logic to check if the path(s) passed to pytest include any of the workflows or tpch tests then auto adds the appropriate flag to run those tests. (or technically, doesn't apply the skip)

A simple example:

pytest tests/ -k test_polars will not run any TPC-H Polars tests, where pytest tests/tpch -k test_polars will, because it includes the path into the tpch tests; same goes for pytest tests/workflows will then run all workflows, where pytest tests/ won't without the appropriate --run-workflows flag.

[crusaderky]

LGTM; rerunning in both use cases before I approve