Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Superfluous downstream checks in Makefile parallelism #173

Closed
wlandau-lilly opened this issue Nov 26, 2017 · 1 comment
Closed

Superfluous downstream checks in Makefile parallelism #173

wlandau-lilly opened this issue Nov 26, 2017 · 1 comment

Comments

@wlandau-lilly
Copy link
Collaborator

wlandau-lilly commented Nov 26, 2017

In make(..., parallelism = "Makefile"), drake bulids outdated targets and then checks all the downstream targets to see if they are also outdated. This snowball effect is often unnecessary, and it is not present in any of the other parallel backends.

load_basic_example()
make(my_plan)
clean(large)
make(my_plan, parallelism = "Makefile", jobs = 8)
## ...
## check 1 item: large
## target large
## Rscript -e 'drake::mk(target = drake::as_drake_filename("report.md"), cache_path = "C:/projects/.drake")'
## Rscript -e 'drake::mk(target = "regression1_large", cache_path = "C:/projects/.drake")'
## Rscript -e 'drake::mk(target = "regression2_large", cache_path = "C:/projects/.drake")'
## load 1 item: large
## check 1 item: regression1_large
## load 3 items: coef_regression2_small, large, small
## Rscript -e 'drake::mk(target = "summ_regression1_large", cache_path = "C:/projects/.drake")'
## check 1 item: 'report.md'
## Rscript -e 'drake::mk(target = "coef_regression1_large", cache_path = "C:/projects/.drake")'
## load 1 item: large
## check 1 item: regression2_large
## Rscript -e 'drake::mk(target = "summ_regression2_large", cache_path = "C:/projects/.drake")'
## Rscript -e 'drake::mk(target = "coef_regression2_large", cache_path = "C:/projects/.drake")'
## load 1 item: regression1_large
## check 1 item: summ_regression1_large
## load 1 item: regression1_large
## check 1 item: coef_regression1_large
## load 1 item: regression2_large
## check 1 item: summ_regression2_large
## load 1 item: regression2_large
## check 1 item: coef_regression2_large

Standaline Makefiles do not seem to have this problem.

all: c

c: b
	touch -a c

b: a
	touch -a b

a:
	touch -a a

clean:
	rm -f a b c
make
## touch -a a
## touch -a b
## touch -a c
make
## make: Nothing to be done for `all`.

And when I modify a by hand in a text editor, c does not build.

make
## touch -a b
@wlandau-lilly
Copy link
Collaborator Author

wlandau-lilly commented Nov 26, 2017

Fixed, thanks to #168. Added a new next_stage() function inspired by parallel_stages(). FYI: @kendonB, outdated() should be faster now too. It just uses next_stage() to grab the first outdated targets and then assumes everything downstream is also out of date. In other words, all the downstream hashing and checking is skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant