Experiment with clustermq for persistent workers

[wlandau]

Summary

This PR is for discussion only. Do not merge.

Implementing persistent clustermq was relatively easy:

drake/R/clustermq_persistent.R

Lines 3 to 34 in 0718982

	run_clustermq_persistent <- function(config){
	if (!requireNamespace("clustermq")){
	drake_error(
	"drake::make(parallelism = \"clustermq_persistent\") requires ",
	"the clustermq package: https://github.com/mschubert/clustermq.",
	config = config
	)
	}
	prepare_distributed(config = config)
	mc_init_worker_cache(config)
	console_persistent_workers(config)
	path <- normalizePath(config$cache_path, winslash = "/")
	rscript <- grep(
	"Rscript",
	dir(R.home("bin"), full.names = TRUE),
	value = TRUE
	)
	tmp <- system2(
	rscript,
	shQuote(c("-e", paste0("drake::remote_master('", path, "')"))),
	wait = FALSE
	)
	clustermq::Q(
	worker = mc_worker_id(seq_len(config$jobs)),
	cache_path = config$cache_path,
	fun = function(worker, cache_path){
	drake::remote_worker(worker = worker, cache_path = cache_path)
	},
	n_jobs = config$jobs
	)
	finish_distributed(config = config)
	}

Workers initiate quickly, but just as I thought, execution still suffers from a lot of (probably target-level) overhead. Transient workers, with caching on the master process, are the best way to use clustermq.

Related GitHub issues

• Ref: Implement a clustermq backend with persistent workers #425, Support clustermq as backend for drake mschubert/clustermq#86

Checklist

• I have read drake's code of conduct, and I agree to follow its rules.
• I have read the guidelines for contributing.
• I have listed any substantial changes in the development news.
• I have added testthat unit tests to tests/testthat to confirm that any new features or functionality work correctly.
• I have tested this pull request locally with devtools::check()
• This pull request is ready for review.
• I think this pull request is ready to merge.

[codecov-io]

Codecov Report

Merging #431 into master will decrease coverage by 0.53%.
The diff coverage is 12.12%.

@@            Coverage Diff             @@
##           master     #431      +/-   ##
==========================================
- Coverage     100%   99.46%   -0.54%     
==========================================
  Files          66       67       +1     
  Lines        5349     5379      +30     
==========================================
+ Hits         5349     5350       +1     
- Misses          0       29      +29

Impacted Files	Coverage Δ
R/clustermq_persistent.R	0% <0%> (ø)
R/parallel_ui.R	100% <100%> (ø)	⬆️
R/future_lapply.R	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8948b17...c09644d. Read the comment docs.

[lintr-bot]

inst/examples/sge_future/run.R:13:3: style: Commented code should be removed.

​# make(my_plan, parallelism = "future_lapply", jobs = 4) # persistent workers
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[wlandau]

This solution relies too much on the file system still. A clustermq backend should maintain a common pool of non-blocking persistent refreshable workers (spawned with clustermq::workers()) which the master can send targets as they become ready. Ref: mschubert/clustermq#86 (comment).