R on high performance clusters

[lmullen]

Late suggestion, but I would be interested in discussing how R can be used on high-performance clusters. In my case that's a university cluster, but I imagine people are doing this in lots of different ways. There are already well established packages for this, such as drake and sparklyr. Perhaps an outcome of this would be a tutorial or guide to using them for some rOpenSci specific applications.

[wlandau]

Ooh I love where this is going! For drake, I just rewrote the vignettes on HPC and timing, plus an rOpenSci tech note with an overview of some new HPC features. For tasks not in reproducible pipelines, I recommend future.batchtools. rslurm looks great for SLURM specifically, though I have not used it.

Traditional HPC can be intimidating at first, and I think more work to reduce the friction would be well spent.

[zachary-foster]

I have also encountered the flowr package, although I have not used it much.

[juyeongkim]

ping @sahilseth

[sahilseth]

thanks @juyeongkim, this sounds exciting. We have been developing several NGS pipelines on flowr. Guess this is a late reply, happy to discuss more, and can skype in.

[wlandau]

I recently encountered clustermq, and I love it. The learning curve is shallow, overhead is tiny, and it requires astonishingly little setup. So far, my experience has been totally frictionless.