targets + version control + network drives

[matthiasgomolka]

Right now, at our team we are using drake to produce research data sets. I wrote a package which wraps some drake functions and adds additional functionality to make the experience in our specific setting even smoother.

I'll shortly describe our setting, because it relies on the fact that drake can handle arbitrary cache locations.

• Our data lives on a network drive which is accessible to all colleages (I know this is bad, but this is what we have for now; improvements are on the way).
• Each of us has a personal network drive where we store our local copies under version control.
• We share a number of virtual machines. None of them "belongs" to any of us.

Thus,

• our RStudio projects and the code containing the drake plans are stored under version control on the personal network drives and
• the drake caches are stored on the shared network drive.

I would make no sense to store the caches beside the code on the personal network drives because they are very limited in size and it would not make sense to store terabytes of data in a bunch of personal network drives in cases where we work together on the same projects.

It also makes little sense to store the code beneath the caches because then the code under version control would live on a shared network drive and it would be very confusing to have several people working on the same "local copy" under version control.

---

Obviously, I'm thinking of making the switch to targets since it would simplify many steps I've now built into the internal package.

But until now, I've no idea how to solve the dilemma right now where it is just extremely handy that drake handles arbitrary cache locations.

I can only think of creating a different branch in SVN for every person working on the shared code to prevent conflicts. But this would add a lot of complexity because many on my colleages have only little experience with version control.

Therefore I would be very thankful for any suggestions on how to use targets in this context.

[wlandau]

How feasible is it to work in the shared network drive for everything? targets is likely to perform better than drake under network lag, and you can get an idea of the bottlenecks with proffer::pprof(tar_make(callr_function = NULL)) (see the flame graph).

It also makes little sense to store the code beneath the caches because then the code under version control would live on a shared network drive and it would be very confusing to have several people working on the same "local copy" under version control.

Can you create user-specific subdirectories on the shared drive with permissions set to protect colleagues from overwriting each other's work? In my experience, this is standard practice for shared file systems.

I think a cache without accompanying code is a reproducibility problem. Without accompanying code, it is not clear what to do with it in the general case.

[wlandau]

I may have missed something: is it possible to use version control from the shared drive if you have user-specific storage to prevent people from overwriting each other's files? I am all in on Git, and it has been over a decade since I last used SVN.

[matthiasgomolka]

In theory, it should be possible to create user specific directories for the code on the shared network drive. However, this would increase the amount of administrative tasks. I'll think about it. Thanks for the suggestion.

[boshek]

@matthiasgomolka will just flag for you this digest issue which unfortunately is proving to make {targets} unuseable in my Windows + network drive environment because Windows keeps telling R I don't have read permission to the _targets cache. You may have already experienced this or your environment might be different but just sharing my experience here.

[wlandau]

@boshek, please try af44564. I relaxed that digest error to a warning for files because targets already checks files before hashing them. You will get a lot of warnings, but development targets should be usable now.

[boshek]

Actually on further inspection, the workaround suggested in that thread also works here. So something like this in your _targets.R file:

library(digest)

custom_check_file <- function(object, errormode) {
  if (!file.exists(object)) {
    return(.errorhandler("The file does not exist: ",
      object,
      mode = errormode
    ))
  }
  if (!isTRUE(!file.info(object)$isdir)) {
    return(.errorhandler("The specified pathname is not a file: ",
      object,
      mode = errormode
    ))
  }
  if (R.utils::fileAccess(object, 4)) {
    return(.errorhandler("The specified file is not readable: ",
      object,
      mode = errormode
    ))
  }
}

R.utils::reassignInPackage('check_file', 'digest', custom_check_file)

---- rest of _targets.R file

Allows targets to properly read the cache.

[boshek]

@wlandau I will try that. I wrote the above workaround before I saw your comment. Thanks!

[wlandau]

Thanks, and your workaround is a good one for current CRAN targets. Though I recommend it for your project's local .Rprofile file rather than _targets.R because otherwise parallel workers would just load digest as is (tar_make_clustermq(), tar_make_future()).