Cancel targets mid-build

[wlandau]

Prework

• Read and abide by drake's code of conduct.
• Search for duplicates among the existing issues, both open and closed.

Problem

drake does not support dynamic file-oriented workflows. If the whole point of a target is to produce a file and you do not know the path in advance, drake cannot track the file in the general case.

drake_plan(
  path = write_data_to_random_file() # file not tracked
)

This limitation creates frustration (example here) and workarounds like #1127 are not a good fit for the package. Clever workarounds with triggers are sometimes possible, e.g. #1127 (comment), but triggers are a static-branching-only feature and will not help with dynamic branching.

Proposal

In situations like these, it could be convenient to cancel a target mid-build if some condition is met. Below, cancel_if() (and possibly file_ok() and file_value()) would be new drake functions. cancel_if(TRUE) should immediately return from the build and tell drake to not store a new value for the target.

write_data_to_random_file <- function() {
  path <- random_file_path()
  cancel_if(file_ok(path)) # Return early if the file is okay.
  data <- long_computation()
  write_data_to_file(data, path)
  # Bill Denney and I each came up with this one independently:
  file_value(path) # structure(path, hash = digest(path, file = TRUE, algo = "xxhash64"))
}

drake_plan(
  path = target(
    write_data_to_random_file(),
    trigger = trigger(condition = TRUE) # Always check the file.
  ),
  # Already responds when the file changes because
  # file_value() decorates the path with the hash.
  analysis = run_analysis(path)
)

Challenges and decisions

1. How do we tell cancel_if() to return to right place from anywhere in the call stack?
2. Should file_ok() be part of drake or an external package as suggested in Special format for files? #1127 (comment)?
3. Is file_value() really the right name for the function?

[wlandau]

I am not sure this approach is logically comprehensive enough to replace triggers, but it's a start. It is worth thinking about what it would take to replace triggers. Triggers, especially condition and change, are cumbersome and difficult to maintain on the implementation side. Plus, they can be slow in certain cases, and they do not apply to individual dynamic sub-targets. In fact, I almost regret implementing triggers in the first place. If there are quicker, more elegant, more dynamic alternatives, it would simplify drake considerably.

[wlandau]

Triggers were originally proposed in #131 to deal with slowness. These days, drake is several orders of magnitude faster, and checking triggers is not a bottleneck. Since then, a bunch more use cases have cropped up, e.g. checking an external API over a network, and we should keep those working somehow. But hopefully in a simpler way if we can think of something appropriate.

[wlandau]

#1132 could help implement (1) and (2). If we can reach the calling environment of with_call_stack(), we can make cancel_if() return from there.

[wlandau]

Just hit a snag: https://stackoverflow.com/questions/59698176/dynamically-move-control-somewhere-else-in-the-call-stack

[wlandau]

I've got it! We just throw a custom error condition. Yet another trick I learned from following @richfitz' work: https://github.com/richfitz/storr/blob/27508231b3c061afed9bb243d4422f81291f2e94/R/exceptions.R#L1-L19