file_in() URL authentication

[wlandau]

Prework

• Read and abide by drake's code of conduct.
• Search for duplicates among the existing issues, both open and closed.

Description

file_in() supports URLs, but there is currently no way to pass along credentials user may need to get eTAGs.

[wlandau]

cc @noamross

[noamross]

One possibility would be for file_in() to take an argument like curl_options() which would set the handle options for checking headers (The nobody option I suggest in #982 would be the default but could be overridden). You would still have to pass your authentication options again in the function call that reads from the URL, but that would be OK I think.

[noamross]

Or maybe file_in should just have a single, second options argument that gets the curl options or anything else that might be thrown at it in the future, which would be easier to carry forward as a property of the target.

[wlandau]

drake handles arguments to file_in() using static code analysis. It detects all the literal strings, puts them in a lightweight hash table, and then converts the hash table into an unnamed character vector. In fact, the dependencies of a piece of code are represented as a list of character vectors.

drake:::analyze_code( # Walk the AST of some code.
  quote({
    base::sqrt(x)
    file_in("a")
    file_out("b")
  })
)
#> $globals
#> [1] "x"
#> 
#> $namespaced
#> [1] "n-MJQXGZJ2HJZXC4TU"
#> 
#> $file_in
#> [1] "p-ME"
#> 
#> $file_out
#> [1] "p-MI"

^{Created on 2019-08-08 by the reprex package (v0.3.0)}

I hesitate to make these internals more complicated for the sake of authentication. On the other hand, we could add an argument to make() to allow users to supply credentials for each domain. Does that sound reasonable?

[noamross]

The thing that worries me is the variety of mechanisms that sites use for authentication. But I think if one provides a handle, one could load it with tokens, cookies, passwords, or other header variables.

Handles could vary by different URLs within a domain, so maybe you can base the it on a partial match? e.g.,

curl_handles = list(
`https://ropensci.org/endpoint1/` = curl::new_handle(...),
`https://ropensci.org/endpoint2/` = curl::new_handle(...)
)

And https://ropensci.org/endpoint1/ would apply to https://ropensci.org/endpoint1/values, etc.

If the user provides handles then we should probably modify these handles to add the nobody option when rehashing the URL with something like:

if(!is.null(handle)) {
  handle <- curl::handle_setopt(handle, nobody = TRUE)
} else {
  handle <- new_handle(nobody = TRUE)
}

[wlandau]

I like your idea of supplying a named list of curl handles because it generalizes beyond authentication. And I think we can use partial matching as long as we explain it.

[wlandau]

With #982 merged, I think this issue is now actionable. The test case at https://httr.r-lib.org/reference/authenticate.html looks sufficient.

[wlandau]

I incorporated #981 (comment) into 0ba179b. Let's follow up again if it does not work for you.