Cross failing when called with previous targets

[bart1]

When I generate targets (in this case dataCutPerMonth) based on two other targets (dataSet and dataRadar) by crossing these two targets I seem to not get the right combinations. In this case I do not have any targets for radar a in month 9 and radar b in month 10. There seems to be a failure in crossing here.

require(drake)
#> Loading required package: drake
months=9:10 
years=c(2015, 2016)
plann<-drake_plan(
  dataRadar=target(get_radar_info(radar), transform=map(radar=c('a','b'))),
  dataSet=target(getData(year, month), 
                              transform=cross(year=!!years, month=!!months)),
  dataCutPerMonth=target(someCropFun(dataSet,dataRadar),
                                              transform=cross(dataRadar, dataSet)),
    trace = T
)
plann$target
#>  [1] "dataRadar_a"                                   
#>  [2] "dataRadar_b"                                   
#>  [3] "dataSet_2015_9L"                               
#>  [4] "dataSet_2016_9L"                               
#>  [5] "dataSet_2015_10L"                              
#>  [6] "dataSet_2016_10L"                              
#>  [7] "dataCutPerMonth_dataRadar_a_dataSet_2015_10L"  
#>  [8] "dataCutPerMonth_dataRadar_a_dataSet_2015_10L_2"
#>  [9] "dataCutPerMonth_dataRadar_b_dataSet_2015_9L"   
#> [10] "dataCutPerMonth_dataRadar_b_dataSet_2015_9L_2" 
#> [11] "dataCutPerMonth_dataRadar_a_dataSet_2016_10L"  
#> [12] "dataCutPerMonth_dataRadar_a_dataSet_2016_10L_2"
#> [13] "dataCutPerMonth_dataRadar_b_dataSet_2016_9L"   
#> [14] "dataCutPerMonth_dataRadar_b_dataSet_2016_9L_2"
sessionInfo()
#> R version 3.6.1 (2019-07-05)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 18.04.3 LTS
#> 
#> Matrix products: default
#> BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=nl_NL.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=nl_NL.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=nl_NL.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=nl_NL.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] drake_7.5.2.9000
#> 
#> loaded via a namespace (and not attached):
#>  [1] igraph_1.2.4.1   Rcpp_1.0.2       knitr_1.24       magrittr_1.5    
#>  [5] R6_2.4.0         rlang_0.4.0      stringr_1.4.0    highr_0.8       
#>  [9] storr_1.2.1      tools_3.6.1      xfun_0.8         cli_1.1.0       
#> [13] htmltools_0.3.6  base64url_1.4    yaml_2.2.0       digest_0.6.20   
#> [17] assertthat_0.2.1 tibble_2.1.3     crayon_1.3.4     txtq_0.1.4      
#> [21] evaluate_0.14    rmarkdown_1.14   stringi_1.4.3    pillar_1.4.2    
#> [25] compiler_3.6.1   filelock_1.0.2   backports_1.1.4  pkgconfig_2.0.2

^{Created on 2019-08-13 by the reprex package (v0.3.0)}

[wlandau]

Wow, that's serious! Thanks for finding the bug! Should be fixed in bd7588e.

[bart1]

Thanks for the quick fix, it seems to work. Ps is there an easy way to simplify target names, i notice with my code they seem to grow out of hand quite quickly (e.g. >100 characters). In the past I have written custom functions to avoid the inheritance of the name of the previews target to avoid to much duplication. Is this the best way or are there other solutions?

[wlandau]

Yes there is. Transformations have an .id argument that allows you to specify which grouping variables become part of the target names. Below, the terms "radar" and "set" are missing from the cut_* targets.

library(drake)
drake_plan(
  radar = target(
    get_radar_info(x),
    transform = map(x = c("a", "b"))
  ),
  set = target(
    get_data(year, month),
    transform = cross(year = c(2015, 2016), month = c(9, 10))
  ),
  cut = target(
    some_crop_function(set, radar),
    transform = cross(radar, set, .id = c(x, year, month))
  )
)
#> # A tibble: 14 x 2
#>    target        command                                 
#>    <chr>         <expr>                                  
#>  1 radar_a       get_radar_info("a")                     
#>  2 radar_b       get_radar_info("b")                     
#>  3 set_2015_9    get_data(2015, 9)                       
#>  4 set_2016_9    get_data(2016, 9)                       
#>  5 set_2015_10   get_data(2015, 10)                      
#>  6 set_2016_10   get_data(2016, 10)                      
#>  7 cut_a_2015_9  some_crop_function(set_2015_9, radar_a) 
#>  8 cut_b_2015_9  some_crop_function(set_2015_9, radar_b) 
#>  9 cut_a_2016_9  some_crop_function(set_2016_9, radar_a) 
#> 10 cut_b_2016_9  some_crop_function(set_2016_9, radar_b) 
#> 11 cut_a_2015_10 some_crop_function(set_2015_10, radar_a)
#> 12 cut_b_2015_10 some_crop_function(set_2015_10, radar_b)
#> 13 cut_a_2016_10 some_crop_function(set_2016_10, radar_a)
#> 14 cut_b_2016_10 some_crop_function(set_2016_10, radar_b)

^{Created on 2019-08-14 by the reprex package (v0.3.0)}

You can also set .id = FALSE to use numeric suffixes.

library(drake)
drake_plan(
  radar = target(
    get_radar_info(x),
    transform = map(x = c("a", "b"), .id = FALSE)
  ),
  set = target(
    get_data(year, month),
    transform = cross(year = c(2015, 2016), month = c(9, 10), .id = FALSE)
  ),
  cut = target(
    some_crop_function(set, radar),
    transform = cross(radar, set, .id = FALSE)
  )
)
#> # A tibble: 14 x 2
#>    target  command                           
#>    <chr>   <expr>                            
#>  1 radar   get_radar_info("a")               
#>  2 radar_2 get_radar_info("b")               
#>  3 set     get_data(2015, 9)                 
#>  4 set_2   get_data(2016, 9)                 
#>  5 set_3   get_data(2015, 10)                
#>  6 set_4   get_data(2016, 10)                
#>  7 cut     some_crop_function(set, radar)    
#>  8 cut_2   some_crop_function(set, radar_2)  
#>  9 cut_3   some_crop_function(set_2, radar)  
#> 10 cut_4   some_crop_function(set_2, radar_2)
#> 11 cut_5   some_crop_function(set_3, radar)  
#> 12 cut_6   some_crop_function(set_3, radar_2)
#> 13 cut_7   some_crop_function(set_4, radar)  
#> 14 cut_8   some_crop_function(set_4, radar_2)

^{Created on 2019-08-14 by the reprex package (v0.3.0)}

[wlandau]

Also (re #979) development drake now has help files for map(), split(), cross(), and combine() as if they were true functions.

[bart1]

Thanks that helps a lot to keep stuff manageable