Tests Not Idempotent #174

alex-gable · 2020-06-14T05:30:53Z

Tests fail if run multiple times in the same R session.

Session Info

r$> sessionInfo()                                                                                                                                                                                                 
R version 4.0.0 (2020-04-24)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.4

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] vcr_0.5.4.91   httr_1.4.1     crul_0.9.0     testthat_2.3.2

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.4.6       compiler_4.0.0     base64enc_0.1-3    prettyunits_1.1.1  remotes_2.1.1      tools_4.0.0        digest_0.6.25      pkgbuild_1.0.8     pkgload_1.1.0      jsonlite_1.6.1    
[11] memoise_1.1.0.9000 rlang_0.4.6        cli_2.0.2          rstudioapi_0.11    curl_4.3           yaml_2.2.1         xml2_1.3.2         withr_2.2.0        desc_1.2.0         fauxpas_0.5.0     
[21] fs_1.4.1           devtools_2.3.0     triebeard_0.3.0    rprojroot_1.3-2    glue_1.4.1         httpcode_0.3.0     R6_2.4.1           processx_3.4.2     fansi_0.4.1        sessioninfo_1.1.1 
[31] callr_3.4.3        webmockr_0.6.2     magrittr_1.5       whisker_0.4        backports_1.1.7    urltools_1.7.3     ps_1.3.3           ellipsis_0.3.1     usethis_1.6.1      assertthat_0.2.1  
[41] mime_0.9           lazyeval_0.2.2     crayon_1.3.4.9000

Example

r$> devtools::test()                                                                                                                                                                                              
Loading vcr
Testing vcr
✔ |  OK F W S | Context
✔ |  15       | Cassette [0.2 s]
✔ |  12       | FileSystem
✔ |  11       | HttpInteraction [0.4 s]
✔ |  13       | HTTPInteractionList [0.1 s]
✔ |   6       | Persisters
✔ |  35       | Request
✔ |   7       | RequestHandler [0.5 s]
✔ |  25     1 | RequestIgnorer [1.2 s]
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-RequestIgnorer.R:114: skip: RequestIgnorer usage: w/ vcr_configure() usage
Reason: port 8000 not available
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ |  51       | RequestMatcherRegistry
✔ |  14       | use_cassette: works as expected [0.4 s]
✔ |   8       | use_cassette fails well
✔ |   6       | ause_cassette_match_body_empty_body [0.5 s]
✔ |  32       | ause_cassette_match_requests_on [1.2 s]
✔ |  15       | ause_cassette_match_requests_on_json [0.5 s]
✔ |   3       | use_cassette options: re_record_interval [10.6 s]
✔ |  34       | use_cassette: record modes work as expected [2.4 s]
✔ |  21       | use_cassette: write to disk [0.6 s]
✔ |  30       | use_cassette: handle binary files on disk [3.0 s]
✔ |   0     1 | handling images: httr
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-binary_images.R:8: skip: use_cassette w/ with images: httr
Reason: jpeg cannot be loaded
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ |  10       | handling images: crul [0.5 s]
✔ |   6       | cassette_options
✔ |   4       | cassettes
✔ |   5       | current_cassette
✔ |   3       | cassette_path
✔ |   4       | check_cassette_names
✔ |  14       | configuration-inheritance
✔ |  12       | Configuration [0.2 s]
✔ |  27       | adapter-crul: POST requests works [0.7 s]
✔ |   4       | eject_cassette
✔ |   8       | UnhandledHTTPRequestError
✔ |  20       | UnhandledHTTPRequestError: secret handling [0.1 s]
✔ |   8       | adapter-httr: status code works [0.2 s]
✔ |  12       | adapter-httr: use_cassette works [0.1 s]
✔ |  10       | adapter-httr: use_cassette w/ preserve_exact_body_bytes [0.1 s]
✔ |  17       | adapter-httr: use_cassette w/ >1 request per cassette [0.4 s]
✔ |   3       | adapter-httr: use_cassette w/ simple auth [0.1 s]
✔ |  22       | adapter-httr: POST requests works [0.8 s]
✔ |   7       | insert_cassette fails well
✔ |  25       | insert_cassette works [0.1 s]
✔ |  39       | lightswitch functions [0.5 s]
✔ |   3       | logger: vcr_log_file fails well
✔ |  17       | logger: vcr_log_file works
✔ |   0       | no_cassette_in_use
✔ |  23       | request_summary [0.1 s]
✔ |  15       | response_summary [0.1 s]
✔ |  23       | serializers: YAML
✔ |  16       | use_vcr works
✔ |  36       | utilities
✔ |  10       | vcr_last_error
✔ |   2       | write_disk_path: package context
✔ |  13       | write_interactions
✔ |  14       | write: utils

══ Results ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 27.0 s

OK:       770
Failed:   0
Warnings: 0
Skipped:  2

r$> devtools::test()                                                                                                                                                                                              
Loading vcr
Testing vcr
✔ |  OK F W S | Context
✔ |  15       | Cassette
✔ |  12       | FileSystem
✔ |  11       | HttpInteraction [0.4 s]
✔ |  13       | HTTPInteractionList [0.1 s]
✔ |   6       | Persisters
✔ |  35       | Request
✔ |   7       | RequestHandler [0.4 s]
✔ |  25     1 | RequestIgnorer [1.2 s]
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-RequestIgnorer.R:114: skip: RequestIgnorer usage: w/ vcr_configure() usage
Reason: port 8000 not available
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ |  51       | RequestMatcherRegistry
✔ |  14       | use_cassette: works as expected [0.4 s]
✔ |   8       | use_cassette fails well
✔ |   6       | ause_cassette_match_body_empty_body [0.5 s]
✔ |  32       | ause_cassette_match_requests_on [1.0 s]
✔ |  15       | ause_cassette_match_requests_on_json [0.6 s]
✔ |   3       | use_cassette options: re_record_interval [10.5 s]
✔ |  34       | use_cassette: record modes work as expected [2.4 s]
✔ |  21       | use_cassette: write to disk [0.6 s]
✔ |  30       | use_cassette: handle binary files on disk [3.3 s]
✔ |   0     1 | handling images: httr
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-binary_images.R:8: skip: use_cassette w/ with images: httr
Reason: jpeg cannot be loaded
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ |  10       | handling images: crul [0.5 s]
✔ |   6       | cassette_options
✖ |   2 2     | cassettes
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-cassettes.R:8: failure: cassettes works
length(aa) not equal to 0.
1/1 mismatches
[1] 3 - 0 == 3

test-cassettes.R:14: failure: cassettes works
Names of `bb` ('abcd', 'crul_post_upload_file_no_list', 'foobar24', 'httr_test1') don't match 'foobar24'
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ |   5       | current_cassette
✔ |   3       | cassette_path
✔ |   4       | check_cassette_names
✔ |  14       | configuration-inheritance
✔ |  12       | Configuration
✔ |  27       | adapter-crul: POST requests works [0.8 s]
✔ |   4       | eject_cassette
✔ |   8       | UnhandledHTTPRequestError
✔ |  20       | UnhandledHTTPRequestError: secret handling [0.1 s]
✔ |   8       | adapter-httr: status code works [0.2 s]
✔ |  12       | adapter-httr: use_cassette works [0.1 s]
✔ |  10       | adapter-httr: use_cassette w/ preserve_exact_body_bytes [0.1 s]
✔ |  17       | adapter-httr: use_cassette w/ >1 request per cassette [0.4 s]
✔ |   3       | adapter-httr: use_cassette w/ simple auth [0.1 s]
✔ |  22       | adapter-httr: POST requests works [0.8 s]
✔ |   7       | insert_cassette fails well
✔ |  25       | insert_cassette works [0.1 s]
✔ |  39       | lightswitch functions [0.5 s]
✔ |   3       | logger: vcr_log_file fails well
✔ |  17       | logger: vcr_log_file works
✔ |   0       | no_cassette_in_use
✔ |  23       | request_summary [0.1 s]
✔ |  15       | response_summary [0.1 s]
✔ |  23       | serializers: YAML
✖ |  15 1 1   | use_vcr works
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
test-use_vcr.R:36: failure: use_vcr fails well
`use_vcr(dir)` threw an error with unexpected message.
Expected match: "'dir' does not exist"
Actual message: "'DESCRIPTION' not found; are you sure it's an R package?"
Backtrace:
 1. testthat::expect_error(use_vcr(dir), "'dir' does not exist") tests/testthat/test-use_vcr.R:36:2
 6. vcr::use_vcr(dir)
 7. vcr::pkg_name(dir) R/vcr_setup.R:76:2

test-use_vcr.R:40: warning: use_vcr fails well
'/var/folders/74/yl_jm18j02z_s2_m11v203k40000gn/T//RtmpckmCGf/foobar2' already exists
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ |  36       | utilities
✔ |  10       | vcr_last_error
✔ |   2       | write_disk_path: package context
✔ |  13       | write_interactions [0.1 s]
✔ |  14       | write: utils

══ Results ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 26.8 s

OK:       767
Failed:   3
Warnings: 1
Skipped:  2

sckott · 2020-06-16T00:59:40Z

I think that fixes it, let me know if it doesn't

by the way - CI is on gh actions now - see https://github.com/ropensci/vcr/actions?query=workflow%3AR-check

sckott added this to the v0.6 milestone Jun 16, 2020

sckott added a commit that referenced this issue Jun 16, 2020

#174 bump version, fixes for tests lacking idempotency

22392bc

sckott closed this as completed Sep 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests Not Idempotent #174

Tests Not Idempotent #174

alex-gable commented Jun 14, 2020

sckott commented Jun 16, 2020

Tests Not Idempotent #174

Tests Not Idempotent #174

Comments

alex-gable commented Jun 14, 2020

sckott commented Jun 16, 2020