[R-package] [ci] unit test failures don't cause MSVC CI jobs to fail

[jameslamb]

Description

The R-package CI jobs which compile the library with CMake + MSVC do not fail if the R package's unit tests fail.

• r-package (windows-2019, MSVC, R 3.6, cmake)
• r-package (windows-2022, MSVC, R 4.1, cmake)

Reproducible example

I noticed this while looking at test failures on #5437 (build link), but I suspect that this would happen for any change that causes the R unit tests to fail.

For example, I suspect (haven't confirmed) that adding something like that following anywhere in an R test case would reproduce this issue.

stop("intentionally failing")

But at the end of the tests, I see logs like the following.

Error: Test failures
Execution halted

== Failed ======================================================================
-- 1. Failure (test_basic.R:93:3): train and predict binary classification
...
-- 10. Failure (test_basic.R:3190:5): lgb.train() only prints eval metrics when 

Maximum number of 10 failures reached, some test results may be missing.

Additional Comments

I haven't investigated this much yet, just creating this issue so it isn't forgotten.

[jameslamb]

I tried to reproduce the relevant code on my Mac tonight.

LightGBM/.ci/test_r_package_windows.ps1

Lines 23 to 29 in ef006b7

	function Run-R-Code-Redirect-Stderr {
	param(
	[string]$rcode
	)
	$decorated_code = "out_file <- file(tempfile(), open = 'wt'); sink(out_file, type = 'message'); $rcode; sink()"
	Rscript --vanilla -e $decorated_code
	}

LightGBM/.ci/test_r_package_windows.ps1

Lines 283 to 287 in ef006b7

	if ($env:COMPILER -eq "MSVC") {
	Write-Output "Running tests with testthat.R"
	cd R-package/tests
	Run-R-Code-Redirect-Stderr "source('testthat.R')" ; Check-Output $?
	}

Like this:

cd R-package/tests
Rscript \
    --vanilla \
    -e "out_file <- file(tempfile(), open = 'wt'); sink(out_file, type = 'message'); source('testthat.R'); sink()"

And saw that a non-0 exit code was raised as expected

echo $?
# 1

That is evidence that this might be something specific to Windows, and not a generic problem with the pattern of using Rscript -e "source('testthat.R')" to run the tests.

Will open a draft PR to test some approaches.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.