Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source verbose from -test.v for cacheable runs. #137

Closed
wants to merge 1 commit into from

Conversation

hpidcock
Copy link

@hpidcock hpidcock commented Sep 4, 2023

It is unfortunate that results of go test -check.v are not cached in the go test cache. This is due to -check.v not being in the small set of test arguments that can be cached (see go test help below). This PR seeks to allow this by consuming the test.v flag when enabling verbosity in check test runs.

Since -test.v and the other cache-safe arguments are a part of the hash used to lookup the cache, it is safe for us to alter the behaviour of check to print the verbose output (as this is cached separately to a test result without verbosity).

go help test

The rule for a match in the cache is that the run involves the same
test binary and the flags on the command line come entirely from a
restricted set of 'cacheable' test flags, defined as -benchtime, -cpu,
-list, -parallel, -run, -short, -timeout, -failfast, and -v.
If a run of go test has any test or non-test flags outside this set,
the result is not cached.

@hpidcock
Copy link
Author

hpidcock commented Sep 4, 2023

Example:

$ go test ./... -v
=== RUN   Test
PASS: benchmark_test.go:18: BenchmarkS.TestBasicTestTiming	0.002s
PASS: benchmark_test.go:39: BenchmarkS.TestBenchmark	0.023s
PASS: benchmark_test.go:62: BenchmarkS.TestBenchmarkBytes	0.022s
PASS: benchmark_test.go:77: BenchmarkS.TestBenchmarkMem	0.022s
PASS: benchmark_test.go:15: BenchmarkS.TestCountSuite	0.000s
PASS: benchmark_test.go:29: BenchmarkS.TestStreamTestTiming	0.002s
PASS: bootstrap_test.go:26: BootstrapS.TestCountSuite	0.000s
PASS: bootstrap_test.go:29: BootstrapS.TestFailedAndFail	0.000s
PASS: bootstrap_test.go:40: BootstrapS.TestFailedAndSucceed	0.000s
PASS: bootstrap_test.go:48: BootstrapS.TestLogAndGetTestLog	0.000s
PASS: bootstrap_test.go:56: BootstrapS.TestLogfAndGetTestLog	0.000s
PASS: foundation_test.go:299: BootstrapS.TestMinLogger	0.000s
PASS: bootstrap_test.go:74: BootstrapS.TestRunDoesntShowSuccesses	0.000s
PASS: bootstrap_test.go:64: BootstrapS.TestRunShowsErrors	0.000s
PASS: checkers_test.go:39: CheckersS.TestComment	0.000s
PASS: checkers_test.go:120: CheckersS.TestDeepEquals	0.000s
PASS: checkers_test.go:88: CheckersS.TestEquals	0.000s
PASS: checkers_test.go:165: CheckersS.TestErrorMatches	0.000s
PASS: checkers_test.go:254: CheckersS.TestFitsTypeOf	0.000s
PASS: checkers_test.go:154: CheckersS.TestHasLen	0.000s
PASS: checkers_test.go:277: CheckersS.TestImplements	0.000s
PASS: checkers_test.go:47: CheckersS.TestIsNil	0.000s
PASS: checkers_test.go:179: CheckersS.TestMatches	0.000s
PASS: checkers_test.go:76: CheckersS.TestNot	0.000s
PASS: checkers_test.go:62: CheckersS.TestNotNil	0.000s
PASS: checkers_test.go:229: CheckersS.TestPanicMatches	0.000s
PASS: checkers_test.go:199: CheckersS.TestPanics	0.000s
PASS: fixture_test.go:17: FixtureS.TestCountSuite	0.000s
PASS: fixture_test.go:355: FixtureS.TestFixtureDoesntRunWithoutTests	0.000s
PASS: fixture_test.go:447: FixtureS.TestFixtureLogging	0.000s
PASS: fixture_test.go:23: FixtureS.TestOrder	0.001s
PASS: fixture_test.go:129: FixtureS.TestPanicOnSetUpSuite	0.000s
PASS: fixture_test.go:68: FixtureS.TestPanicOnSetUpTest	0.001s
PASS: fixture_test.go:152: FixtureS.TestPanicOnTearDownSuite	0.001s
PASS: fixture_test.go:98: FixtureS.TestPanicOnTearDownTest	0.001s
PASS: fixture_test.go:40: FixtureS.TestPanicOnTest	0.001s
PASS: fixture_test.go:222: FixtureS.TestPanicOnWrongSetUpSuiteArg	0.000s
PASS: fixture_test.go:279: FixtureS.TestPanicOnWrongSetUpSuiteArgCount	0.000s
PASS: fixture_test.go:206: FixtureS.TestPanicOnWrongSetUpTestArg	0.000s
PASS: fixture_test.go:263: FixtureS.TestPanicOnWrongSetUpTestArgCount	0.000s
PASS: fixture_test.go:184: FixtureS.TestPanicOnWrongTestArg	0.001s
PASS: fixture_test.go:241: FixtureS.TestPanicOnWrongTestArgCount	0.001s
PASS: fixture_test.go:409: FixtureS.TestSetUpSuiteAssert	0.002s
PASS: fixture_test.go:394: FixtureS.TestSetUpSuiteCheck	0.001s
PASS: fixture_test.go:461: FixtureS.TestSkipSuite	0.000s
PASS: fixture_test.go:472: FixtureS.TestSkipTest	0.001s
PASS: foundation_test.go:171: FoundationS.TestCallerLoggingInDifferentFile	0.001s
PASS: foundation_test.go:154: FoundationS.TestCallerLoggingInsideTest	0.001s
PASS: foundation_test.go:26: FoundationS.TestCountSuite	0.000s
PASS: foundation_test.go:48: FoundationS.TestError	0.001s
PASS: foundation_test.go:29: FoundationS.TestErrorf	0.001s
PASS: foundation_test.go:207: FoundationS.TestExpectFailureFail	0.000s
PASS: foundation_test.go:229: FoundationS.TestExpectFailureSucceed	0.001s
PASS: foundation_test.go:238: FoundationS.TestExpectFailureSucceedVerbose	0.001s
PASS: foundation_test.go:62: FoundationS.TestFailNow	0.000s
PASS: foundation_test.go:93: FoundationS.TestFailureHeader	0.000s
PASS: foundation_test.go:110: FoundationS.TestFatal	0.001s
PASS: foundation_test.go:132: FoundationS.TestFatalf	0.001s
PASS: foundation_test.go:267: FoundationS.TestSkip	0.000s
PASS: foundation_test.go:277: FoundationS.TestSkipVerbose	0.000s
PASS: foundation_test.go:78: FoundationS.TestSucceedNow	0.000s
PASS: foundation_test.go:324: EmbeddedS.TestCountSuite	0.000s
PASS: foundation_test.go:331: EmbeddedS.TestMethod	0.000s
PASS: helpers_test.go:245: HelpersS.TestAssertFailWithExpected	0.003s
PASS: helpers_test.go:258: HelpersS.TestAssertFailWithExpectedAndMessage	0.003s
PASS: helpers_test.go:272: HelpersS.TestAssertFailWithoutExpected	0.003s
PASS: helpers_test.go:284: HelpersS.TestAssertFailWithoutExpectedAndMessage	0.002s
PASS: helpers_test.go:223: HelpersS.TestAssertSucceedWithExpected	0.000s
PASS: helpers_test.go:234: HelpersS.TestAssertSucceedWithoutExpected	0.000s
PASS: helpers_test.go:311: HelpersS.TestAssertWithError	0.003s
PASS: helpers_test.go:297: HelpersS.TestAssertWithMissingExpected	0.002s
PASS: helpers_test.go:325: HelpersS.TestAssertWithNilChecker	0.001s
PASS: helpers_test.go:96: HelpersS.TestCheckFailWithExpected	0.001s
PASS: helpers_test.go:108: HelpersS.TestCheckFailWithExpectedAndComment	0.001s
PASS: helpers_test.go:121: HelpersS.TestCheckFailWithExpectedAndStaticComment	0.002s
PASS: helpers_test.go:135: HelpersS.TestCheckFailWithoutExpected	0.003s
PASS: helpers_test.go:146: HelpersS.TestCheckFailWithoutExpectedAndMessage	0.003s
PASS: helpers_test.go:76: HelpersS.TestCheckSucceedWithExpected	0.000s
PASS: helpers_test.go:86: HelpersS.TestCheckSucceedWithoutExpected	0.000s
PASS: helpers_test.go:184: HelpersS.TestCheckWithError	0.003s
PASS: helpers_test.go:158: HelpersS.TestCheckWithMissingExpected	0.003s
PASS: helpers_test.go:197: HelpersS.TestCheckWithNilChecker	0.003s
PASS: helpers_test.go:208: HelpersS.TestCheckWithParamsAndNamesMutation	0.002s
PASS: helpers_test.go:171: HelpersS.TestCheckWithTooManyExpected	0.001s
PASS: helpers_test.go:67: HelpersS.TestCheckerInterface	0.000s
PASS: helpers_test.go:435: HelpersS.TestConcurrentLogging	0.001s
PASS: helpers_test.go:19: HelpersS.TestCountSuite	0.000s
PASS: helpers_test.go:411: HelpersS.TestMkDir	0.001s
PASS: helpers_test.go:470: HelpersS.TestTestName	0.000s
PASS: helpers_test.go:340: HelpersS.TestValueLoggingWithArrays	0.001s
PASS: helpers_test.go:352: HelpersS.TestValueLoggingWithMultiLine	0.002s
PASS: helpers_test.go:369: HelpersS.TestValueLoggingWithMultiLineException	0.001s
PASS: integration_test.go:38: integrationS.TestOutput	0.001s
PASS: printer_test.go:12: PrinterS.TestCountSuite	0.000s
PASS: printer_test.go:98: PrinterS.TestIndent	0.000s
PASS: printer_test.go:79: PrinterS.TestPrintLine	0.004s
PASS: reporter_test.go:23: reporterS.TestWrite	0.000s
PASS: reporter_test.go:60: reporterS.TestWriteCallProblemWithStreamFlag	0.000s
PASS: reporter_test.go:73: reporterS.TestWriteCallProblemWithoutStreamFlag	0.000s
PASS: reporter_test.go:89: reporterS.TestWriteCallProblemWithoutStreamFlagWithLog	0.000s
PASS: reporter_test.go:35: reporterS.TestWriteCallStartedWithStreamFlag	0.000s
PASS: reporter_test.go:48: reporterS.TestWriteCallStartedWithoutStreamFlag	0.000s
PASS: reporter_test.go:107: reporterS.TestWriteCallSuccessWithStreamFlag	0.000s
PASS: reporter_test.go:120: reporterS.TestWriteCallSuccessWithStreamFlagAndReason (test skip reason)	0.000s
PASS: reporter_test.go:136: reporterS.TestWriteCallSuccessWithoutStreamFlagWithVerboseFlag	0.000s
PASS: reporter_test.go:149: reporterS.TestWriteCallSuccessWithoutStreamFlagWithoutVerboseFlag	0.000s
PASS: run_test.go:102: RunS.TestAdd	0.000s
PASS: run_test.go:18: RunS.TestCountSuite	0.000s
PASS: run_test.go:36: RunS.TestFailure	0.000s
PASS: run_test.go:258: RunS.TestFilterAllOut	0.000s
PASS: run_test.go:274: RunS.TestFilterError	0.000s
PASS: run_test.go:229: RunS.TestFilterSuiteName	0.001s
PASS: run_test.go:245: RunS.TestFilterSuiteNameAndTestName	0.000s
PASS: run_test.go:200: RunS.TestFilterTestName	0.000s
PASS: run_test.go:213: RunS.TestFilterTestNameWithAll	0.000s
PASS: run_test.go:48: RunS.TestFixture	0.000s
PASS: run_test.go:410: RunS.TestKeepWorkDir	0.000s
PASS: run_test.go:294: RunS.TestList	0.000s
PASS: run_test.go:287: RunS.TestListFiltered	0.000s
PASS: run_test.go:86: RunS.TestPanicOnSetUpSuite	0.000s
PASS: run_test.go:73: RunS.TestPanicOnSetUpTest	0.000s
PASS: run_test.go:60: RunS.TestPanicOnTest	0.001s
PASS: run_test.go:134: RunS.TestPassed	0.000s
PASS: run_test.go:183: RunS.TestPrintAll	0.000s
PASS: run_test.go:163: RunS.TestPrintExpectedFailures	0.000s
PASS: run_test.go:153: RunS.TestPrintFailure	0.000s
PASS: run_test.go:173: RunS.TestPrintFixturePanicked	0.000s
PASS: run_test.go:178: RunS.TestPrintMissed	0.000s
PASS: run_test.go:168: RunS.TestPrintPanicked	0.000s
PASS: run_test.go:191: RunS.TestPrintRunError	0.000s
PASS: run_test.go:158: RunS.TestPrintSkipped	0.000s
PASS: run_test.go:148: RunS.TestPrintSuccess	0.000s
PASS: run_test.go:266: RunS.TestRequirePartialMatch	0.001s
PASS: run_test.go:360: RunS.TestStreamMode	0.001s
PASS: run_test.go:387: RunS.TestStreamModeWithMiss	0.000s
PASS: run_test.go:24: RunS.TestSuccess	0.000s
PASS: run_test.go:305: RunS.TestVerboseMode	0.001s
PASS: run_test.go:317: RunS.TestVerboseModeWithFailBeforePass	0.001s
OK: 138 passed
--- PASS: Test (0.16s)
PASS
ok  	gopkg.in/check.v1	(cached)

@hpidcock
Copy link
Author

hpidcock commented Sep 4, 2023

Or if we land this PR #126

@hpidcock hpidcock closed this Oct 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant