How to write benchmarks using testify / suite ?

[msonawane]

No description provided.

[mohanraj-r]

You should be able to use the *testing.B directly with testify.

func BenchmarkFoo(b *testing.B) {
   require.NoError(b, fmt.Errorf("test error"))
}

Testify uses TestingT interface which is satisfied by both testing.B and testing.T

[mrsndmn]

But there is no way to run benchmarks with testify/suite.
suite.Run accepts only t *testing.T

[boyan-soubachov]

Suite now has an extra HandleStats() function. If you have any ideas & code to make it compatible with testing.B, I'd be very happy to help you get it merged :)

[updogliu]

Do you think we can change
func Run(t *testing.T, suite TestingSuite) to
func Run(t testing.TB, suite TestingSuite) ?

More essentially, change TestingSuite to

type TestingSuite interface {
    TB() testing.TB
    SetTB(testing.TB)
}

[boyan-soubachov]

We've had that change before and it breaks backwards compatibility. We'd be happy to consider it but it would be implemented in testify v2

[milosgajdos]

Doesn #495 cover this use case?

[joewreschnig]

@boyan-soubachov Suites don't seem to expose much about their testing.T - is there a point of concern other than func (suite *Suite) T() *testing.T?

If that's the only place it's exposed, I think it would be possible to do it compatibly by having it return a testing.T if it was given one, and panic nil if it was given a non-testing.T (at Run or via SetT). Since all existing code must be passing a testing.T, I think it wouldn't affect any existing tests.

I think direct use of T() (and even SetT) are pretty rare.

[carlcarl]

Here is an workaround for anyone who has the same problem:

type Suite struct {
    suite.Suite
}

......

func BenchmarkExample(b *testing.B) {
    s := new(Suite)
    s.SetT(&testing.T{})
    s.SetupSuite()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        s.SetupTest()
        b.StartTimer()
        // Put the function you want to benchmark here.
        b.StopTimer()
        s.TearDownTest()
    }
    s.TearDownSuite()
}

[rmscoal]

Why does mine shows a s.SetupSuite undefined? Any suggestion? Or by any chance, you defined it yourselves?

[rmscoal]

Ah nvm, it's an interface contract.. Okay thanks

[pplmx]

Any progress?

[msf]

I'm doing benchmarks w/ Suite and I suggest a small modification to @carlcarl suggestion:

func BenchmarkExample(b *testing.B) {
    s := new(Suite)
    s.SetT(&testing.T{})
    s.SetupSuite()
    s.SetupTest()
    // do any additional prep-work for the benchmark here
    b.StartTimer()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        // Put the function you want to benchmark here.
    }
    b.StopTimer()
    s.TearDownTest()
    s.TearDownSuite()
}

This serves two purposes:

• clearly isolate prepwork from the benchmark
• more conventionally use the benchmark traditional for loop for i := 0; i < b.N; i++ {, tracking its behavior as i progresses to N (can make a diff for mem allocations..etc..)