migrate test-infra to testify for br/pkg

[tisonkun]

• migrate test-infra to testify for br/pkg/backup #28164
• migrate test-infra to testify for br/pkg/cdclog #28165
• migrate test-infra to testify for br/pkg/checksum #28166
• migrate test-infra to testify for br/pkg/conn #28167
• migrate test-infra to testify for br/pkg/gluetikv #28168
• migrate test-infra to testify for br/pkg/kv #28169
• migrate test-infra to testify for br/pkg/lightning #28170
• migrate test-infra to testify for br/pkg/logutil #28171
• migrate test-infra to testify for br/pkg/membuf #28172
• migrate test-infra to testify for br/pkg/metautil #28173
• migrate test-infra to testify for br/pkg/mock #28174
• migrate test-infra to testify for br/pkg/pdutil #28175
• migrate test-infra to testify for br/pkg/redact #28176
• migrate test-infra to testify for br/pkg/restore #28177
• migrate test-infra to testify for br/pkg/rtree #28178
• migrate test-infra to testify for br/pkg/storage #28179
• migrate test-infra to testify for br/pkg/summary #28180
• migrate test-infra to testify for br/pkg/task #28181
• migrate test-infra to testify for br/pkg/trace #28182
• migrate test-infra to testify for br/pkg/utils #28183
• migrate test-infra to testify for br/pkg/version #28184

[tisonkun]

br is original pingcap/br, we should break down more subtasks.

[YuJuncen]

Some analyses 🤔

General

The most simple way for adapting go check is to use the suite interface of testify: replace the call of c.Assert with corresponding testify method.

Some of "stateless" tests, which:

1. Has a suite of type struct{}
2. Defined nop lifecycle methods.

...can be refactored as plain go test cases.

Suites

The lifecycle methods of gocheck can be adapted via:

testify	gocheck
SetupSuite	SetUpSuite
SetupTest	SetUpTest
TearDownTest	TearDownTest
TearDownSuite	TearDownSuite

Checkers

The following Checker are used by BR:

BytesEquals // can be replaced with require.Equal
DeepEquals // same as above
Equals // same as above
// Reason see https://github.com/stretchr/testify/blob/acba37e5db06f0093b465a7d47822bf13644b66c/assert/assertions.go#L58-L76 

ErrorMatches // require.EqualError
FieldEquals // custom checker, using `zapcore.Field.Equals`
FitsTypeOf // maybe require.IsType
Greater // require.Greater
GreaterEqual // require.GreaterOrEqual
HasKey // ?
HasLen // require.Len
IsFalse // require.False
IsNil // require.Nil
IsTrue // require.True
Less // require.Less
LessEqual // require.LessOrEqual
Matches // require.Regexp?
Not // testify didn't support `Checker`, let alone combinators over them...
NotNil // require.NotNil
PanicMatches // maybe require.PanicsWith{Error,Object}?
RangeEquals // custom checker, checking the start key and end keys are equal.
isAbout // custom checker, checking whether two floating numbers are near enough.
jsonEquals // require.JSONEq
posEq // custom checker, checking the pos of a parser.
unblocksBetween // custom checker, checks whether a wait group done in a time interval. 
versionEquals // custom checker for semvers.

Some of the checkers has alternative methods in Assertions from testify. For custom checkers, we may make functions like posEq(t *testing.T, <arg list of checker>, interface{}... msgAndArgs), and delegate the check to methods in require.

Commentf(...)

Each assertions from testify accepts a sequence of msgAndArgs, the items in Commentf can be placed here.

the possibility of auto migration

Given a normal test file here, we can migrate it from gocheck to testify via:

1. Check the test suite in the file(unexpectedly, some files may contains more than one or no suites).
2. if it is a "stateless" suite, do:
   a. remove the test suite.
   b. remove the accepter of test functions bound to this suite.
   c. replace the argument c *C with t *testing.T.
   d. replace calling to c.Assert with corresponding method in testify.
   e. if (d) meets a checker cannot be converted to method in testify, leave a hole here and hint the user, like:

// TODO impl the custom checker
TODO_posEq(t, ...)

3. if it is a "stateful" suite, do:
   a. compose a testify.Suite in to the test suite.
   b. rename the lifecycle methods bound to the suite.
   (accessing to the c might would be fixed manually.)
   c. do the same thing for assertions as stateless test cases.

These steps (maybe?) are deterministic, which means it's possible to write a script with go/ast and go/parser to automate them. 🤔

Or we can also do these things manually, as a refinement or inspection of our test cases.

[tisonkun]

The most simple way for adapting go check is to use the suite interface of testify

@YuJuncen No. We don't do that. Please see also #26082 (comment) and this Chinese post for details. And #27747 for an example.

[tisonkun]

the possibility of auto migration

@YuJuncen @feitian124 already creates a tool you may make use of to automatically migrate #26022 (comment) . However, I'd prefer just do it manually with some "global find and replace" for common pattern.

If you try to do something automatically and correctly, you're always in trouble - you're now find a new, and probably huge, question to resolve, which is unnecessary to the original problem.

[tisonkun]

Commentf(...)

Just use require.Equalf and require.Greaterf.

[tisonkun]

Also, it is not a 1 to 1 mapping as c.Assert(err, IsNil) is almost require.NoError(t, err), but c.Assert(sth, IsNil) is require.Nil(t, sth). We don't push the migration to one man or two, as you see; we run it with join force and for br topic the current problem is to find a proper separation: possibly in packages or files.

If you can find such one, I'm glad to update the description and create subtasks.

[tisonkun]

The first thing I found is that these tests are all under br/pkg

[YuJuncen]

@tisonkun
For suites: Well, if suite isn't acceptable, maybe TestMain can help to set up a test.

需要 Setup/TearDown Suite 式的逻辑的时候，可以通过手动装配测试与子测试来达成。

func TestTeardownParallel(t *testing.T) {
    // This Run will not return until the parallel tests finish.
    t.Run("group", func(t *testing.T) {
        t.Run("Test1", parallelTest1)
        t.Run("Test2", parallelTest2)
        t.Run("Test3", parallelTest3)
    })
    // <tear-down code>
}

Sounds good, but sometimes when setting up, some variables in the suite also have been set.

Any idea better than global variables or closures? Well, yet another suite implementation.

func TestTeardownParallel(t *testing.T) {
    suite := setupSuite()
    t.Run("group", func(t *testing.T) {
        t.Run("Test1", suite.parallelTest1)
        t.Run("Test2", suite.parallelTest2)
        t.Run("Test3", suite.parallelTest3)
    })
    // <tear-down code>
}

For automating: Maybe refactor tools like eg can help?

[tisonkun]

For automating: Maybe refactor tools like eg can help?

@YuJuncen You're welcome to try on a subtask. And if it works well, I hope you can write it down to teach others how to achieve it.

[tisonkun]

Closed as all subtasks done.