*: make unit test great #7952

HuSharp · 2024-03-20T06:31:59Z

What problem does this PR solve?

Issue Number: Ref #7969

What is changed and how does it work?

Add back parallel by run a single test case in a isolated OS process

We use a wrong way to parallel. Some test cases are not side effect free, for example:

the enable and disable of the failpoint
modification of the configuration
change of the global variable in a single process.
etc...

If the test cases with side effect run parallelly with others, it could cause some test fail unexpectedly.
When they're in a single OS process, make them parallel is not a big deal.

usage

// run all tests
pd-ut

// show usage
pd-ut -h

// list all packages
pd-ut list

// list test cases of a single package
pd-ut list $package

// list test cases that match a pattern
pd-ut list $package 'r:$regex'

// run all tests
pd-ut run

// run test all cases of a single package
pd-ut run $package

// run test cases of a single package
pd-ut run $package $test

// run test cases that match a pattern
pd-ut run $package 'r:$regex'

// build all test package
pd-ut build

// build a test package
pd-ut build xxx

// write the junitfile
pd-ut run --junitfile xxx

// test with race flag
pd-ut run --race

use xprog to accelerate bin.test build

go test --exec=xprog, the Go compile toolchain would call your xprog program passing the test binary path as the argument

So you can do this trick, write you own xprog, and move the test binary to some place else, save it for later use.

print function time

when using ./pd-ut run --junitfile xxx, can get junitfile as XML file to get every function cost time.

	<testsuite tests="11" failures="0" time="1.789263" name="github.com/tikv/pd/pkg/btree">
		<properties>
			<property name="go.version" value="go1.21.0 linux/amd64"></property>
		</properties>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestAscendLessThanG" time="0.038767"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestAscendGreaterOrEqualG" time="0.033858"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDeleteMaxG" time="0.201940"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestBTreeSizeInfo" time="0.136755"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDescendGreaterThanG" time="0.013433"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDeleteMinG" time="0.262618"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestCloneConcurrentOperationsG" time="0.335037"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestBTreeG" time="0.348103"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDescendLessOrEqualG" time="0.018799"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestDescendRangeG" time="0.302686"></testcase>
		<testcase classname="github.com/tikv/pd/pkg/btree" name="TestAscendRangeG" time="0.097267"></testcase>
	</testsuite>

Performance

Since we run parallel now, the CI runs faster than before, on my server computer (Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GH, 144 core 72 CPU):

reduce from 6min -> 1min

for make ut:

time make ut
building task finish, parallelism=144, count=641, takes=10.154869881s
run all tasks takes 50.559951196s
> make ut  327.15s user 48.61s system 615% cpu 1:01.03 total

for make basic-test

time make basic-test
CGO_ENABLED=1 go test -tags tso_function_test,deadlock -timeout 20m -race
> make basic-test 1176.46s user 169.41s system 473% cpu 5:44.15 total

Check List

Tests

Unit test
Manual test

Release note

introduce pd-ut to accelerate unit-test

ti-chi-bot · 2024-03-20T06:32:01Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

nolouch
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2024-03-20T06:32:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

pkg/mcs/scheduling/server/rule/watcher_test.go

pkg/slice/slice_test.go

pkg/utils/etcdutil/etcdutil_test.go

pkg/utils/typeutil/size_test.go

lhy1024 · 2024-03-20T06:39:24Z

server/api/member_test.go

-}
-
-func (suite *resignTestSuite) TestResignMyself() {
+func (suite *memberTestSuite) TestResignMyself() {


Do we still need to call suite.Clean()?

Signed-off-by: husharp <jinhao.hu@pingcap.com>

codecov · 2024-03-21T03:50:48Z

Codecov Report

Merging #7952 (c235324) into master (b9240a0) will decrease coverage by 3.64%.
The diff coverage is n/a.

❗ Current head c235324 differs from pull request most recent head 9f93b0e. Consider uploading reports for the commit 9f93b0e to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7952      +/-   ##
==========================================
- Coverage   77.30%   73.66%   -3.64%     
==========================================
  Files         468      436      -32     
  Lines       60879    48525   -12354     
==========================================
- Hits        47061    35748   -11313     
+ Misses      10284     9719     -565     
+ Partials     3534     3058     -476

Flag	Coverage Δ
unittests	`73.66% <ø> (-3.64%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

nolouch · 2024-03-21T05:31:00Z

tools/pd-ut/ut.go

+
+func usage() bool {
+	msg := `// run all tests
+pd-ut


I'd prefer to name this as dev, as we can do more scope with dev command , such as

./dev lint ./dev test pkg/util ./dev generator swagger ./dev bench heartbeat

maybe can unify all tools in another pr

Signed-off-by: husharp <jinhao.hu@pingcap.com>

rleungx · 2024-03-28T09:43:41Z

Makefile

@@ -225,6 +228,12 @@ failpoint-disable: install-tools

 #### Test ####

+ut: pd-ut


Do we have a timeout?

maybe need to reduce timeout for each test when enabled race
https://github.com/tikv/pd/pull/7952/files#diff-d1de7e6511458d33a769c0c501ec0ea45494dd9d2e830dc315e8fe623eb34b76R569-R574

Signed-off-by: husharp <jinhao.hu@pingcap.com>

HuSharp · 2024-04-10T10:14:38Z

/merge

ti-chi-bot · 2024-04-10T10:14:41Z

@HuSharp: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-04-10T10:14:42Z

This pull request has been accepted and is ready to merge.

Commit hash: c235324

ti-chi-bot · 2024-04-10T10:14:55Z

@HuSharp: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot bot requested review from disksing and rleungx March 20, 2024 06:32

HuSharp removed request for disksing and rleungx March 20, 2024 06:33

HuSharp force-pushed the support_ut branch from 26f6f43 to 091bc46 Compare March 20, 2024 06:36

lhy1024 reviewed Mar 20, 2024

View reviewed changes

HuSharp force-pushed the support_ut branch 2 times, most recently from 3d8d96f to 2508337 Compare March 20, 2024 06:57

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Mar 20, 2024

HuSharp force-pushed the support_ut branch 2 times, most recently from bffc075 to a982d7b Compare March 20, 2024 08:42

support ut ctl

532b15b

Signed-off-by: husharp <jinhao.hu@pingcap.com>

HuSharp force-pushed the support_ut branch 2 times, most recently from 532b15b to 5437e68 Compare March 21, 2024 03:39

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Mar 21, 2024

nolouch reviewed Mar 21, 2024

View reviewed changes

HuSharp marked this pull request as ready for review March 21, 2024 05:42

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 21, 2024

HuSharp changed the title ~~[DNM] make unit test great~~ tests: make unit test great Mar 21, 2024

HuSharp changed the title ~~tests: make unit test great~~ */unit-test: make unit test great Mar 21, 2024

HuSharp changed the title */unit-test: make unit test great *: make unit test great Mar 21, 2024

HuSharp mentioned this pull request Mar 22, 2024

Make Unit Test Great! #7969

Closed

HuSharp force-pushed the support_ut branch from 9ec5242 to 4708984 Compare March 25, 2024 02:41

add pd-ut in tools

e75e186

Signed-off-by: husharp <jinhao.hu@pingcap.com>

HuSharp force-pushed the support_ut branch from 4708984 to e75e186 Compare March 25, 2024 02:47

fix conflict

46385af

Signed-off-by: husharp <jinhao.hu@pingcap.com>

rleungx reviewed Mar 28, 2024

View reviewed changes

update timeout

6ebb0ab

Signed-off-by: husharp <jinhao.hu@pingcap.com>

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 3, 2024

merge master

c235324

Signed-off-by: husharp <jinhao.hu@pingcap.com>

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 7, 2024

nolouch approved these changes Apr 9, 2024

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 9, 2024

rleungx approved these changes Apr 10, 2024

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 10, 2024

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 10, 2024

Merge branch 'master' into support_ut

9f93b0e

ti-chi-bot bot merged commit f0eb74b into tikv:master Apr 10, 2024
22 checks passed

HuSharp deleted the support_ut branch April 10, 2024 10:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: make unit test great #7952

*: make unit test great #7952

HuSharp commented Mar 20, 2024 •

edited

Loading

ti-chi-bot bot commented Mar 20, 2024 •

edited

Loading

ti-chi-bot bot commented Mar 20, 2024

lhy1024 Mar 20, 2024

codecov bot commented Mar 21, 2024 •

edited

Loading

nolouch Mar 21, 2024

HuSharp Mar 25, 2024

rleungx Mar 28, 2024

HuSharp Mar 29, 2024

HuSharp commented Apr 10, 2024

ti-chi-bot bot commented Apr 10, 2024

ti-chi-bot bot commented Apr 10, 2024

ti-chi-bot bot commented Apr 10, 2024

		@@ -225,6 +228,12 @@ failpoint-disable: install-tools

		#### Test ####

		ut: pd-ut

*: make unit test great #7952

*: make unit test great #7952

Conversation

HuSharp commented Mar 20, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Add back parallel by run a single test case in a isolated OS process

use xprog to accelerate bin.test build

print function time

Performance

Check List

Release note

ti-chi-bot bot commented Mar 20, 2024 • edited Loading

ti-chi-bot bot commented Mar 20, 2024

lhy1024 Mar 20, 2024

Choose a reason for hiding this comment

codecov bot commented Mar 21, 2024 • edited Loading

Codecov Report

nolouch Mar 21, 2024

Choose a reason for hiding this comment

HuSharp Mar 25, 2024

Choose a reason for hiding this comment

rleungx Mar 28, 2024

Choose a reason for hiding this comment

HuSharp Mar 29, 2024

Choose a reason for hiding this comment

HuSharp commented Apr 10, 2024

ti-chi-bot bot commented Apr 10, 2024

ti-chi-bot bot commented Apr 10, 2024

ti-chi-bot bot commented Apr 10, 2024

HuSharp commented Mar 20, 2024 •

edited

Loading

ti-chi-bot bot commented Mar 20, 2024 •

edited

Loading

codecov bot commented Mar 21, 2024 •

edited

Loading