*: fix memory leak introduced by timer.After #6720

lhy1024 · 2023-06-29T11:10:28Z

What problem does this PR solve?

Issue Number: Close #6719

What is changed and how does it work?

use sync.Pool to maintain TSDeadline
replace time.After with NewTimer and timer.Stop, or ticker

for loop retry use ticker, if the loop is broken, we will stop it by deferring or checking each branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

only bench with forward c.serviceModeUpdateCb(pdpb.ServiceMode_PD_SVC_MODE)

~/pingcap/tiup/bin/tiup-playground v6.6.0 --pd.micro '[{"api":true, "amount":3},{"tso":true,"amount":3}]' --pd.binpath ~/pd-cse/bin/pd-server --db.binpath ~/tidb-cse/bin/tidb-server --kv 3 --db 1 --tiflash 0 --db.port 14000 --host 192.168.8.88

./pd-tso-bench -v -duration 250000s -client 10 -c 1 -interval 3s -pd 192.168.8.88:2379

Before this pr

After this pr

Release note

None.

ti-chi-bot · 2023-06-29T11:10:30Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

binshi-bing
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Signed-off-by: lhy1024 <admin@liudos.us>

codecov · 2023-06-29T11:31:51Z

Codecov Report

Patch coverage: 75.65% and project coverage change: +0.30 🎉

Comparison is base (05f71e0) 74.27% compared to head (a40b03a) 74.58%.

❗ Current head a40b03a differs from pull request most recent head ac7bb65. Consider uploading reports for the commit ac7bb65 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6720      +/-   ##
==========================================
+ Coverage   74.27%   74.58%   +0.30%     
==========================================
  Files         410      418       +8     
  Lines       42871    43706     +835     
==========================================
+ Hits        31844    32599     +755     
- Misses       8169     8215      +46     
- Partials     2858     2892      +34

Flag	Coverage Δ
unittests	`74.58% <75.65%> (+0.30%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/keyspace/keyspace.go	`62.32% <0.00%> (+0.11%)`	⬆️
pkg/mcs/discovery/register.go	`61.53% <33.33%> (-13.98%)`	⬇️
server/grpc_service.go	`52.87% <33.33%> (+0.52%)`	⬆️
pkg/election/lease.go	`85.88% <50.00%> (-3.87%)`	⬇️
client/timerpool/pool.go	`60.00% <60.00%> (ø)`
client/client.go	`62.66% <66.66%> (-0.30%)`	⬇️
client/resource_manager_client.go	`70.69% <66.66%> (+1.68%)`	⬆️
client/tso_stream.go	`74.46% <66.66%> (-1.62%)`	⬇️
pkg/mcs/resourcemanager/server/server.go	`53.38% <66.66%> (+0.39%)`	⬆️
pkg/mcs/utils/util.go	`61.53% <66.66%> (+6.99%)`	⬆️
... and 9 more

... and 38 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Signed-off-by: lhy1024 <admin@liudos.us>

binshi-bing

left one comment. The reset LGTM

binshi-bing · 2023-06-29T12:00:45Z

pkg/utils/tsoutil/tso_dispatcher.go

@@ -208,8 +214,10 @@ func NewTSDeadline(
 	done chan struct{},
 	cancel context.CancelFunc,
 ) *TSDeadline {
+	timer := timerPool.Get().(*time.Timer)
+	timer.Reset(timeout)
 	return &TSDeadline{


How about we use sync.Pool for TSDeadline and a timer is always attached to a TSDeadline and can be reused and reset every time? In this way, we use sync.Pool for both TSDeadline and Timer in fact.

It will make code complex and test show that done and cancel don't cost some memory.

You're right. Timer is the main issue and we can keep smaller change.

binshi-bing

LGTM

binshi-bing · 2023-06-29T12:11:40Z

pkg/utils/tsoutil/tso_dispatcher.go

@@ -208,8 +214,10 @@ func NewTSDeadline(
 	done chan struct{},
 	cancel context.CancelFunc,
 ) *TSDeadline {
+	timer := timerPool.Get().(*time.Timer)
+	timer.Reset(timeout)
 	return &TSDeadline{


You're right. Timer is the main issue and we can keep smaller change.

client/tso_dispatcher.go

server/grpc_service.go

Signed-off-by: lhy1024 <admin@liudos.us>

2. replace timer with ticker in for-retry 3. add reset prefunc in lease.go Signed-off-by: lhy1024 <admin@liudos.us>

client/tso_dispatcher.go

Signed-off-by: lhy1024 <admin@liudos.us>

binshi-bing

LGTM

client/tso_dispatcher.go

reset ticker to support dynamic config Signed-off-by: lhy1024 <admin@liudos.us>

Signed-off-by: lhy1024 <admin@liudos.us>

close #6719, ref #6720 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

Signed-off-by: lhy1024 <admin@liudos.us>

JmPotato · 2023-07-03T08:10:07Z

/merge

ti-chi-bot · 2023-07-03T08:10:08Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-07-03T08:10:10Z

This pull request has been accepted and is ready to merge.

Commit hash: a40b03a

ti-chi-bot · 2023-07-03T08:10:24Z

@lhy1024: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close tikv#6719, ref tikv#6720 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close tikv#6719 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close tikv#6719, ref tikv#6720 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

close tikv#6719 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* mcs: use patch method in keyspace group (tikv#6713) ref tikv#6233 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * *: fix memory leak introduced by timer.After (tikv#6720) close tikv#6719 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * tso: implement groupSplitPatroller to speed up the split process (tikv#6736) ref tikv#5895, close tikv#6696 Implement `groupSplitPatroller` to speed up the split process. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * client: fix tso service discovery at the first time for NewClientWithAPIContext (tikv#6749) close tikv#6748 After NewClientWithAPIContextV2 returns, the keyspace group should be discovered by the passed keyspace name immediately Signed-off-by: Bin Shi <binshi.bing@gmail.com> * *: add test for misusing keyspace ID when creating the client (tikv#6754) ref tikv#6747, ref tikv#6748, ref tikv#6749 Signed-off-by: Ryan Leung <rleungx@gmail.com> * tso: support multi-keyspace, fault injection and keyspace-name in pd-tso-bench (tikv#6608) ref tikv#5895 support multi-keyspace, fault injection and keyspace-name in pd-tso-bench Signed-off-by: Bin Shi <binshi.bing@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * *: make test great again (tikv#6767) close tikv#6761 Signed-off-by: Ryan Leung <rleungx@gmail.com> * tso: implement deletedGroupCleaner to clean up the legacy TSO key (tikv#6745) close tikv#6589 - Implement `deletedGroupCleaner` to clean up the legacy TSO key. - Extract the timestamp key path constructor. Signed-off-by: JmPotato <ghzpotato@gmail.com> * Add TestUpgradingAPIandTSOClusters (tikv#6534) ref tikv#5895 Add TestUpgradingAPIandTSOClusters to test the scenario that after we restart the API cluster then restart the TSO cluster, the TSO service can still serve TSO requests normally. Signed-off-by: Bin Shi <binshi.bing@gmail.com> * client, tests: allow TSO fallback happens in TestMixedTSODeployment (tikv#6740) close tikv#6634 Introduce `WithAllowTSOFallback` client option to bypass the panic in `TestMixedTSODeployment`. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 <admin@liudos.us> * tso: allow mergedTS to be zero in mergingChecker (tikv#6758) ref tikv#6589 Since it's possible that a keyspace group is to be deleted and merged before its TSO is initialized, we should allow `mergedTS` to be zero in `mergingChecker`. This PR allows this case and only block the merging when loading the TSO meets the error. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * *: move keyspace group primary path code to key_path.go (tikv#6755) ref tikv#5895 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: lhy1024 <admin@liudos.us> * mcs: add log flags (tikv#6777) ref tikv#5766 Signed-off-by: Ryan Leung <rleungx@gmail.com> Signed-off-by: lhy1024 <admin@liudos.us> * keyspace, tso, apiv2: impl the interface to merge all keyspace groups into the default (tikv#6757) ref tikv#6756 Impl the interface to merge all keyspace groups into the default. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * pdctl: support show keyspace group primary (tikv#6747) close tikv#6746 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * pd-ctl, tests: impl the merge all keyspace groups command (tikv#6782) close tikv#6756 - Impl the merge all keyspace groups command. - Further reuse of TSO cluster-related test code. Signed-off-by: JmPotato <ghzpotato@gmail.com> Signed-off-by: lhy1024 <admin@liudos.us> * pd-ctl, tests: impl the merge all keyspace groups command (tikv#6782) close tikv#6756 - Impl the merge all keyspace groups command. - Further reuse of TSO cluster-related test code. Signed-off-by: JmPotato <ghzpotato@gmail.com> Signed-off-by: lhy1024 <admin@liudos.us> * *: fix test suite race (tikv#6784) close tikv#6772 Signed-off-by: Ryan Leung <rleungx@gmail.com> * keyspace: fix data race (tikv#6797) Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * pdctl: support show keyspace meta with refresh group id (tikv#6751) close tikv#6746 Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * mcs: Refactor ServicePath to make caller's life easier (tikv#6799) close tikv#6800 Signed-off-by: Xiaoguang Sun <sunxiaoguang@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> * Remove the lastPhysical check in dispatchClient (tikv#6812) * *: fix `TestGetTSOImmediately` (tikv#6811) close tikv#6795 Signed-off-by: Ryan Leung <rleungx@gmail.com> * fix test Signed-off-by: lhy1024 <admin@liudos.us> --------- Signed-off-by: Bin Shi <binshi.bing@gmail.com> Signed-off-by: Ryan Leung <rleungx@gmail.com> Signed-off-by: JmPotato <ghzpotato@gmail.com> Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Bin Shi <39923490+binshi-bing@users.noreply.github.com> Co-authored-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

ti-chi-bot bot added the release-note-none Denotes a PR that doesn't merit a release note. label Jun 29, 2023

ti-chi-bot bot requested review from disksing and JmPotato June 29, 2023 11:10

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 29, 2023

*: fix memory leak introduced by timer.After

1ae3f52

Signed-off-by: lhy1024 <admin@liudos.us>

lhy1024 force-pushed the timer branch from 1e2828f to 1ae3f52 Compare June 29, 2023 11:33

lhy1024 requested review from rleungx and binshi-bing June 29, 2023 11:39

add timerPool.Put

b5c5c68

Signed-off-by: lhy1024 <admin@liudos.us>

binshi-bing reviewed Jun 29, 2023

View reviewed changes

binshi-bing approved these changes Jun 29, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 29, 2023