Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support go time package format gc time #996

Merged
merged 11 commits into from
Aug 10, 2020

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Aug 7, 2020

What problem does this PR solve?

resolve #994

What is changed and how it works?

Support gc time for "8m", "7h". If no unit is specified, gc duration will use day as unit as before.

It should be noticed that the max time duration unit that supported in golang is hour "h".
golang/go#11473
golang/go#17767

Check List

Tests

  • Unit test
  • Integration test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to be included in the release note

Release note

  • support go time duration format for pump GC configuration

@sre-bot
Copy link

sre-bot commented Aug 7, 2020

Copy link
Contributor

@july2993 july2993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls update cmd/pump/pump.toml too

pkg/util/util.go Outdated Show resolved Hide resolved
pkg/util/util_test.go Outdated Show resolved Hide resolved
pump/config.go Show resolved Hide resolved
@CLAassistant
Copy link

CLAassistant commented Aug 7, 2020

CLA assistant check
All committers have signed the CLA.

@lichunzhu
Copy link
Contributor Author

@july2993 Already support both int and string and add some tests. PTAL again.

Copy link
Contributor

@july2993 july2993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

pump/config.go Outdated
@@ -176,6 +177,15 @@ func (cfg *Config) Parse(arguments []string) error {
util.AdjustString(&cfg.DataDir, defaultDataDir)
util.AdjustInt(&cfg.HeartbeatInterval, defaultHeartbeatInterval)

if cfg.GC.Duration == 0 && cfg.GCStr == "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this park looks confusing, when we both set int the config file and the command, the command line value should take effect instead of the one in the config file.

can we change

type Duration string
``
and using the same object for both the config file and command line flag?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this part can make it to prioritize the config in the command. If we set type Duration to string, how do we get the actual duration? Unmarshall it every time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain it more clearly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this part can make it to prioritize the config in the command

yes, I got it wrong.

If we set type Duration to string, how do we get the actual duration? Unmarshall it every time?

I think parsing it every time is ok here.

I am ok with this now, this will not block this pr.

pump/config.go Outdated
@@ -100,7 +101,7 @@ func NewConfig() *Config {
fs.StringVar(&cfg.EtcdURLs, "pd-urls", defaultEtcdURLs, "a comma separated list of the PD endpoints")
fs.StringVar(&cfg.DataDir, "data-dir", "", "the path to store binlog data")
fs.IntVar(&cfg.HeartbeatInterval, "heartbeat-interval", defaultHeartbeatInterval, "number of seconds between heartbeat ticks")
fs.IntVar(&cfg.GC, "gc", defaultGC, "recycle binlog files older than gc days")
fs.StringVar(&cfg.GCStr, "gc", "", "recycle binlog files older than gc time. default unit is day. also accept 8h format time(max unit is hour)")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set default value, so user can see the default value like before?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we set default value here, the value in toml will always be covered by default "7". Can we add it in the prompt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, that's using two objects making it complex。

Copy link
Contributor

@july2993 july2993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

july2993
july2993 previously approved these changes Aug 10, 2020
Copy link
Contributor

@july2993 july2993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

*d = Duration(s)
return nil
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var _ toml.** = &Duration{}

--
add this make sure it impl the interface and let's reader know this.

@july2993
Copy link
Contributor

/run-all-tests

1 similar comment
@IANTHEREAL
Copy link
Collaborator

/run-all-tests

@lichunzhu
Copy link
Contributor Author

/run-all-tests

Copy link
Collaborator

@IANTHEREAL IANTHEREAL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@IANTHEREAL IANTHEREAL merged commit 0bcd683 into pingcap:master Aug 10, 2020
@lichunzhu lichunzhu deleted the supportGoTimeFormatGCConfig branch August 10, 2020 06:52
@ti-srebot
Copy link

cherry pick to release-3.1 in PR #997

@ti-srebot
Copy link

cherry pick to release-3.0 failed

lichunzhu pushed a commit that referenced this pull request Aug 10, 2020
* support go time package format gc time
lichunzhu added a commit that referenced this pull request Aug 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pump supports setting the binlog file GC time according to a smaller time granularity
6 participants