binlog: support `binlog.pump.config` configurations for pump and drainer #693

weekface · 2019-07-25T06:58:11Z

What problem does this PR solve?

support binlog.pump.config and binlog.drainer.config configurations for pump and drainer
support pump and drainer configuration rollout

What is changed and how does it work?

Check List

Tests

E2E test
Stability test

Code changes

Has Helm charts change
Has Go code change
Has documents change

Side effects

Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Does this PR introduce a user-facing change?:

support `binlog.pump.config` and `binlog.drainer.config` configurations for pump and drainer

charts/tidb-cluster/templates/pump-statefulset.yaml

charts/tidb-cluster/templates/drainer-statefulset.yaml

aylei · 2019-07-25T08:39:04Z

charts/tidb-cluster/values.yaml

+      heartbeat-interval = 2
+      [storage]
+      sync-log = true
+      stop-write-at-available-space = "100 gib"


# stop write when disk available space less then the configured size # 42 MB -> 42000000, 42 mib -> 44040192 # default: 10 gib # stop-write-at-available-space = "10 gib"

I think it should be 1 Gigabytes or less

Yes, this is available space size, using default.

aylei · 2019-07-25T08:40:19Z

charts/tidb-cluster/values.yaml

+    # drainer configurations (change to the tags of your drainer version),
+    # just follow the format in the file and configure in the 'config' section
+    # as below if you want to customize any configuration.
+    config: |


This causes the old configuration key invalid, which is an in-compatible change

I think we should document a migration guide for RC and beta users

yes, this is an in-compatible change like: #638, i think we should do it before GA. @tennix

What about we keep these two ways, but make it clear in the document that we do not maintain the config for the original way and users should migrate to the new way if they want to configure any new parameters that are not exposed in values.yaml.

Have we decided to make this change incompatible?

What do you think? @tennix

Agree with @DanielZhangQD we should not make this incompatible change.

I think we can keep both for binlog configurations to keep compatible.

comments addressed.

aylei · 2019-07-25T08:46:10Z

I think rollout the binlog configuration is not a small change, can we put this to 1.1?

…ator into weekface/binlog-config

xiaojingchen · 2019-07-25T09:39:42Z

/run-e2e-tests

xiaojingchen · 2019-07-25T12:10:34Z

charts/tidb-cluster/templates/scripts/_start_drainer.sh.tpl

@@ -26,6 +26,7 @@ done

 /drainer \
 -L={{ .Values.binlog.drainer.logLevel | default "info" }} \
+-pd-urls={{ template "cluster.name" . }}-pd:2379 \


add http:// scheme

xiaojingchen · 2019-07-25T12:10:46Z

charts/tidb-cluster/templates/scripts/_start_pump.sh.tpl

@@ -1,5 +1,6 @@
 set -euo pipefail
 /pump \
+-pd-urls={{ template "cluster.name" . }}-pd:2379 \


xiaojingchen · 2019-07-25T17:03:27Z

/run-e2e-tests

xiaojingchen · 2019-07-26T02:11:01Z

/run-e2e-tests

xiaojingchen · 2019-07-29T08:16:18Z

/run-e2e-tests

…ator into weekface/binlog-config

xiaojingchen · 2019-07-29T09:34:48Z

/run-e2e-tests

xiaojingchen · 2019-07-29T09:41:12Z

/run-e2e-tests

xiaojingchen · 2019-07-29T11:12:15Z

/run-e2e-tests

xiaojingchen · 2019-08-12T07:37:54Z

/run-e2e-in-kind

weekface · 2019-08-12T08:14:30Z

/run-e2e-in-kind

xiaojingchen · 2019-08-13T05:34:36Z

/run-e2e-in-kind

…ator into binlog-config

weekface · 2019-08-13T07:33:12Z

@aylei @DanielZhangQD @tennix @xiaojingchen

This PR changed to support both configurations. PTAL again.

weekface · 2019-08-13T07:33:52Z

/run-e2e-in-kind

tennix

Rest LGTM.

charts/tidb-cluster/values.yaml

Co-Authored-By: Tennix <tennix@users.noreply.github.com>

tennix

LGTM

weekface · 2019-08-13T09:17:35Z

/run-e2e-in-kind

aylei

LGTM

sre-bot · 2019-08-14T02:21:00Z

cherry pick to release-1.1 failed

sre-bot · 2019-08-14T02:22:20Z

cherry pick to release-1.0 failed

…ner (pingcap#693) (cherry picked from commit a320e52)

…ner (#693) (#767) (cherry picked from commit a320e52)

change tidb-binlog configration

ee2e0ee

weekface added this to the v1.0.0 milestone Jul 25, 2019

weekface requested review from tennix, aylei, xiaojingchen, csuzhangxc and DanielZhangQD and removed request for csuzhangxc July 25, 2019 07:08

weekface commented Jul 25, 2019

View reviewed changes

charts/tidb-cluster/templates/pump-statefulset.yaml Show resolved Hide resolved

charts/tidb-cluster/templates/drainer-statefulset.yaml Show resolved Hide resolved

aylei reviewed Jul 25, 2019

View reviewed changes

tiny fix

bdb857c

weekface modified the milestones: v1.0.0, v1.1 Jul 25, 2019

xiaojingchen added 2 commits July 25, 2019 17:36

adjust stability test to support new binlog config way

731855a

Merge branch 'binlog-config' of https://github.com/weekface/tidb-oper…

3d141fa

…ator into weekface/binlog-config

xiaojingchen reviewed Jul 25, 2019

View reviewed changes

xiaojingchen added 2 commits July 25, 2019 20:17

add http scheme for pd url

616797c

Merge branch 'master' into binlog-config

c78c631

xiaojingchen added 2 commits July 29, 2019 17:18

fix template

71bb0a4

Merge branch 'binlog-config' of https://github.com/weekface/tidb-oper…

bc58375

…ator into weekface/binlog-config

Merge branch 'master' into binlog-config

f2fcbf8

fix e2e

4f300a4

weekface force-pushed the binlog-config branch from f5386d6 to 4f300a4 Compare August 12, 2019 06:38

Merge branch 'master' into binlog-config

fd8208c

Merge branch 'master' into binlog-config

051876c

weekface changed the title ~~change tidb-binlog configration~~ [DNM] change tidb-binlog configration Aug 13, 2019

weekface added 2 commits August 13, 2019 15:21

address comment

9531a77

Merge branch 'binlog-config' of https://github.com/weekface/tidb-oper…

1c33136

…ator into binlog-config

weekface force-pushed the binlog-config branch from f8bbb58 to 1c33136 Compare August 13, 2019 07:26

weekface changed the title ~~[DNM] change tidb-binlog configration~~ binlog: support binlog.pump.config and binlog.drainer.config configurations for pump and drainer Aug 13, 2019

weekface changed the title ~~binlog: support binlog.pump.config and binlog.drainer.config configurations for pump and drainer~~ binlog: support binlog.pump/drainer.config configurations for pump and drainer Aug 13, 2019

weekface changed the title ~~binlog: support binlog.pump/drainer.config configurations for pump and drainer~~ binlog: support binlog.pump.config configurations for pump and drainer Aug 13, 2019

tennix reviewed Aug 13, 2019

View reviewed changes

charts/tidb-cluster/values.yaml Outdated Show resolved Hide resolved

Update charts/tidb-cluster/values.yaml

419a902

Co-Authored-By: Tennix <tennix@users.noreply.github.com>

tennix approved these changes Aug 13, 2019

View reviewed changes

weekface requested a review from aylei August 13, 2019 09:24

aylei approved these changes Aug 13, 2019

View reviewed changes

weekface added the needs-cherry-pick-1.0 label Aug 14, 2019

weekface merged commit a320e52 into pingcap:master Aug 14, 2019

weekface deleted the binlog-config branch August 14, 2019 02:19

weekface added a commit to weekface/tidb-operator that referenced this pull request Aug 14, 2019

binlog: support binlog.pump.config configurations for pump and drai…

491a104

…ner (pingcap#693) (cherry picked from commit a320e52)

weekface mentioned this pull request Aug 14, 2019

cherry pick PR-693: support binlog.pump.config configurations for pump and drainer #767

Merged

weekface added a commit that referenced this pull request Aug 14, 2019

binlog: support binlog.pump.config configurations for pump and drai…

8382f84

…ner (#693) (#767) (cherry picked from commit a320e52)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

binlog: support `binlog.pump.config` configurations for pump and drainer #693

binlog: support `binlog.pump.config` configurations for pump and drainer #693

weekface commented Jul 25, 2019 •

edited

Loading

aylei Jul 25, 2019

weekface Jul 25, 2019

aylei Jul 25, 2019 •

edited

Loading

aylei Jul 25, 2019

weekface Jul 25, 2019

DanielZhangQD Jul 26, 2019

DanielZhangQD Aug 13, 2019

weekface Aug 13, 2019

weekface Aug 13, 2019

tennix Aug 13, 2019

weekface Aug 13, 2019

aylei commented Jul 25, 2019 •

edited

Loading

xiaojingchen commented Jul 25, 2019

xiaojingchen Jul 25, 2019

xiaojingchen Jul 25, 2019

xiaojingchen commented Jul 25, 2019

xiaojingchen commented Jul 26, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Aug 12, 2019

weekface commented Aug 12, 2019

xiaojingchen commented Aug 13, 2019

weekface commented Aug 13, 2019 •

edited

Loading

weekface commented Aug 13, 2019

tennix left a comment

tennix left a comment

weekface commented Aug 13, 2019

aylei left a comment

sre-bot commented Aug 14, 2019

sre-bot commented Aug 14, 2019

binlog: support binlog.pump.config configurations for pump and drainer #693

binlog: support binlog.pump.config configurations for pump and drainer #693

Conversation

weekface commented Jul 25, 2019 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei Jul 25, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aylei commented Jul 25, 2019 • edited Loading

xiaojingchen commented Jul 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaojingchen commented Jul 25, 2019

xiaojingchen commented Jul 26, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Jul 29, 2019

xiaojingchen commented Aug 12, 2019

weekface commented Aug 12, 2019

xiaojingchen commented Aug 13, 2019

weekface commented Aug 13, 2019 • edited Loading

weekface commented Aug 13, 2019

tennix left a comment

Choose a reason for hiding this comment

tennix left a comment

Choose a reason for hiding this comment

weekface commented Aug 13, 2019

aylei left a comment

Choose a reason for hiding this comment

sre-bot commented Aug 14, 2019

sre-bot commented Aug 14, 2019

binlog: support `binlog.pump.config` configurations for pump and drainer #693

binlog: support `binlog.pump.config` configurations for pump and drainer #693

weekface commented Jul 25, 2019 •

edited

Loading

aylei Jul 25, 2019 •

edited

Loading

aylei commented Jul 25, 2019 •

edited

Loading

weekface commented Aug 13, 2019 •

edited

Loading