Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syncer: support TLS for region syncer #1728

Merged
merged 4 commits into from
Sep 5, 2019
Merged

syncer: support TLS for region syncer #1728

merged 4 commits into from
Sep 5, 2019

Conversation

rleungx
Copy link
Member

@rleungx rleungx commented Sep 4, 2019 •
edited
Loading

What problem does this PR solve?

Currently, the region syncer establish the connection by using WithInsecure() which doesn't use TLS. Closes #1682.

What is changed and how it works?

This PR adds TLS support for the region syncer.

Check List

Tests

  • Manual test

Related changes

  • Need to cherry-pick to the release branch
  • Need to be included in the release notes

@rleungx
support tls for region syncer
7cd454b 
Signed-off-by: Ryan Leung <rleungx@gmail.com>
@rleungx rleungx added the needs-cherry-pick-release-3.0 The PR needs to cherry pick to release-3.0 branch. label Sep 4, 2019
@codecov-io
Copy link

codecov-io commented Sep 4, 2019 •
edited
Loading

Codecov Report

Merging #1728 into master will increase coverage by 0.16%.
The diff coverage is 36.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1728      +/-   ##
==========================================
+ Coverage   76.95%   77.12%   +0.16%     
==========================================
  Files         160      161       +1     
  Lines       15731    15734       +3     
==========================================
+ Hits        12106    12135      +29     
+ Misses       2608     2593      -15     
+ Partials     1017     1006      -11
Impacted Files Coverage Δ
client/client.go 69.76% <100%> (+3.98%) ⬆️
server/region_syncer/server.go 82.05% <100%> (+0.15%) ⬆️
pkg/grpcutil/grpcutil.go 18.51% <18.51%> (ø)
server/region_syncer/client.go 82.89% <50%> (+6.94%) ⬆️
server/tso/tso.go 77.06% <0%> (-2.76%) ⬇️
server/grpc_service.go 57.91% <0%> (-0.44%) ⬇️
server/cluster.go 83.35% <0%> (+0.25%) ⬆️
server/member/leader.go 80.61% <0%> (+0.51%) ⬆️
server/server.go 82.79% <0%> (+0.57%) ⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca5cae6...ef5fce3. Read the comment docs.

@rleungx rleungx requested a review from Connor1996 September 5, 2019 05:25
shafreeck
shafreeck approved these changes Sep 5, 2019
View reviewed changes
Copy link
Contributor

@shafreeck shafreeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

rleungx and others added 2 commits September 5, 2019 18:05
@rleungx
address the comment
ea506b6 
Signed-off-by: Ryan Leung <rleungx@gmail.com>
@nolouch
Merge branch 'master' into tls-region-syncer
3055937
@nolouch nolouch added the status/can-merge Indicates a PR has been approved by a committer. label Sep 5, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Sep 5, 2019

Your auto merge job has been accepted, waiting for #1729

@sre-bot
Copy link
Contributor

sre-bot commented Sep 5, 2019

/run-all-tests

@sre-bot sre-bot merged commit f2487bf into tikv:master Sep 5, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Sep 5, 2019

cherry pick to release-3.0 failed

rleungx added a commit to rleungx/pd that referenced this pull request Sep 9, 2019
@rleungx
syncer: support TLS for region syncer (tikv#1728)
9d773d0 
Signed-off-by: Ryan Leung <rleungx@gmail.com>
@rleungx rleungx mentioned this pull request Sep 9, 2019
Merged
Luffbee added a commit that referenced this pull request Sep 9, 2019
@Luffbee
[new-hotspot-scheduler] add operator influence and more metrics (#1713)
83ab4c2 
* *: unify get store function everywhere (#1671)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* remove unnecessary parentheses

*  server: use leader lease to determine tso service validity (#1676)

Signed-off-by: disksing <i@disksing.com>

* change internal stat values to float64

* add pending operator influence

* add metrics of pending influence

* fix metrics

* fix panic

* adjust pending influence of balanceHotWrite

* change weight of pending influence

* test: fix tests (#1696)

* test: fix region syncer test

Signed-off-by: disksing <i@disksing.com>

* decrease region rolling window; store pending influence in scheduler

* add config-check flag for pd-server (#1695)

Signed-off-by: cwen0 <cwenyin0@gmail.com>

* decrease possiblility transfer hot write leader

* change pending influence weight

* add unstarted op metrics

* add logs for debug

* add log for debug

* add logs for debug

* add logs for debug

* add logs for debug

* add logs for debug

* add logs for debug

* add logs for debug

* Revert "add logs for debug"

This reverts commit e74c7a9.

* add metrics for hotspot operators

* operator: rewrite move region related functions (#1667)

* add metrics for pending operators

* *: support setting endKey for ScanRange (#1700)

Signed-off-by: disksing <i@disksing.com>

* fix bug

* fix bug

* fix bug

* fix metrics thread-safe bug

* fix logic bug

* *: reduce some unnecessary parameters (#1698)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* schedule: Do not send an operator of a region wth a stale epoch (#1659)

* schedule: Do not send an operator of a region wth a stale epoch

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* schedule: check the version changed by the operator self

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* schedule: fix unit test

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* schedule: fix to avoid dispatching a stale opstep

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool"

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* dispatch: fix typo in comment

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* fix typo

Co-Authored-By: Ryan Leung <rleungx@gmail.com>

* dispatch: fix unittest

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* dispatch: refine format

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* server: fix the dead lock in scatter region (#1706)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* add drop time for operator

* use IsDropped to recognize canceled ops

* try to fix trans leader burst

* try to fix trans leader burst

* add zombie influence

* change select src dst strategy; improve op_controller

* change select src strategy

* fix bug

* tools: fix set namespace in pd-ctl (#1701)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* tools: fix parse url without http prefix (#1703)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* tests: support deadlock detection in make test (#1704)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* Makefile: fix failpoint enable (#1722)

Signed-off-by: nolouch <nolouch@gmail.com>

* checker: fix the issue that a region does not merge to the sibling with smaller size (#1723)

Signed-off-by: disksing <i@disksing.com>

* tools: balance region simulator (#1708)

* scheduler: do not remove the operator when the step does not finish (#1715)

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* operator: fix the AddLearner config version judgment (#1732)

Signed-off-by: nolouch <nolouch@gmail.com>

* tools: fix TLS in pd control (#1729)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* syncer: support TLS for region syncer (#1728)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* schedule: fix a thread-safe bug and improve code (#1719)
Luffbee added a commit that referenced this pull request Sep 11, 2019
@Luffbee
[new-hotspot-scheduler] merge master (#1752)
bcfcada 
* *: unify get store function everywhere (#1671)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

*  server: use leader lease to determine tso service validity (#1676)

Signed-off-by: disksing <i@disksing.com>

* test: fix tests (#1696)

* test: fix region syncer test

Signed-off-by: disksing <i@disksing.com>

* add config-check flag for pd-server (#1695)

Signed-off-by: cwen0 <cwenyin0@gmail.com>

* operator: rewrite move region related functions (#1667)

* *: support setting endKey for ScanRange (#1700)

Signed-off-by: disksing <i@disksing.com>

* *: reduce some unnecessary parameters (#1698)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* schedule: Do not send an operator of a region wth a stale epoch (#1659)

* schedule: Do not send an operator of a region wth a stale epoch

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* schedule: check the version changed by the operator self

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* schedule: fix unit test

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* schedule: fix to avoid dispatching a stale opstep

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool"

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* dispatch: fix typo in comment

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* fix typo

Co-Authored-By: Ryan Leung <rleungx@gmail.com>

* dispatch: fix unittest

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* dispatch: refine format

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* server: fix the dead lock in scatter region (#1706)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* tools: fix set namespace in pd-ctl (#1701)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* tools: fix parse url without http prefix (#1703)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* tests: support deadlock detection in make test (#1704)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* Makefile: fix failpoint enable (#1722)

Signed-off-by: nolouch <nolouch@gmail.com>

* checker: fix the issue that a region does not merge to the sibling with smaller size (#1723)

Signed-off-by: disksing <i@disksing.com>

* tools: balance region simulator (#1708)

* scheduler: do not remove the operator when the step does not finish (#1715)

Signed-off-by: Shafreeck Sea <shafreeck@gmail.com>

* operator: fix the AddLearner config version judgment (#1732)

Signed-off-by: nolouch <nolouch@gmail.com>

* tools: fix TLS in pd control (#1729)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* syncer: support TLS for region syncer (#1728)

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* schedule: fix a thread-safe bug and improve code (#1719)

* statistics: fix region flow calculation (#1688)

Signed-off-by: jiyingtk <jiyingtk@mail.ustc.edu.cn>

* makefile: improve deadlock-enable/disable (#1736)

* api: fix missing keys statistic in region information (#1741)

Signed-off-by: nolouch <nolouch@gmail.com>

* *: update go version to 1.13 (#1742)

Signed-off-by: disksing <i@disksing.com>

* coordinator: add the operator cost time in log field (#1748)

Signed-off-by: nolouch <nolouch@gmail.com>
sre-bot pushed a commit that referenced this pull request Sep 12, 2019
@rleungx @sre-bot
syncer: support TLS for region syncer (#1728) (#1739)
dc03c83 
Signed-off-by: Ryan Leung <rleungx@gmail.com>
nolouch pushed a commit to nolouch/pd that referenced this pull request Nov 12, 2019
@rleungx @nolouch
syncer: support TLS for region syncer (tikv#1728) (tikv#1739)
49a1bc0 
Signed-off-by: Ryan Leung <rleungx@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Connor1996 Connor1996 Connor1996 left review comments

@lhy1024 lhy1024 lhy1024 approved these changes

@nolouch nolouch nolouch approved these changes

@shafreeck shafreeck shafreeck approved these changes

@disksing disksing Awaiting requested review from disksing

Assignees
No one assigned
Labels
needs-cherry-pick-release-3.0 The PR needs to cherry pick to release-3.0 branch. status/can-merge Indicates a PR has been approved by a committer.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

gRPC connection fail when --join with TLS enabled
7 participants
@rleungx @codecov-io @sre-bot @shafreeck @nolouch @Connor1996 @lhy1024