Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

syncer: add more metrics; refine batch handle #590

Merged
merged 7 commits into from
Apr 10, 2020

Conversation

csuzhangxc
Copy link
Member

@csuzhangxc csuzhangxc commented Apr 9, 2020

What problem does this PR solve?

add more metrics for binlog replication unit.

What is changed and how it works?

  • add more metrics:
    • binlogReadDurationHistogram
    • binlogEventSizeHistogram
    • conflictDetectDurationHistogram
    • queueSizeGauge
    • stmtHistogram
  • refine batch handle (use time.After to replcate time.Sleep)
  • update Grafana dashboard

image

image

Check List

Tests

  • Unit test
  • Integration test

Code changes

  • Has exported variable/fields change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to update the documentation
  • Need to be included in the release note

@csuzhangxc csuzhangxc added priority/normal Minor change, requires approval from ≥1 primary reviewer status/WIP This PR is still work in progress type/enhancement Performance improvement or refactoring labels Apr 9, 2020
@codecov
Copy link

codecov bot commented Apr 9, 2020

Codecov Report

Merging #590 into release-1.0 will increase coverage by 0.0262%.
The diff coverage is 63.1578%.

@@                 Coverage Diff                 @@
##           release-1.0       #590        +/-   ##
===================================================
+ Coverage      57.2636%   57.2899%   +0.0262%     
===================================================
  Files              161        161                
  Lines            16672      16708        +36     
===================================================
+ Hits              9547       9572        +25     
- Misses            6182       6191         +9     
- Partials           943        945         +2     

IANTHEREAL
IANTHEREAL previously approved these changes Apr 9, 2020
IANTHEREAL
IANTHEREAL previously approved these changes Apr 9, 2020
IANTHEREAL
IANTHEREAL previously approved these changes Apr 9, 2020
@csuzhangxc
Copy link
Member Author

@GregoryIan PTAL again

IANTHEREAL
IANTHEREAL previously approved these changes Apr 9, 2020
@csuzhangxc
Copy link
Member Author

/run-all-tests

@csuzhangxc
Copy link
Member Author

/run-all-tests

@@ -998,6 +998,7 @@ func (s *Syncer) sync(tctx *tcontext.Context, queueBucket string, db *DBConn, jo
for {
select {
case sqlJob, ok := <-jobChan:
queueSizeGauge.WithLabelValues(s.cfg.Name, queueBucket).Set(float64(len(jobChan)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multiple queues share same bucket, it would be problem i under some cases.
It is recommended that both slow query
threshold and bucket count be configurable, and these can be optimized later

@csuzhangxc csuzhangxc merged commit f69b2ca into pingcap:release-1.0 Apr 10, 2020
@csuzhangxc csuzhangxc deleted the add-metrics branch April 10, 2020 01:06
csuzhangxc added a commit to csuzhangxc/dm that referenced this pull request Apr 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
priority/normal Minor change, requires approval from ≥1 primary reviewer status/WIP This PR is still work in progress type/enhancement Performance improvement or refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants