relay: use new `Reader`, `Transformer` and `Writer` #171

csuzhangxc · 2019-06-12T09:18:15Z

What problem does this PR solve?

this PR is a part of #91.

What is changed and how it works?

use new Reader, Transformer and Writer to:

read binlog events from upstream
transform some binlog events
write binlog events into relay log files
do recover for relay log file if needed before processing
refine some code and add some test cases about this refactoring
- more test cases for the interface methods can be added later.

Check List

Tests

Unit test
Integration test

Code changes

Has persistent data change

Side effects

Possible performance regression

codecov · 2019-06-12T10:07:27Z

Codecov Report

Merging #171 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master       #171   +/-   ##
===========================================
  Coverage   54.7612%   54.7612%           
===========================================
  Files           122        122           
  Lines         14492      14492           
===========================================
  Hits           7936       7936           
  Misses         5766       5766           
  Partials        790        790

…IDsEvent

csuzhangxc · 2019-06-18T12:24:33Z

/run-all-tests

csuzhangxc · 2019-06-19T01:53:00Z

/run-all-tests

…upstream

csuzhangxc · 2019-06-19T09:37:53Z

/run-all-tests

csuzhangxc · 2019-06-19T09:40:14Z

/run-all-tests

csuzhangxc · 2019-06-19T09:49:54Z

@GregoryIan @amyangfei PTAL

IANTHEREAL · 2019-06-20T06:48:43Z

pkg/binlog/reader/tcp_test.go

-	_, err = t.db.Exec(query)
-	c.Assert(err, IsNil)
+	maxRetryCount := 5
+	for i := 0; i < maxRetryCount; i++ {


maybe we can have a function in pkg/unit, like

func WaitSomething(backoff int, fn func() bool) bool { for i := 0; i < backoff; i++ { if fn() { return true } time.Sleep(10 * time.Millisecond) } return false }

done in 6c3fab5.

IANTHEREAL · 2019-06-20T08:24:26Z

relay/relay.go

 			case replication.ErrChecksumMismatch:
 				relayLogDataCorruptionCounter.Inc()
 			case replication.ErrSyncClosed, replication.ErrNeedSyncAgain:
-				// do nothing
+				// do nothing, but the error will be returned


also stat it into metric?

it will be recorded in relayExitWithErrorCounter, should we need to add another metric like relayGoMySQLError?

IANTHEREAL · 2019-06-20T08:30:04Z

relay/relay.go

+		lastPos.Pos = tResult.LogPos
+		err = lastGTID.Set(tResult.GTIDSet)
+		if err != nil {
+			log.Errorf("[relay] update last GTID set to %v error %v", tResult.GTIDSet, err)


it's different with

err = r.meta.Save(lastPos, lastGTID) if err != nil { return errors.Trace(err) }

one is lenient, one is strict, can both be lenient？

change both to be strict in 201f960.

IANTHEREAL · 2019-06-21T08:51:47Z

Rest LGTM

amyangfei · 2019-06-21T09:05:26Z

pkg/binlog/reader/tcp_test.go

+		}
+		return true
+	}
+	utils.WaitSomething(backoff, waitTime, waitFn)


does it need to check the wait result?
besides, in which scenario will other test case write binlog even after this test case is started

WaitSomething can only decrease the possibility of errors.
I observed after RESET MASTER, other test cases write binlog event, may it because multi cases run in different goroutines at the same time?

I observed the fact

see https://stackoverflow.com/questions/24375966/does-go-test-run-unit-tests-concurrently.

IANTHEREAL · 2019-06-22T05:34:07Z

relay/reader/reader.go

@@ -27,6 +28,16 @@ import (
 	"github.com/pingcap/dm/pkg/log"
 )

+const (
+	// event timeout when trying to read events from upstream master server.
+	eventTimeout = 1 * time.Hour


In order to the delay requirements of replication, let it 10 or 1minute?

It only takes effect for getting events from go-mysql (the events are already read from upstream server).
In fact, we can even use a Cancel rather than a Timeout, but use a Timeout we can log something to indicate still running.

the timeout between master and salve is controlled by slaveReadTimeout (default 1m) and masterHeartbeatPeriod (default 30s).

every step will have an impact on the delay, for example, if a master-slave switch happend, there may be no data in 1h, delay would be 1h

every step will have an impact on the delay

agree.

if a master-slave switch happend

An error will get with small delay

right, set a reasonable small value and let user or dm know something ASAP

changed to 10m in 0527d16.

IANTHEREAL · 2019-06-22T06:28:56Z

LGTM

csuzhangxc · 2019-06-22T10:27:28Z

/run-all-tests

relay/relay.go

amyangfei · 2019-06-25T06:25:16Z

dm/worker/relay_test.go

+
+// Process implements Process interface
+func (d *DummyRelay) Process(ctx context.Context, pr chan pb.ProcessResult) {
+	select {


here we can use a simple channel send/receive instead of select with a single case

refined in f9b1663.

amyangfei · 2019-06-25T06:28:03Z

pkg/binlog/reader/tcp_test.go

+		time.Sleep(time.Second)
+		_, gs, err2 := utils.GetMasterStatus(t.db, flavor)
+		c.Assert(err2, IsNil)
+		if len(gs.String()) > 0 {


return gs.String() == "" is simpler?

done in f9b1663.

amyangfei · 2019-06-25T06:30:14Z

pkg/utils/util_test.go

+	// wait success
+	f2 := func() bool {
+		count++
+		if count >= 5 {


return count >= 5

done in f9b1663.

amyangfei · 2019-06-25T06:31:41Z

relay/relay.go

-						tryReSync = false // do not support repeat try re-sync
-						continue
-					}
+					// TODO: try auto fix GTID, and can support auto switching between upstream server later.


do we need to add some log here

added some log in f9b1663.

amyangfei · 2019-06-25T06:33:15Z

relay/relay.go

-
-	// record current active relay log file, and keep it until newer file opened
-	// when current file's fd closed, we should not reset this, because it may re-open again
-	r.setActiveRelayLog(filename)


do we need to call setActiveRelayLog somewhere?

Good catch! add a call to setActiveRelayLog in f9b1663.

Co-Authored-By: amyangfei <amyangfei@gmail.com>

csuzhangxc · 2019-06-26T15:11:28Z

/run-all-tests

amyangfei

LGTM

*: use reader to reader binlog from upstream; do not check gap anymore

fec35ed

csuzhangxc added priority/important Major change, requires approval from ≥2 primary reviewers status/WIP This PR is still work in progress type/enhancement Performance improvement or refactoring labels Jun 12, 2019

*: let reader handle more errors

3e6a305

csuzhangxc added 11 commits June 12, 2019 20:04

*: use transformer and writer; TODO, handle duplicated PreviousGT…

f22c105

…IDsEvent

relay: fix meta update

12d4cac

Merge remote-tracking branch 'remotes/origin/master' into relay-writer-3

242ea36

relay: remove some useless code

fece692

relay: refine code

481b624

relay: test for handling events

c5460f9

relay: move some test only code into _test.go file

dd604c0

relay: extract and test for isNewServer

c314dd4

relay: test reSetupMeta

92d9983

relay: test processing

67c2f7c

relay: try fix CI

f255895

csuzhangxc added 5 commits June 19, 2019 10:25

reader: try fix CI

93ec22e

reader: try fix CI

9f22aaa

relay: try fix CI

9e108d7

*: writer return value rather than pointer

8e83cf1

*: recover relay log file before starting to read binlog events from …

a18e395

…upstream

csuzhangxc added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/WIP This PR is still work in progress labels Jun 19, 2019

IANTHEREAL reviewed Jun 20, 2019

View reviewed changes

csuzhangxc added 5 commits June 21, 2019 10:37

Merge remote-tracking branch 'remotes/origin/master' into relay-writer-3

2f0de38

*: address comments, add a WaitSomething function

6c3fab5

relay: address comments

201f960

relay: check GTID sets Equal before Contain

2e8ed95

Merge remote-tracking branch 'remotes/origin/master' into relay-writer-3

f20bf92

IANTHEREAL and others added 3 commits June 21, 2019 16:51

Merge branch 'master' into relay-writer-3

fed7086

*: address comments

071bc72

Merge remote-tracking branch 'origin/relay-writer-3' into relay-writer-3

c93c496

amyangfei reviewed Jun 21, 2019

View reviewed changes

IANTHEREAL reviewed Jun 22, 2019

View reviewed changes

IANTHEREAL added status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Jun 22, 2019

reader: address comment, change eventTimeout

0527d16

amyangfei reviewed Jun 25, 2019

View reviewed changes

csuzhangxc and others added 4 commits June 26, 2019 15:14

Update relay/relay.go

184d1b8

Co-Authored-By: amyangfei <amyangfei@gmail.com>

*: address comments

f9b1663

Merge branch 'master' into relay-writer-3

88f6455

Merge branch 'master' into relay-writer-3

3bc0cc1

amyangfei approved these changes Jun 27, 2019

View reviewed changes

amyangfei added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Jun 27, 2019

csuzhangxc merged commit 6fbddf3 into pingcap:master Jun 27, 2019

csuzhangxc deleted the relay-writer-3 branch June 27, 2019 02:11

lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020

relay: use new Reader, Transformer and Writer (pingcap#171)

9386fb1

relay: use new Reader, Transformer and Writer #171

relay: use new Reader, Transformer and Writer #171

Conversation

csuzhangxc commented Jun 12, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Jun 12, 2019 • edited Loading

Codecov Report

csuzhangxc commented Jun 18, 2019

csuzhangxc commented Jun 19, 2019

csuzhangxc commented Jun 19, 2019

csuzhangxc commented Jun 19, 2019

csuzhangxc commented Jun 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IANTHEREAL commented Jun 21, 2019

amyangfei Jun 21, 2019 • edited Loading

Choose a reason for hiding this comment

csuzhangxc Jun 21, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csuzhangxc Jun 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csuzhangxc Jun 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IANTHEREAL commented Jun 22, 2019

csuzhangxc commented Jun 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csuzhangxc commented Jun 26, 2019

amyangfei left a comment

Choose a reason for hiding this comment

relay: use new `Reader`, `Transformer` and `Writer` #171

relay: use new `Reader`, `Transformer` and `Writer` #171

csuzhangxc commented Jun 12, 2019 •

edited

Loading

codecov bot commented Jun 12, 2019 •

edited

Loading

amyangfei Jun 21, 2019 •

edited

Loading

csuzhangxc Jun 21, 2019 •

edited

Loading

csuzhangxc Jun 22, 2019 •

edited

Loading

csuzhangxc Jun 22, 2019 •

edited

Loading