Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

engine: merge dataflow engine into tiflow #5343

Merged
merged 281 commits into from
May 20, 2022
Merged

engine: merge dataflow engine into tiflow #5343

merged 281 commits into from
May 20, 2022

Conversation

amyangfei
Copy link
Contributor

@amyangfei amyangfei commented May 6, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #5352

What is changed and how it works?

  • add dataflow engine code base as subtree in engine dir
  • add copyright to all dataflow engine go files
  • update import path of dataflow engine
  • update error toml
  • Minor change to github action CI script, Makefile and go mod
    • In go mod, upgrade github.com/prometheus/client_golang from v1.11.0 to v1.12.2

Check List

Tests

  • Unit test
  • Integration test

Release note 

None

amyangfei and others added 30 commits December 14, 2021 16:08
@amyangfei
[resource manager]: add separated resouce manager interface (pingcap#44)
5355972 
* [resource manager]: add separated resouce manager interface

* fix merge conflicts

* address comments

* fix data race
@hanfei1991
client: decouple clients out of executor managers (pingcap#46)
766eefe 
* client: decouple clients out of executor managers
* refine pkg
* add tests
* address comment and add more comments
@hanfei1991
[jobmaster]: immigrate job master from master to executor (pingcap#48)
7f1d18a
@amyangfei
master: support single master failover (pingcap#47)
6754006
@amyangfei
go-mod: replace pingcap/ticdc with pingcap/tiflow (pingcap#51)
4a237a6 
* go-mod: replace pingcap/ticdc with pingcap/tiflow

* update go.mod
@hanfei1991
adjust the structure of directories (pingcap#52)
63cdbb5
@amyangfei
master: support leader to serve only and return leader info (pingcap#53)
a6d6417
@liuzix
master: implements Election interface (pingcap#55)
d5cc08e
@amyangfei
master: support forward rpc to server master leader (pingcap#56)
b3357b0
@amyangfei
master: check self etcd is leader before campaign leader (pingcap#59)
b61d082
@hanfei1991
persist all the job and tasks info (pingcap#58)
8d040b4 
* persist job master

* pass test

* prepare for test

* address comments
@liuzix
pkg/p2p: add some helper structs for integrating p2p (pingcap#65)
ed9f956
@amyangfei
election: fix panic missing cancel error in context (pingcap#66)
ffb36e6
@amyangfei
server master: refine example config, add a join master sample (pingc…
88ef201 
…ap#68)
@amyangfei
server-master: resign after leader service exits (pingcap#67)
dd21811
@amyangfei
*: add more linters (pingcap#69)
a57d407
@amyangfei
server master/executor: add pprof http api (pingcap#71)
ad8a0d6
@amyangfei
executor: fix cpu usage exhaust (pingcap#72)
f9ea138
@amyangfei
executor: clean exit in some operators (pingcap#74)
ccb5f40
@amyangfei
executor: add task resource unit abstraction (pingcap#76)
6045319
@nongfushanquan
create the demo task and server (pingcap#75)
31409d8 
* add demotask server and client

* remove the temp data files

* fix the make check error

* fix the some errors case
@amyangfei
client: support multi targets and round robin in master client (pingc…
1d8ca8b 
…ap#77)
@amyangfei
client: support auto member update in master client (pingcap#78)
0cd7d4d
@hanfei1991
*: support suspend jobs, part 1 (pingcap#79)
5af0812 
part job of pingcap#60
This PR supports the suspending of jobs and tasks, but it hasn't considered the consistency after fail-over.

The "suspend" is implemented in an async way, because suspending might happen when rescheduling or fail recovering.
After the user sends suspend request, we mark the "target status" for the target tasks. A goroutine will check the status regularly, and will try to suspend it util success. There are some work to do furture.

* The commands should be serializable. If several commands are sent in the same time, we should order them.
* The command have to be persisted and to be continued when node crashes.
@amyangfei
master: make embed etcd server startup cancelable (pingcap#80)
601990b
@amyangfei
jobmaster: refine check running tasks in job master (pingcap#81)
a1931f6 
* jobmaster: refine check running tasks in job master

* use free port instead of hard code port
@amyangfei
master: fix nil leader used in isLeaderAndNeedForward (pingcap#83)
ae38746
@amyangfei
master: use advertise-addr in server master membership (pingcap#82)
adb7042
@amyangfei
test: fix unstable pause job test (pingcap#87)
1425522 
* test: fix unstable pause job test

* add paused check in sink operator

* increase wait time
@liuzix @amyangfei
Implement the master-worker framework (pingcap#84)
055ab52
maxshuang and others added 10 commits May 13, 2022 15:44
test(engine): fix mock backend db start fail when has concurrent test (…
f2a7d6c 
…pingcap#385)
@liuzix
externalresource(engine): Adjust resource manager (pingcap#389)
ca4a8c0
feat(test): support sqlite as backend DB for unit tests (pingcap#391)
2b8a245 
* feat(test): support sqlite as backend DB for unit tests

* fix(orm): remove some useless code

* fix(orm): add some comment
@liuzix
implement CancelJob and GetJobStatus (pingcap#393)
aaa2615
fix(orm): using more proper dsn for sqlite (pingcap#394)
fabe57a
@amyangfei
ci(engine): use gh-docker-logs to dump logs in github action (pingcap…
c16e9f0 
…#392)
@liuzix
lib/*(engine): Implement worker exit fast path (pingcap#361)
88da691
@amyangfei
ci(engine): add chaos-mesh test cases in github action, part-1 (pingc…
fb4028a 
…ap#396)

* ci(engine): add chaos-mesh test cases in github action
@liuzix
pkg/notifier: implement notifier (pingcap#397)
18a651e 
* pkg/notifier: implement notifier
metric(engine): Add Multi-projects/tasks prometheus metric support (p…
9a7a74c 
…ingcap#387)
@amyangfei amyangfei force-pushed the merge-engine branch 2 times, most recently from 04997a2 to c4e07d0 Compare May 19, 2022 07:58
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

@GMHDBJD
DM: adjust job config and support precheck (pingcap#376)
65d5bf4 
* adjust job config and support precheck
@amyangfei amyangfei force-pushed the merge-engine branch 2 times, most recently from 020b4a7 to 399645f Compare May 20, 2022 04:15
maxshuang and others added 3 commits May 20, 2022 12:19
@amyangfei
Copy link
Contributor Author

/run-all-tests

GMHDBJD
GMHDBJD approved these changes May 20, 2022
View reviewed changes
Copy link
Contributor

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels May 20, 2022
@amyangfei
Copy link
Contributor Author

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 20, 2022
@amyangfei amyangfei merged commit 94bb236 into pingcap:master May 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CharlesCheung96 CharlesCheung96 CharlesCheung96 left review comments

@GMHDBJD GMHDBJD GMHDBJD approved these changes

@Rustin170506 Rustin170506 Rustin170506 approved these changes

@liuzix liuzix liuzix approved these changes

@lance6716 lance6716 Awaiting requested review from lance6716

@overvenus overvenus Awaiting requested review from overvenus

@nongfushanquan nongfushanquan Awaiting requested review from nongfushanquan

@niubell niubell Awaiting requested review from niubell

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/LGT2 Indicates that a PR has LGTM 2. status/ptal Could you please take a look? tide/merge-method-merge Denotes a PR that should be merge by tide when it merges.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.