Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TiCDC owner availability check #3354

Open
amyangfei opened this issue Nov 9, 2021 · 0 comments
Open

TiCDC owner availability check #3354

amyangfei opened this issue Nov 9, 2021 · 0 comments
Labels
area/ticdc Issues or PRs related to TiCDC. component/metrics-logging Metrics and logging component. subject/new-feature Denotes an issue or pull request adding a new feature.

Comments

@amyangfei
Copy link
Contributor

Is your feature request related to a problem?

Such as https://github.com/pingcap/ticdc/issues/3352 and https://github.com/pingcap/ticdc/issues/3331, ticdc owner could be stuck and the whole TiCDC cluster will be abnormal. Note the owner may be fallen into some long time wait condition, including IO wait, some logic wait condition etc, instead of etcd session lost.

Describe the feature you'd like

Add a background goroutine to detect the availability of cdc owner, it can check either the metric counter of ticdc_owner_ownership_counter or something else, to ensure TiCDC owner works well. If the checker finds TiCDC owner is not available, it could do following things

  • Dump the goroutine of TiCDC owner node
  • Provide some mechanism to evicit TiCDC owner.

Describe alternatives you've considered

No response

Teachability, Documentation, Adoption, Migration Strategy

No response

@amyangfei amyangfei added subject/new-feature Denotes an issue or pull request adding a new feature. component/metrics-logging Metrics and logging component. labels Nov 9, 2021
@ben1009 ben1009 added the area/ticdc Issues or PRs related to TiCDC. label Nov 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ticdc Issues or PRs related to TiCDC. component/metrics-logging Metrics and logging component. subject/new-feature Denotes an issue or pull request adding a new feature.
Projects
None yet
Development

No branches or pull requests

2 participants