Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drainer OOM in old cluster #1112

Open
YuJuncen opened this issue Dec 28, 2021 · 2 comments
Open

drainer OOM in old cluster #1112

YuJuncen opened this issue Dec 28, 2021 · 2 comments

Comments

@YuJuncen
Copy link

YuJuncen commented Dec 28, 2021

Bug Report

Please answer these questions before submitting your issue. Thanks!

From AskTUG: https://asktug.com/t/topic/274497

  1. What did you do?
    If possible, provide a recipe for reproducing the error.
    Start drainer in a cluster with long history.

  2. What did you expect to see?
    Drainer can startup (even slowly).

  3. What did you see instead?
    OOM. The peak memory usage is up to 64G. Note that the GlobalID in the cluster is 4593562(which means there probably are millions of DDL jobs).

  4. Please provide the relate downstream type and version of drainer.
    (run drainer -V in terminal to get drainer's version)
    v5.1.2

  5. Notes
    As a solution, we can use the CDC like method to resolve this: loading the snapshot and then send DDL jobs incrementally.
    Or, we can paginate the scan of history DDL jobs for reducing the peak memory usage.

@lichunzhu lichunzhu added status/help-wanted type/bug This issue is a bug type/enhancement and removed type/bug This issue is a bug labels Dec 28, 2021
@lichunzhu
Copy link
Contributor

Possible solution:
Use CDC's snapshot method and just load the following jobs. https://github.com/pingcap/tiflow/blob/c063509/cdc/entry/schema_storage.go#L728

@kennytm
Copy link
Contributor

kennytm commented Jul 8, 2024

already "fixed" in #1237 i believe?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants