Test improvements in 2021 Q2

[amyangfei]

Feature Request

Is your feature request related to a problem? Please describe:

We devote to improve the tests of TiCDC for the following purpose

• Confirm the boundary of TiCDC, including the incremental scan data size, the resource usage under different workload, the cluster size of TiDB that TiCDC can handle, etc.
• Find more potential bugs of TiCDC, such as the resolved ts block issue, the incorrect state transition, etc.
• Increase the development efficiency, including some known issues such as increase the pass rate of CI, decrease the running time of CI, etc.

Describe the feature you'd like:

Some existing tasks are as follows, we need to set proper priority for these tasks

• Port bank test in CDC CI. tests/bank: add bank case #1076 (Note the case should be refined, because the replication of TiCDC is eventual consistency
• Decrease running time of github action test
  • A full run of integration tests in github action costs 1hour.
  • The failure of one job will cause the rerun of the whole workflow.
• CI optimization
  • Increase CI pass rate
    • existing unstable tests, some unstable integration/unit tests #1501, data race in unit tests #1567
    • some integration tests run slowly and will cause K8s pod OOM when CI resource is busy.
  • Decrease CI running time, currently the integration test costs 25-30minutes
• Boundary test (Need refine)
  • The max data size of historical incremental scan with one table.
  • The max data size of historical incremental scan with 50 tables.
  • Test case with large amount of region of one table.
  • Test case of replication with large amount of table.
• Stability test (Need refine)
  • Chaos test of TiCDC
    • add a basic chaos-mesh workflow , action: Integrate chaos mesh into ticdc #1858
    • add one chaos
    • add all selected chaos