ticdc: Support Vector data type | tidb=master pd=master tikv=master

[wk989898]

What problem does this PR solve?

Issue Number: ref #11530

What is changed and how it works?

Detect TypeTiDBVectorFloat32, and convert vector to string in DML sink.

CREATE table test.simple(id int primary key, data VECTOR(5));
INSERT INTO test.simple(id, data) VALUES (1, '[1,2,3,4,5]');

Some changes like converting VECTOR to LONGTEXT on an unsupported downstream, supporting multiple output protocols may be temporary. Converting VECTOR to LONGTEXT is automatical when downstream is not TiDB or TiDB version less than 8.3.0.

CREATE table test.simple(id int primary key, data LONGTEXT);
INSERT INTO test.simple(id, data) VALUES (1, '[1,2,3,4,5]');

Check List

Tests

• Unit test
• Integration test
• Manual test
  a. start tidb cluster
  b. create a changefeed
  c. load vector data with https://github.com/breezewish/tidb-vector-bench
  d. check data consistency with sync_diff_inspector

A total of 1 tables need to be compared

Comparing the table structure of `test`.`sample` ... equivalent
Comparing the table data of `test`.`sample` ... equivalent
_____________________________________________________________________________
Progress [============================================================>] 100% 0/0
A total of 1 table have been compared and all are equal.

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

yes

Release note

support vector data type

[ti-chi-bot]

Hi @wk989898. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[3AceShowHand]

/ok-to-test

[wk989898]

/area ticdc

[3AceShowHand]

why this is merged to pingcap:feature/vector-data-type, instead of the master branch ?

[3AceShowHand]

only rewrite the DDL ?

[wk989898]

Yes, the vector format is similar to string in DML.

[wk989898]

why this is merged to pingcap:feature/vector-data-type, instead of the master branch ?

To keep the code tidy. Vector index support will also be provided in the future.

[wk989898]

/retest

[wk989898]

/retest

[wk989898]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [3AceShowHand,lance6716]
• dm/OWNERS [lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-09-20 06:28:08.28334901 +0000 UTC m=+1201758.023772949: ☑️ agreed by 3AceShowHand.
• 2024-09-23 09:35:12.848310763 +0000 UTC m=+1472182.588734701: ☑️ agreed by lance6716.