*: support MySQL compatible `AUTO_INCREMENT` #38449

tiancaiamao · 2022-10-13T08:01:00Z

What problem does this PR solve?

Issue Number: close #38442

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2022-10-13T08:01:02Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

mjonss · 2022-10-13T09:55:37Z

docs/design/2022-10-13-mysql-compatible-auto-increment.md

+## Compatibility
+
+The old implementation can still be kept, its performance is better and might meet some user's needs.
+We can rename the syntax for it, such as `AUTO_INCREMENT_FAST` or `AUTO_INCREMENT_OLD` or `DEFICIT_AUTO_INCREMENT`.


I would propose the name 'AUTO_INCREMENT_CACHED' for the old implementation or 'CACHED_AUTO_INCREMENT'.

If we name this:

`AUTO_INCREMENT`` for the new fully MySQL/InnoDB compatible method

AUTO_INCREMENT_FAST for the old, more distributed autoid

Then when migrating, using ORMs, tools, etc people are likely to use the new MySQL compatible autoid instead of the more scalable distributed autoid while they might not need the new behavior.

Another way of doing this would be this:

SET tidb_autoid_mysql_compatible=ON; CREATE TABLE t1 (id INT AUTO_INCREMENT PRIMARY KEY);

By using a session variable instead of a different keyword we allow the default to use the distributed autoid while only using the new MySQL compatibility when really needed. The session variable can be set on the connection or via a global variable, this makes it possible to run third-party applications that rely on the new MySQL compatible autoid without having to change the SQL queries.

I've update this part.

Use AUTO_ID_CACHE 1 to make it compatible with both old TiDB and MySQL

mjonss · 2022-10-13T10:03:55Z

docs/design/2022-10-13-mysql-compatible-auto-increment.md

+
+## Proposal
+
+This proposes that we introduce a centralized auto ID allocating service, the `AUTO_INCREMENT` ID is allocated from the centralized service and there is no caching mechanism on the TiDB layer, so the behaviour can be MySQL compatible.


So this would be another service in a primary and a backup TiDB node? Which are the reasons for having it in TiDB vs using the primary PD node for this service?

Now it's embeded in TiDB ... just like the ddl owner

mjonss · 2022-10-13T10:05:11Z

docs/design/2022-10-13-mysql-compatible-auto-increment.md

+The old implementation can still be kept, its performance is better and might meet some user's needs.
+We can rename the syntax for it, such as `AUTO_INCREMENT_FAST` or `AUTO_INCREMENT_OLD` or `DEFICIT_AUTO_INCREMENT`.
+
+The new `AUTO_INCREMENT` behaviour is still a compatibility-breaker between old and new versioned TiDB, but it's worthy on behalf of MySQL compatibility.


Are all the auto_increment allocations done during INSERT and UPDATE done the same ways in TiDB as in MySQL? (which may be another issue of compatibility)

INSERT and UPDATE handling is the same between TiDB and MySQL...

mjonss · 2022-10-13T10:14:42Z

docs/design/2022-10-13-mysql-compatible-auto-increment.md

+
+When the switch happen, the current max allocated ID information is required so as to guarantee the uniqueness. The new primary process allocate IDs begin from the max allocated ID plus one.
+
+We can persist the max allocated ID every time, but that's costly. Another choice is persisting it periodically, it's safe to allocate IDs in range `[base, max persisted)`. When the primary process crash abnormally, the backup process gets the max persisted ID as its base. Etcd is also used as the storage for the max persisted ID. This optimization could still make the ID not be consecutive, but it's not so common (and I believe MySQL may also crash and facing such problem?), so this is not a big issue.


For MySQL it depends on version, it changed in 8.0 vs 5.7, see https://dev.mysql.com/doc/refman/8.0/en/innodb-auto-increment-handling.html#innodb-auto-increment-initialization

In 5.7 on (re)start of the server, it would initialise the auto_increment like SELECT MAX(auto_inc_col) FROM t, while in 8.0 it persist each allocation in the redo log. So there should not be any 'lost' values due to caching and restart.

Also there are a lot of different details that can still differ between TiDB and MySQL/InnoDB, like the setting of sysvar_innodb_autoinc_lock_mode.

But getting the basics more compatible is good!

after reload table, the allocator is change to the basic one Lines starting

Please enter the commit message for your changes. Lines starting

tiancaiamao · 2022-10-31T12:45:58Z

/run-build-arm64 comment=true

sre-bot · 2022-10-31T12:56:02Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/cba21767e97d0afdce02ca62bef0a78de71b3888/centos7/tidb-linux-arm64.tar.gz

tiancaiamao · 2022-10-31T14:48:31Z

/run-build-arm64 comment=true

sre-bot · 2022-10-31T14:58:30Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/5d7d63c062dcfc81c144cf4e3d706cb661bef776/centos7/tidb-linux-arm64.tar.gz

tiancaiamao · 2022-11-01T12:52:39Z

/run-build-arm64 comment=true

sre-bot · 2022-11-01T13:02:37Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/1ef80ab69539db36511a5123a9f4ff310d92e6fa/centos7/tidb-linux-arm64.tar.gz

tiancaiamao · 2022-11-02T12:08:23Z

/run-build-arm64 comment=true

sre-bot · 2022-11-02T12:18:26Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/9891dc672523596ed1df8df463e8b307dc46733d/centos7/tidb-linux-arm64.tar.gz

ti-chi-bot · 2022-11-03T11:32:25Z

@tiancaiamao: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiancaiamao · 2022-11-03T12:53:08Z

/run-build-arm64 comment=true

sre-bot · 2022-11-03T13:03:03Z

download tidb binary(linux arm64) at http://fileserver.pingcap.net/download/builds/pingcap/test/tidb/40ddeff6de71cc1331104d6a456194394060af68/centos7/tidb-linux-arm64.tar.gz

tiancaiamao · 2023-06-16T16:05:55Z

This one had been splited into several PRs and merged.

tiancaiamao added 7 commits September 27, 2022 15:03

try to alloc auto id from PD

b985ced

add metrics

a78f2b6

implement increment & offset, update kvproto

cecce8a

add standalone auto id service

2d794ab

add etcd persist store, debug

fa0a75c

support client side service discover, auto recover

ff53e6e

Merge branch 'master' into autoid

84fb7dc

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 13, 2022

tiancaiamao added 3 commits October 13, 2022 16:16

docs: add the design document for MySQL compatible AUTO_INCREMENT

7ac0013

fix a markdown format error

b593327

add target autoid_server to Makefile

349a371

mjonss reviewed Oct 14, 2022

View reviewed changes

tiancaiamao added 3 commits October 14, 2022 22:33

mock for UT

968d75b

mock server instance should be unique!

50ae05a

fix some test case

1cf761a

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 19, 2022

tiancaiamao added 4 commits October 20, 2022 15:59

fix 'alter table auto_increment_id =xxx'

33d047f

after reload table, the allocator is change to the basic one Lines starting

save ... use tikv as persist storage to keep compatiblity

5f3d182

built autoid service into tidb process

4ce0573

fix more unit test cases

d2fbe42

tiancaiamao mentioned this pull request Oct 26, 2022

auto-increment: update doc for the new implementation pingcap/docs-cn#11798

Merged

14 tasks

tiancaiamao added 4 commits October 27, 2022 00:27

debugging

259ceb8

Please enter the commit message for your changes. Lines starting

copy the allocator's code to fix some max / min bug

664d549

fix all test cases

db4d06f

Merge branch 'master' into autoid

3cdda66

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 27, 2022

tiny cleanup

106e97d

add ALTER TABLE t AUTO_ID_CACHE test case and fix bug

832454b

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 31, 2022

add test case for rename table and check auto_id not change

cba2176

fix a bug casued sysbench Duplicated key error

5d7d63c

tiancaiamao added 2 commits November 1, 2022 16:11

use owner package to replace the old leader.go

e22b411

handle HA when tidb exit

1ef80ab

Merge branch 'master' into autoid

6705336

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 1, 2022

update document

9891dc6

This was referenced Nov 2, 2022

*: support mysql compatible auto_increment, the client side changes #38809

Merged

*: support mysql compatible auto_increment, the server side changes #38810

Merged

MySQL compatible AUTO_INCREMENT #38442

Closed

release resource to avoid leak when GetClient() fail

15c2636

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 3, 2022

add a backoff when reconnect

40ddeff

tiancaiamao mentioned this pull request Nov 10, 2022

*: separate auto_increment ID allocator from _tidb_rowid allocator when AUTO_ID_CACHE=1 #39041

Merged

12 tasks

tiancaiamao closed this Jun 16, 2023

tiancaiamao deleted the autoid branch June 16, 2023 16:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: support MySQL compatible `AUTO_INCREMENT` #38449

*: support MySQL compatible `AUTO_INCREMENT` #38449

tiancaiamao commented Oct 13, 2022

ti-chi-bot commented Oct 13, 2022

mjonss Oct 13, 2022

dveeden Oct 27, 2022

tiancaiamao Nov 1, 2022

mjonss Oct 13, 2022

tiancaiamao Nov 1, 2022

mjonss Oct 13, 2022

tiancaiamao Nov 1, 2022

mjonss Oct 13, 2022

tiancaiamao commented Oct 31, 2022

sre-bot commented Oct 31, 2022

tiancaiamao commented Oct 31, 2022

sre-bot commented Oct 31, 2022

tiancaiamao commented Nov 1, 2022

sre-bot commented Nov 1, 2022

tiancaiamao commented Nov 2, 2022

sre-bot commented Nov 2, 2022

ti-chi-bot commented Nov 3, 2022

tiancaiamao commented Nov 3, 2022

sre-bot commented Nov 3, 2022

tiancaiamao commented Jun 16, 2023


		## Proposal

		This proposes that we introduce a centralized auto ID allocating service, the `AUTO_INCREMENT` ID is allocated from the centralized service and there is no caching mechanism on the TiDB layer, so the behaviour can be MySQL compatible.


		When the switch happen, the current max allocated ID information is required so as to guarantee the uniqueness. The new primary process allocate IDs begin from the max allocated ID plus one.

		We can persist the max allocated ID every time, but that's costly. Another choice is persisting it periodically, it's safe to allocate IDs in range `[base, max persisted)`. When the primary process crash abnormally, the backup process gets the max persisted ID as its base. Etcd is also used as the storage for the max persisted ID. This optimization could still make the ID not be consecutive, but it's not so common (and I believe MySQL may also crash and facing such problem?), so this is not a big issue.

*: support MySQL compatible AUTO_INCREMENT #38449

*: support MySQL compatible AUTO_INCREMENT #38449

Conversation

tiancaiamao commented Oct 13, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiancaiamao commented Oct 31, 2022

sre-bot commented Oct 31, 2022

tiancaiamao commented Oct 31, 2022

sre-bot commented Oct 31, 2022

tiancaiamao commented Nov 1, 2022

sre-bot commented Nov 1, 2022

tiancaiamao commented Nov 2, 2022

sre-bot commented Nov 2, 2022

ti-chi-bot commented Nov 3, 2022

tiancaiamao commented Nov 3, 2022

sre-bot commented Nov 3, 2022

tiancaiamao commented Jun 16, 2023

*: support MySQL compatible `AUTO_INCREMENT` #38449

*: support MySQL compatible `AUTO_INCREMENT` #38449