lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS #52712

D3Hunter · 2024-04-18T06:32:41Z

What problem does this PR solve?

Issue Number: close #52654

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

mysql> CREATE TABLE t (
    ->   id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    ->   PRIMARY KEY (id) /*T![clustered_index] NONCLUSTERED */
    -> ) /*T![auto_id_cache] AUTO_ID_CACHE=1 */ /*T! SHARD_ROW_ID_BITS=4 PRE_SPLIT_REGIONS=3 */;
Query OK, 0 rows affected (0.09 sec)

mysql>
mysql> import into t from select 1778961125641936898;
Query OK, 1 row affected (0.30 sec)
Records: 1, ID: 0e8a4adb-47df-4790-a237-5a489e25d9c6

mysql> insert into t values ();
Query OK, 1 row affected (0.01 sec)

mysql> select *, _tidb_rowid from t;
+---------------------+---------------------+
| id                  | _tidb_rowid         |
+---------------------+---------------------+
| 1778961125641936898 | 1152921504606846977 |
| 1778961125641936899 | 2882303761517117442 |
+---------------------+---------------------+
2 rows in set (0.01 sec)
mysql> select *, _tidb_rowid & x'07FFFFFFFFFFFFFF' from t;
+---------------------+-----------------------------------+
| id                  | _tidb_rowid & x'07FFFFFFFFFFFFFF' |
+---------------------+-----------------------------------+
| 1778961125641936899 |                                 2 |
| 1778961125641936898 |                                 1 |
+---------------------+-----------------------------------+
2 rows in set (0.01 sec)

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-04-18T06:32:56Z

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

codecov · 2024-04-18T07:02:22Z

Codecov Report

Merging #52712 (10c5767) into master (f4e0d58) will increase coverage by 1.1544%.
Report is 3 commits behind head on master.
The diff coverage is 97.2222%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #52712        +/-   ##
================================================
+ Coverage   72.3622%   73.5166%   +1.1544%     
================================================
  Files          1471       1493        +22     
  Lines        427144     435614      +8470     
================================================
+ Hits         309091     320249     +11158     
+ Misses        98814      95298      -3516     
- Partials      19239      20067       +828

Flag	Coverage Δ
integration	`48.1835% <94.4444%> (?)`
unit	`71.2000% <89.4736%> (-0.0527%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9957% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`41.3923% <ø> (+0.2930%)`	⬆️

GMHDBJD

LGTM

D3Hunter · 2024-04-18T07:18:14Z

/retest

tiprow · 2024-04-18T07:18:35Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

D3Hunter · 2024-04-18T07:26:09Z

/retest

tiprow · 2024-04-18T07:26:30Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lance6716 · 2024-04-18T07:30:37Z

lightning/pkg/importer/table_import.go

 				// for TiDB version >= 6.5.0, a table might have separate allocators for auto_increment column and _tidb_rowid,
 				// especially when a table has auto_increment non-clustered PK, it will use both allocators.
 				// And in this case, ALTER TABLE xxx AUTO_INCREMENT = xxx only works on the allocator of auto_increment column,
 				// not for allocator of _tidb_rowid.
 				// So we need to rebase IDs for those 2 allocators explicitly.
-				err = common.RebaseGlobalAutoID(ctx, adjustIDBase(newBase), tr, tr.dbInfo.ID, tr.tableInfo.Core)
+				err = common.RebaseTableAllocators(ctx, map[autoid.AllocatorType]int64{
+					autoid.RowIDAllocType:    tr.alloc.Get(autoid.RowIDAllocType).Base(),


should we also rebase to Base()+1? The Base() is the used value or next value?

no need add 1, it's last used value

D3Hunter · 2024-04-18T07:35:02Z

/retest

tiprow · 2024-04-18T07:35:24Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ti-chi-bot · 2024-04-18T07:37:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GMHDBJD, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [GMHDBJD,lance6716]
~~lightning/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/disttask/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/executor/importer/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/lightning/OWNERS~~ [GMHDBJD,lance6716]
~~pkg/table/OWNERS~~ [GMHDBJD]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-04-18T07:37:32Z

[LGTM Timeline notifier]

Timeline:

2024-04-18 07:11:19.736047041 +0000 UTC m=+494898.847093490: ☑️ agreed by GMHDBJD.
2024-04-18 07:37:31.546322972 +0000 UTC m=+496470.657369421: ☑️ agreed by lance6716.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2024-04-18T08:11:24Z

In response to a cherrypick label: new pull request created to branch release-7.1: #52722.

ti-chi-bot · 2024-04-18T08:12:07Z

In response to a cherrypick label: new pull request created to branch release-7.5: #52723.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2024-04-18T08:12:48Z

In response to a cherrypick label: new pull request created to branch release-8.1: #52724.

…ROW_ID_BITS (#52712) (#52722) close #52654

… and SHARD_ROW_ID_BITS (#52712) (#52724) close #52654

… and SHARD_ROW_ID_BITS (#52712) (#52723) close #52654

ti-chi-bot · 2024-05-31T04:17:36Z

In response to a cherrypick label: new pull request created to branch release-6.5: #53704.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

…ROW_ID_BITS (#52712) (#53704) close #52654

D3Hunter added 2 commits April 18, 2024 14:23

change

325b2ab

bazel

6974393

ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 18, 2024

D3Hunter changed the title ~~lightning: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS~~ lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS Apr 18, 2024

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Apr 18, 2024

ti-chi-bot added needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. labels Apr 18, 2024

fix test

10c5767

GMHDBJD approved these changes Apr 18, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 18, 2024

lance6716 reviewed Apr 18, 2024

View reviewed changes

lance6716 approved these changes Apr 18, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 18, 2024

ti-chi-bot bot merged commit 72e5460 into pingcap:master Apr 18, 2024
24 checks passed

D3Hunter deleted the lit-autoidcache branch April 18, 2024 08:11

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 18, 2024

This is an automated cherry-pick of pingcap#52712

da4af42

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Apr 18, 2024

lightning: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS (#52712) #52722

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 18, 2024

This is an automated cherry-pick of pingcap#52712

b962b42

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Apr 18, 2024

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS (#52712) #52723

Merged

13 tasks

ti-chi-bot mentioned this pull request Apr 18, 2024

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS (#52712) #52724

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Apr 18, 2024

lightning: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_…

eff0402

…ROW_ID_BITS (#52712) (#52722) close #52654

ti-chi-bot bot pushed a commit that referenced this pull request Apr 18, 2024

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1…

51791a3

… and SHARD_ROW_ID_BITS (#52712) (#52724) close #52654

D3Hunter mentioned this pull request Apr 28, 2024

lightning import is slow when importing lots of tables with AUTO_INCREMENT #50127

Open

ti-chi-bot bot pushed a commit that referenced this pull request May 21, 2024

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1…

3b4de20

… and SHARD_ROW_ID_BITS (#52712) (#52723) close #52654

ti-chi-bot added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label May 31, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request May 31, 2024

This is an automated cherry-pick of pingcap#52712

9617324

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request May 31, 2024

lightning: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS (#52712) #53704

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Jun 4, 2024

lightning: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_…

42669ad

…ROW_ID_BITS (#52712) (#53704) close #52654

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS #52712

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS #52712

D3Hunter commented Apr 18, 2024 •

edited

Loading

tiprow bot commented Apr 18, 2024

codecov bot commented Apr 18, 2024 •

edited

Loading

GMHDBJD left a comment

D3Hunter commented Apr 18, 2024

tiprow bot commented Apr 18, 2024

D3Hunter commented Apr 18, 2024

tiprow bot commented Apr 18, 2024

lance6716 Apr 18, 2024

D3Hunter Apr 18, 2024

D3Hunter commented Apr 18, 2024

tiprow bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot commented Apr 18, 2024

ti-chi-bot commented Apr 18, 2024

ti-chi-bot commented Apr 18, 2024

ti-chi-bot commented May 31, 2024

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS #52712

lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS #52712

Conversation

D3Hunter commented Apr 18, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Apr 18, 2024

codecov bot commented Apr 18, 2024 • edited Loading

Codecov Report

GMHDBJD left a comment

Choose a reason for hiding this comment

D3Hunter commented Apr 18, 2024

tiprow bot commented Apr 18, 2024

D3Hunter commented Apr 18, 2024

tiprow bot commented Apr 18, 2024

lance6716 Apr 18, 2024

Choose a reason for hiding this comment

D3Hunter Apr 18, 2024

Choose a reason for hiding this comment

D3Hunter commented Apr 18, 2024

tiprow bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

[LGTM Timeline notifier]

ti-chi-bot commented Apr 18, 2024

ti-chi-bot commented Apr 18, 2024

ti-chi-bot commented Apr 18, 2024

ti-chi-bot commented May 31, 2024

D3Hunter commented Apr 18, 2024 •

edited

Loading

codecov bot commented Apr 18, 2024 •

edited

Loading