ddl: Corrected index management during REORGANIZE PARTITION (#56786)

[ti-chi-bot]

This is an automated cherry-pick of #56786

What problem does this PR solve?

Issue Number: close #56634

Problem Summary:
issue triggered by injected error, resulting in rollback and panic

Uncovered issues:

• Since Global Indexes needs to be replaced (copied with new references to the new partition ids, since we cannot reference two different partitions in the same unique index entry) both indexes were used during StateWriteOnly, StateWriteReorganization and StateDeleteReorganization, which would cause duplicate entry errors when trying to insert new values during those states for Global Indexes.

What changed and how does it work?

• Avoiding to use StateDeleteReorganization as index state, since during that state, the global indexes may still be read or and should also be kept up-to-date in case of rollback.
• Remove indexes that are not connected to the partition, like for new partitions it should not update indexes that will be removed, those indexes will be kept up-to-date by the old partitions. So double writing, including indexes, will be on partition level, where the data matching:
  • the non-touched partitions should update:
    • all original indexes
    • new indexes
    • So it needs to include all indexes!
  • the old/reorganized partitions should update:
    • all original indexes
    • So it needs to skip new indexes!
  • the new partitions should update:
    • all non-changed indexes
    • new indexes
    • So it needs to skip old indexes that will be replaced with new indexes (All old Global Indexes, and all indexes that will be converted to Global indexes).

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fixed an issue where REORGANIZE PARTITION block inserts during some states, as well as it could fail rollback, in case of internal errors.

[codecov]

Codecov Report

Attention: Patch coverage is 95.50562% with 4 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-8.5@df4b5d5). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-8.5     #57426   +/-   ##
================================================
  Coverage               ?   56.3683%           
================================================
  Files                  ?       1780           
  Lines                  ?     635428           
  Branches               ?          0           
================================================
  Hits                   ?     358180           
  Misses                 ?     252987           
  Partials               ?      24261           

Flag	Coverage Δ
integration	37.2015% <74.1573%> (?)
unit	72.5092% <95.5056%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9478% <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	52.5585% <0.0000%> (?)

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mjonss, tangenta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mjonss,tangenta]
• pkg/ddl/OWNERS [tangenta]
• pkg/meta/OWNERS [tangenta]
• pkg/table/OWNERS [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-11-15 13:59:30.622785955 +0000 UTC m=+623932.813654949: ☑️ agreed by mjonss.
• 2024-11-16 03:57:36.619042297 +0000 UTC m=+674218.809911292: ☑️ agreed by tangenta.