Skip to content

Latest commit

 

History

History
209 lines (154 loc) · 15 KB

CHANGELOG.md

File metadata and controls

209 lines (154 loc) · 15 KB
Raw

DM Changelog

All notable changes to this project will be documented in this file.

[1.0.5] 2020-04-27

Improvements

  • Improve the incremental replication speed when the UNIQUE KEY column has the NULL value
  • Add retry for the Write conflict (9007 and 8005) error returned by TiDB

Bug fixes

  • Fix the issue that the Duplicate entry error might occur during the full data import
  • Fix the issue that the replication task cannot be stopped or paused when the full data import is completed and the upstream has no written data
  • Fix the issue the monitoring metrics still display data after the replication task is stopped

Action required

  • When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together

Detailed Bug Fixes and Changes

  • Improve the incremental replication speed when the UNIQUE KEY column has the NULL value #588 #597
  • Add retry for the Write conflict (9007 and 8005) error returned by TiDB #632
  • Fix the issue that the Duplicate entry error might occur during the full data import #554
  • Fix the issue that the replication task cannot be stopped or paused when the full data import is completed and the upstream has no written data #622
  • Fix the issue the monitoring metrics still display data after the replication task is stopped #616
  • Fix the issue that the Column count doesn't match value count error might be returned during the sharding DDL replication #624
  • Fix the issue that some metrics such as data file size are incorrectly displayed when the paused task of full data import is resumed #570
  • Add and fix multiple monitoring metrics #590 #594

[1.0.4] 2020-03-13

Improvements

  • Add English UI for DM-portal
  • Add the --more parameter in the query-status command to show complete replication status information

Bug fixes

  • Fix the issue that resume-task might fail to resume the replication task which is interrupted by the abnormal connection to the downstream TiDB server
  • Fix the issue that the online DDL operation cannot be properly replicated after a failed replication task is restarted because the online DDL meta information has been cleared after the DDL operation failure
  • Fix the issue that query-error might cause the DM-worker to panic after start-task goes into error
  • Fix the issue that the relay log file and relay.meta cannot be correctly recovered when restarting an abnormally stopped DM-worker process before relay.meta is successfully written

Action required

  • When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together

Detailed Bug Fixes and Changes

  • Add English UI for DM-portal #480
  • Add the --more parameter in the query-status command to show complete replication status information #533
  • Fix the issue that resume-task might fail to resume the replication task which is interrupted by the abnormal connection to the downstream TiDB server #436
  • Fix the issue that the online DDL operation cannot be properly replicated after a failed replication task is restarted because the online DDL meta information is cleared after the DDL operation failure #465
  • Fix the issue that query-error might cause the DM-worker to panic after start-task goes into error #519
  • Fix the issue that the relay log file and relay.meta cannot be correctly recovered when restarting an abnormally stopped DM-worker process before relay.meta is successfully written #534
  • Fix the issue that the value out of range error might be reported when getting server-id from the upstream #538
  • Fix the issue that when Prometheus is not configured DM-Ansible prints the wrong error message that DM-master is not configured #438

[1.0.3] 2019-12-13

Improvements

  • Add the command mode in dmctl
  • Support replicating the ALTER DATABASE DDL statement
  • Optimize the error message output

Bug fixes

  • Fix the panic-causing data race issue occurred when the full import unit pauses or exits
  • Fix the issue that stop-task and pause-task might not take effect when retrying SQL operations to the downstream

Action required

  • When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together

Detailed Bug Fixes and Changes

  • Add the command mode in dmctl #364
  • Optimize the error message output #351
  • Optimize the output of the query-status command #357
  • Optimize the privilege check for different task modes #374
  • Support checking the duplicate quotaed route-rules or filter-rules in task config#385
  • Support replicating the ALTER DATABASE DDL statement #389
  • Optimize the retry mechanism for anomalies #391
  • Fix the panic issue caused by the data race when the import unit pauses or exits #353
  • Fix the issue that stop-task and pause-task might not take effect when retrying SQL operations to the downstream #400
  • Upgrade golang to v1.13 and upgrade the version of other dependencies #362
  • Filter the error that the context is canceled when a SQL statement is being executed #382
  • Fix the issue that the error occurred when performing a rolling update to DM monitor using DM-ansible causes the update to fail #408

[1.0.2] 2019-10-30

Improvements

  • Generate some config items for DM-worker automatically
  • Generate some config items for replication task automatically
  • Simplify the output of query-status without arguments
  • Manage DB connections directly for downstream

Bug fixes

  • Fix some panic when starting up or executing SQL statements
  • Fix abnormal sharding DDL replication on DDL execution timeout
  • Fix starting task failure caused by the checking timeout or any inaccessible DM-worker
  • Fix SQL execution retry for some error

Action required

  • When upgrading from a previous version, note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together

Detailed Bug Fixes and Changes

  • Generate random server-id for DM-worker config automatically #337
  • Generate flavor for DM-worker config automatically #328
  • Generate relay-binlog-name and relay-binlog-gtid for DM-worker config automatically #318
  • Generate table name list for dumping in task config from black & white table lists automatically #326
  • Add concurrency items (mydumper-thread, loader-thread and syncer-thread) for task config #314
  • Simplify the output of query-status without arguments #340
  • Fix abnormal sharding DDL replication on DDL execution timeout #338
  • Fix potential DM-worker panic when restoring subtask from local meta #311
  • Fix DM-worker panic when committing a DML transaction failed #313
  • Fix DM-worker or DM-master panic when the listening port is being used #301
  • Fix retry for error code 1105 #321, #332
  • Fix retry for Duplicate entry and Data too long for column #313
  • Fix task check timeout when having large amounts of tables in upstream #327
  • Fix starting task failure when any DM-worker is not accessible #319
  • Fix potential DM-worker startup failure in GTID mode after being recovered from corrupt relay log #339
  • Fix in-memory TPS count for sync unit #294
  • Manage DB connections directly for downstream #325
  • Improve error system by refining error information passed between components #320

[1.0.1] 2019-09-10

Bug fixes

  • Fix a bug that may cause database connection re-establish too frequent #280
  • Fix a potential panic bug when we query-status during subtask unit transforming between different units #274

[1.0.0] 2019-09-06

v1.0.0 What's New

Improvements

  • Add auto recovery framework
  • Make task retryable when encounters a database driver error
  • Make task retryable when encounters network issue
  • Improve the DDL compatibility in DM

Bug fixes

  • Fix the bug that has a risk of data loss when the upstream database connection is abnormal

Action required

  • When upgrading from a previous version, please note that you must upgrade all DM components (dmctl/DM-master/DM-worker) together

Detailed Bug Fixes and Changes

  • Retry for upstream bad connection #265
  • Added some retryable errors in underlying database implementation #256
  • Added task auto recovery framework #246
  • Unify DB operation into one pkg for load and binlog replication #236
  • Fixed a bug in task handler that may cause dm-worker panic #225
  • Refactor DM error system #216
  • Add strictly config verification in dm-master, dm-worker and task #212
  • Limit DM error and log message length #257
  • Support case insensitive binlog event filter in binlog replication #188
  • Refactor DM log with pingcap/log #195
  • Use INSERT instead of REPLACE under non safe-mode #199

[1.0.0-rc.1] 2019-07-01

  • Remove the restriction of "the next shard DDL statement cannot be executed unless the current shard DDL operation is completely finished in shard merge scene for binlog replication" #177
  • Support retry task on the invalid connection error for binlog replication #66
  • Support generating -schema-create.sql files automatically for full migration #186
  • Use TiDB SQL Parser to parse and restore DDL statements in binlog query events, and remove the use db statement when replicating to downstream #54
  • Support migrating tables with generated column for full migration and binlog replication #42 #60
  • Support appending the task name as a suffix to the dumped data directory (dir in the task configuration file) for full migration #100
  • Support resuming tasks automatically after the DM-worker process restarted #88 #116
  • Support skipping pre-check items when starting tasks #65
  • Support case-insensitive in binlog event type of Binlog Event Filter rules #188
  • Optimize relay log mechanism to improve compatibility with upstream MySQL/MariaDB #92 #108 #117 #140 #171
  • Optimize RPC framework to limit the concurrency between DM-master and DM-worker #157
  • Redirect the log of mydumper into the log of DM-worker #93
  • Support using --print-sample-config command flag to show the sample config of DM-worker and DM-master #28
  • Support showing the status of sharding DDL replication in the Grafana dashboard and optimize the dashboard #96 #101 #120
  • Support setting the max_allowed_packet parameter for the database connection to the upstream and downstream #99
  • Support using unsafe_cleanup in DM Ansible to clean components of DM cluster #128
  • Fix flushing checkpoint wrongly when existing multiple sharding groups #124
  • Fix the wrong progress status when loading dumped files #89
  • Greatly improve the test to ensure correctness
  • Fix many other bugs that don't affect correctness

[1.0.0-alpha] 2019-01-18

  • Support the full data migration and the incremental data migration from MySQL/MariaDB into TiDB
  • Support running multi independent synchronization tasks concurrently on one DM-worker instance
  • Support synchronizing a certain table of the upstream MySQL/MariaDB instance to the specified table in TiDB by table routing
  • Support only synchronizing or filtering all operations of some databases or some tables by black and white lists
  • Support synchronizing or filtering some of the binlog events by binlog event filter
  • Support modifying auto-increment primary key fields to resolve the conflicts for shard tables by column mapping
  • Support merging the original shard instances and tables into TiDB but with some restrictions
  • Support synchronizing data definition changed by online DDL tools (including pt-osc and gh-ost)
  • Support handling synchronization trouble caused by DDL which is not supported by TiDB