Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: add doc about config #3745

Merged
merged 11 commits into from
Jul 3, 2020
60 changes: 60 additions & 0 deletions ticdc/manage-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,18 @@ create changefeed ID: 28c43ffc-2316-4f4f-a70b-d1a7c59ba79f info {"sink-uri":"mys
| `max-message-bytes` | 每次向 Kafka broker 发送消息的最大数据量(可选,默认值 `64MB`) |
| `replication-factor` | kafka 消息保存副本数(可选,默认值 `1`) |
overvenus marked this conversation as resolved.
Show resolved Hide resolved

如需设置更多同步任务的配置,比如指定同步单个数据表,请参阅[同步任务配置文件描述](#同步任务配置文件描述)。

使用方法如下:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

什么的使用方法呢?建议具体一些。Rest LGTM

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已更新,麻烦尽快 review 下

overvenus marked this conversation as resolved.
Show resolved Hide resolved

{{< copyable "shell-regular" >}}

```shell
cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --config changefeed.toml
```

其中 “changefeed.toml” 为同步任务的配置文件。
overvenus marked this conversation as resolved.
Show resolved Hide resolved

#### 查询同步任务列表

使用以下命令来查询同步任务列表:
Expand Down Expand Up @@ -351,6 +363,54 @@ curl -X POST http://127.0.0.1:8301/capture/owner/resign
election: not leader
```

## 同步任务配置文件描述

以下内容详细介绍了同步任务的配置。

```toml
# 指定配置文件中涉及的库名、表名是否为大小写敏感的
overvenus marked this conversation as resolved.
Show resolved Hide resolved
# 该配置会同时影响 filter 和 sink 相关配置,默认为 true
case-sensitive = true

[filter]
# 忽略指定 StartTs 的事务
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# 忽略指定 StartTs 的事务
# 忽略指定 star_ts 的事务

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"start ts" is a term in tidb.

overvenus marked this conversation as resolved.
Show resolved Hide resolved
ignore-txn-start-ts = [1, 2]

# 过滤器规则
# 过滤规则语法: https://github.com/pingcap/tidb-tools/tree/master/pkg/table-filter#syntax
overvenus marked this conversation as resolved.
Show resolved Hide resolved
rules = ['*.*', '!test.*']

[mounter]
# mounter 线程数,用于解码 TiKV 输出的数据
worker-num = 16

[sink]
# 对于 MQ 类的 Sink,可以通过 dispatchers 配置 event 分发器
# 分发器支持 default, ts, rowid, table 四种
overvenus marked this conversation as resolved.
Show resolved Hide resolved
dispatchers = [
{matcher = ['test1.*', 'test2.*'], dispatcher = "ts"},
{matcher = ['test3.*', 'test4.*'], dispatcher = "rowid"},
]
# 对于 MQ 类的 Sink,可以指定消息的协议格式
# 协议目前支持 default, canal 两种,default 为 ticdc-open-protocol
overvenus marked this conversation as resolved.
Show resolved Hide resolved
protocol = "default"

[cyclic-replication]
# 是否开启环形复制
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

环形同步还是复制呢?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“同步”

overvenus marked this conversation as resolved.
Show resolved Hide resolved
enable = false
# 当前 CDC 的复制 ID
overvenus marked this conversation as resolved.
Show resolved Hide resolved
replica-id = 1
# 需要过滤掉的复制 ID
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

overvenus marked this conversation as resolved.
Show resolved Hide resolved
filter-replica-ids = [2,3]
# 是否同步 DDL
sync-ddl = true
```

### 配置文件兼容注意事项
overvenus marked this conversation as resolved.
Show resolved Hide resolved

* 4.0.0 移除了 `ignore-txn-commit-ts`,添加了 `ignore-txn-start-ts`,使用 start ts 过滤事务。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* 4.0.0 移除了 `ignore-txn-commit-ts`,添加了 `ignore-txn-start-ts`,使用 start ts 过滤事务。
* TiDB v4.0.0 中移除了 `ignore-txn-commit-ts`,添加了 `ignore-txn-start-ts`,使用 start_ts 过滤事务。

overvenus marked this conversation as resolved.
Show resolved Hide resolved
* 4.0.2 移除了 `db-dbs`/`db-tables`/`ignore-dbs`/`ignore-tables`,添加了 `rules`,使用新版的数据库数据表过滤规则,[详细语法参考](https://github.com/pingcap/tidb-tools/blob/master/pkg/table-filter/README.md)。
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* 4.0.2 移除了 `db-dbs`/`db-tables`/`ignore-dbs`/`ignore-tables`,添加了 `rules`使用新版的数据库数据表过滤规则,[详细语法参考](https://github.com/pingcap/tidb-tools/blob/master/pkg/table-filter/README.md)
* TiDB v4.0.2 中移除了 `db-dbs`/`db-tables`/`ignore-dbs`/`ignore-tables`,添加了 `rules`使用新版的数据库和数据表过滤规则,详细语法参考 [Table Filter](https://github.com/pingcap/tidb-tools/blob/master/pkg/table-filter/README.md)

overvenus marked this conversation as resolved.
Show resolved Hide resolved

## 环形同步

> **警告:**
Expand Down
4 changes: 4 additions & 0 deletions ticdc/troubleshoot-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,7 @@ cdc cli changefeed create --pd=http://10.0.10.25:2379 --start-ts=415238226621235
> **注意:**
>
> TiCDC(4.0 发布版本)还不支持动态修改文件排序和内存排序。

## 升级 TiCDC 后配置文件不兼容

请参阅[配置文件兼容注意事项](/ticdc/manage-ticdc.md#配置文件兼容注意事项)。
TomShawn marked this conversation as resolved.
Show resolved Hide resolved