Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ticdc: add doc about config (#3745) #3870

Merged
merged 1 commit into from
Jul 3, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions ticdc/manage-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,19 @@ create changefeed ID: 28c43ffc-2316-4f4f-a70b-d1a7c59ba79f info {"sink-uri":"mys
| `partition-num` | 下游 Kafka partition 数量(可选,不能大于实际 partition 数量。如果不填会自动获取 partition 数量。) |
| `max-message-bytes` | 每次向 Kafka broker 发送消息的最大数据量(可选,默认值 `64MB`) |
| `replication-factor` | kafka 消息保存副本数(可选,默认值 `1`) |
| `protocol` | 输出到 kafka 消息协议,可选值有 `default`, `canal`(默认值为 `default`) |

如需设置更多同步任务的配置,比如指定同步单个数据表,请参阅[同步任务配置文件描述](#同步任务配置文件描述)。

使用配置文件创建同步任务的方法如下:

{{< copyable "shell-regular" >}}

```shell
cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="mysql://root:123456@127.0.0.1:3306/" --config changefeed.toml
```

其中 `changefeed.toml` 为同步任务的配置文件。

#### 查询同步任务列表

Expand Down Expand Up @@ -358,6 +371,54 @@ curl -X POST http://127.0.0.1:8301/capture/owner/resign
election: not leader
```

## 同步任务配置文件描述

以下内容详细介绍了同步任务的配置。

```toml
# 指定配置文件中涉及的库名、表名是否为大小写敏感
# 该配置会同时影响 filter 和 sink 相关配置,默认为 true
case-sensitive = true

[filter]
# 忽略指定 start_ts 的事务
ignore-txn-start-ts = [1, 2]

# 过滤器规则
# 过滤规则语法:https://github.com/pingcap/tidb-tools/tree/master/pkg/table-filter#syntax
rules = ['*.*', '!test.*']

[mounter]
# mounter 线程数,用于解码 TiKV 输出的数据
worker-num = 16

[sink]
# 对于 MQ 类的 Sink,可以通过 dispatchers 配置 event 分发器
# 支持 default、ts、rowid、table 四种分发器
dispatchers = [
{matcher = ['test1.*', 'test2.*'], dispatcher = "ts"},
{matcher = ['test3.*', 'test4.*'], dispatcher = "rowid"},
]
# 对于 MQ 类的 Sink,可以指定消息的协议格式
# 目前支持 default 和 canal 两种协议。default 为 TiCDC Open Protocol
protocol = "default"

[cyclic-replication]
# 是否开启环形同步
enable = false
# 当前 TiCDC 的复制 ID
replica-id = 1
# 需要过滤掉的同步 ID
filter-replica-ids = [2,3]
# 是否同步 DDL
sync-ddl = true
```

### 配置文件兼容性的注意事项

* TiCDC v4.0.0 中移除了 `ignore-txn-commit-ts`,添加了 `ignore-txn-start-ts`,使用 start_ts 过滤事务。
* TiCDC v4.0.2 中移除了 `db-dbs`/`db-tables`/`ignore-dbs`/`ignore-tables`,添加了 `rules`,使用新版的数据库和数据表过滤规则,详细语法参考 [Table Filter](https://github.com/pingcap/tidb-tools/blob/master/pkg/table-filter/README.md)。

## 环形同步

> **警告:**
Expand Down
4 changes: 4 additions & 0 deletions ticdc/troubleshoot-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,7 @@ cdc cli changefeed create --sink-uri="mysql://root@127.0.0.1:3306/" --tz=Asia/Sh
> - 最优先使用 `--tz` 传入的时区
> - 没有 `--tz` 参数,会尝试读取 `TZ` 环境变量设置的时区
> - 如果还没有 `TZ` 环境变量,会从 TiCDC server 运行机器的默认时区
## 升级 TiCDC 后配置文件不兼容

请参阅[配置文件兼容注意事项](/ticdc/manage-ticdc.md#配置文件兼容性的注意事项)