-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ticdc: add more faq #3882
Merged
Merged
ticdc: add more faq #3882
Changes from 4 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
af6762b
ticdc: add more faq
overvenus 598eae7
address lints
overvenus c5329cc
add a new faq
overvenus 164b953
address lints
overvenus f22227e
Update ticdc/troubleshoot-ticdc.md
TomShawn e5d9bd3
address comments and add more faq
overvenus 4da2bda
Merge branch 'cdc/faq' of github.com:overvenus/docs-cn into cdc/faq
overvenus 33fc974
address comments
overvenus 37c1a23
refine format
TomShawn 8d254b1
Merge branch 'master' into cdc/faq
ti-srebot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,7 +8,7 @@ aliases: ['/docs-cn/dev/reference/tools/ticdc/troubleshoot/'] | |
|
||
本文档总结了在使用 TiCDC 过程中经常遇到的问题,给出合适的运维方法。本文档还总结了常见的运行故障,并给出相对应的解决方案。 | ||
|
||
## 启动任务时如何选择 start-ts | ||
## TiCDC 创建任务时如何选择 start-ts? | ||
|
||
首先需要理解同步任务的 `start-ts` 对应于上游 TiDB 集群的一个 TSO,同步任务会从这个 TSO 开始请求数据。所以同步任务的 `start-ts` 需要满足以下两个条件: | ||
|
||
|
@@ -17,11 +17,11 @@ aliases: ['/docs-cn/dev/reference/tools/ticdc/troubleshoot/'] | |
|
||
如果不指定 `start-ts` 或者指定 `start-ts=0`,在启动任务的时候会去 PD 获取一个当前 TSO,并从该 TSO 开始同步。 | ||
|
||
## 启动任务时提示部分表不能同步 | ||
## 为什么 TiCDC 创建任务时提示部分表不能同步? | ||
|
||
在使用 `cdc cli changefeed create` 创建同步任务时会检查上游表是否符合[同步限制](/ticdc/ticdc-overview.md#同步限制)。如果存在表不满足同步限制,会提示 `some tables are not eligible to replicate` 并列出这些不满足的表。用户选择 `Y` 或 `y` 则会继续创建同步任务,并且同步过程中自动忽略这些表的所有更新。用户选择其他输入,则不会创建同步任务。 | ||
|
||
## 同步中断的处理方法 | ||
## 如何处理 TiCDC 同步任务的中断? | ||
|
||
目前已知可能发生的同步中断包括以下两类场景: | ||
|
||
|
@@ -39,7 +39,7 @@ aliases: ['/docs-cn/dev/reference/tools/ticdc/troubleshoot/'] | |
2. 使用新的任务配置文件,增加`ignore-txn-start-ts` 参数跳过指定 `start-ts` 对应的事务。 | ||
3. 通过 HTTP API 停止旧的同步任务,使用 `cdc cli changefeed create` ,指定新的任务配置文件,指定 `start-ts` 为刚才记录的 `checkpoint-ts`,启动新的同步任务恢复同步。 | ||
|
||
## `gc-ttl` 和文件排序 | ||
## TiCDC 的 `gc-ttl` 和文件排序是什么? | ||
|
||
从 TiDB v4.0.0-rc.1 版本起,PD 支持外部服务设置服务级别 GC safepoint。任何一个服务可以注册更新自己服务的 GC safepoint。PD 会保证任何小于该 GC safepoint 的 KV 数据不会在 TiKV 中被 GC 清理掉。在 TiCDC 中启用了这一功能,用来保证 TiCDC 在不可用、或同步任务中断情况下,可以在 TiKV 内保留 TiCDC 需要消费的数据不被 GC 清理掉。 | ||
|
||
|
@@ -57,7 +57,7 @@ cdc cli changefeed create --pd=http://10.0.10.25:2379 --start-ts=415238226621235 | |
> | ||
> TiCDC(4.0 发布版本)还不支持动态修改文件排序和内存排序。 | ||
|
||
## 创建同步任务或同步到 MySQL 时遇到 `Error 1298: Unknown or incorrect time zone: 'UTC'` 错误 | ||
## 如何处理 TiCDC 创建同步任务或同步到 MySQL 时遇到 `Error 1298: Unknown or incorrect time zone: 'UTC'` 错误? | ||
|
||
这是因为下游 MySQL 没有加载时区,可以通过 [mysql_tzinfo_to_sql](https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html) 命令加载时区,加载后就可以正常创建任务或同步任务。 | ||
|
||
|
@@ -110,6 +110,78 @@ cdc cli changefeed create --sink-uri="mysql://root@127.0.0.1:3306/" --tz=Asia/Sh | |
> - 没有 `--tz` 参数,会尝试读取 `TZ` 环境变量设置的时区 | ||
> - 如果还没有 `TZ` 环境变量,会从 TiCDC server 运行机器的默认时区 | ||
|
||
## 升级 TiCDC 后配置文件不兼容 | ||
## 如何处理升级 TiCDC 后配置文件不兼容的问题? | ||
|
||
请参阅[配置文件兼容注意事项](/ticdc/manage-ticdc.md#配置文件兼容性的注意事项)。 | ||
|
||
## TiCDC 是否支持输出 Canal 格式的变更数据? | ||
|
||
支持。要开启 Canal 格式输出,只需在 --sink-uri 中指定 protocol 为 cancal 即可,比如: | ||
|
||
{{< copyable "shell-regular" >}} | ||
|
||
```shell | ||
cdc cli changefeed create --pd=http://10.0.10.25:2379 --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&protocol=canal" --config changefeed.toml | ||
``` | ||
|
||
> **注意:** | ||
> | ||
> * 该功能在 TiCDC 4.0.2 版本引入。 | ||
> * 目前 TiCDC 仅支持将 Canal 格式的变更数据输出到 Kafka。 | ||
|
||
更多信息请参考[创建同步任务](/ticdc/manage-ticdc.md#创建同步任务)。 | ||
|
||
## 如何查看 TiCDC 同步任务的延时? | ||
|
||
可以使用 cdc cli 查询同步任务的延时。比如: | ||
|
||
{{< copyable "shell-regular" >}} | ||
|
||
```shell | ||
cdc cli changefeed statistics --pd=http://10.0.10.25:2379 --changefeed-id 28c43ffc-2316-4f4f-a70b-d1a7c59ba79f | ||
``` | ||
|
||
上述命令的输出中: | ||
|
||
* `replication_gap` 是 TiCDC 已经同步到下游的数据于上游的数据差了多少时间。 | ||
* `sink_gap` 是 TiCDC 从上游获取的数据和最新的数据差了多少时间。 | ||
|
||
## 如何查看 TiCDC 同步任务是否正常? | ||
|
||
可以使用 cdc cli 查询同步任务的状态。比如: | ||
|
||
{{< copyable "shell-regular" >}} | ||
|
||
```shell | ||
cdc cli changefeed query --pd=http://10.0.10.25:2379 --changefeed-id 28c43ffc-2316-4f4f-a70b-d1a7c59ba79f | ||
``` | ||
|
||
上述命令的输出中 `admin-job-type` 标志这个同步的任务的状态: | ||
|
||
* `0`: 状态正常。 | ||
* `1`: 任务暂停,停止任务后所有同步 `processor` 会结束退出,同步任务的配置和同步状态都会保留,可以从 `checkpoint-ts` 恢复任务。 | ||
* `2`: 任务恢复,同步任务从 `checkpoint-ts` 继续同步。 | ||
* `3`: 任务已删除,接口请求后会结束所有同步 `processor`,并清理同步任务配置信息。同步状态保留,只提供查询,没有其他实际功能。 | ||
|
||
## 为什么 TiCDC 到 Kafka 的同步任务延时越来越大? | ||
|
||
* 请参考 [如何查看 TiCDC 同步任务是否正常?](/ticdc/troubleshoot-ticdc.md##如何查看-ticdc-同步任务是否正常) 检查下同步任务的状态是否正常。 | ||
TomShawn marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* 请适当调整 Kafka 的以下参数: | ||
* `message.max.bytes`,将 Kafka 的 server.properties 中该参数调大到 33554432 (32 MB)。 | ||
* `replica.fetch.max.bytes`,将 Kafka 的 server.properties 中该参数调大到 33554432 (32 MB)。 | ||
* `fetch.message.max.bytes`,适当调大 comsumer.properties 中该参数,确保大于 `message.max.bytes`。 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 这 3 个值都建议大一点,可以调到 1G (1073741824) |
||
|
||
## 如何查看 TiCDC Open protocol 输出变更数据中的时间戳、表名和库名? | ||
|
||
这些信息包含在 Kafka 消息的 Key 中,比如: | ||
|
||
```json | ||
{ | ||
"ts":<TS>, | ||
"scm":<Schema Name>, | ||
"tbl":<Table Name>, | ||
"t":1 | ||
} | ||
``` | ||
|
||
更多信息请参考[Open protocol Event 格式定义](/ticdc/ticdc-open-protocol.md#event-格式定义) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不推荐用户看这个状态,建议使用最新的 cdc cli changefeed list / cdc cli changefeed query --simple 查看
ref: https://github.com/pingcap/ticdc/pull/721
ps: v4.0.3 引入