Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdc: add description about claim-check-raw-value #18184

Merged
merged 10 commits into from
Oct 15, 2024
23 changes: 22 additions & 1 deletion ticdc/ticdc-sink-to-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -535,4 +535,25 @@ Kafka 消费者会收到一条含有大消息在外部存储服务中的地址
}
```

`key` 和 `value` 分别存有编码后的大消息,该消息原本应该发送到 Kafka 消息中的对应字段。消费者可以通过解析这两部分的数据,还原大消息的内容。
`key` 和 `value` 分别对应 Kafka 消息中的同名字段。消费者可以通过解析这两部分的数据,还原大消息的内容。只有 Open Protocol 编码的 Kafka 消息包含 `key` 字段,TiCDC 将 `key` 和 `value` 编码到同一个 JSON 对象中,一次性发送完整的消息。对于其他协议,`key` 字段始终为空。
Oreoxmt marked this conversation as resolved.
Show resolved Hide resolved
3AceShowHand marked this conversation as resolved.
Show resolved Hide resolved

### 只发送 `value` 部分到外部存储
Oreoxmt marked this conversation as resolved.
Show resolved Hide resolved

从 v8.4.0 开始,TiCDC 支持仅将 Kafka 消息的 `value` 部分发送到外部存储,该功能仅适用于非 Open Protocol 协议。你可以通过设置 `claim-check-raw-value` 参数控制是否开启该功能,该参数默认值为 `false`。

> **注意:**
>
> 当使用 Open Protocol 协议时,将 `claim-check-raw-value` 参数设置为 `true` 会报错。

当 `claim-check-raw-value` 设置为 `true` 时,changefeed 直接将 Kafka 消息的 `value` 部分发送到外部存储,而无需对 `key` 和 `value` 进行额外的 JSON 序列化。这样可以降低 CPU 开销。此外,消费端可以直接从外部存储读取可直接消费的数据,减少了反序列化过程的开销。
3AceShowHand marked this conversation as resolved.
Show resolved Hide resolved

配置样例如下所示:

```toml
protocol = "simple"

[sink.kafka-config.large-message-handle]
large-message-handle-option = "claim-check"
claim-check-storage-uri = "s3://claim-check-bucket"
claim-check-raw-value = true
```