Skip to content

Commit

Permalink
[Feature] support avro format (apache#5084)
Browse files Browse the repository at this point in the history
  • Loading branch information
@liunaijie
liunaijie authored and chaorongzhi committed Aug 21, 2024
1 parent 3279b95 commit 40914f0
Show file tree
Hide file tree
Showing 24 changed files with 1,296 additions and 46 deletions.
26 changes: 13 additions & 13 deletions docs/en/connector-v2/sink/Kafka.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,19 @@ They can be downloaded via install-plugin.sh or from the Maven central repositor

## Sink Options

| Name | Type | Required | Default | Description |
|----------------------|--------|----------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String | Yes | - | When the table is used as sink, the topic name is the topic to write data to. |
| bootstrap.servers | String | Yes | - | Comma separated list of Kafka brokers. |
| kafka.config | Map | No | - | In addition to the above parameters that must be specified by the `Kafka producer` client, the user can also specify multiple non-mandatory parameters for the `producer` client, covering [all the producer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#producerconfigs). |
| semantics | String | No | NON | Semantics that can be chosen EXACTLY_ONCE/AT_LEAST_ONCE/NON, default NON. |
| partition_key_fields | Array | No | - | Configure which fields are used as the key of the kafka message. |
| partition | Int | No | - | We can specify the partition, all messages will be sent to this partition. |
| assign_partitions | Array | No | - | We can decide which partition to send based on the content of the message. The function of this parameter is to distribute information. |
| transaction_prefix | String | No | - | If semantic is specified as EXACTLY_ONCE, the producer will write all messages in a Kafka transaction,kafka distinguishes different transactions by different transactionId. This parameter is prefix of kafka transactionId, make sure different job use different prefix. |
| format | String | No | json | Data format. The default format is json. Optional text format, canal-json and debezium-json.If you use json or text format. The default field separator is ", ". If you customize the delimiter, add the "field_delimiter" option.If you use canal format, please refer to [canal-json](../formats/canal-json.md) for details.If you use debezium format, please refer to [debezium-json](../formats/debezium-json.md) for details. |
| field_delimiter | String | No | , | Customize the field delimiter for data format. |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |
| Name | Type | Required | Default | Description |
|----------------------|--------|----------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| topic | String | Yes | - | When the table is used as sink, the topic name is the topic to write data to. |
| bootstrap.servers | String | Yes | - | Comma separated list of Kafka brokers. |
| kafka.config | Map | No | - | In addition to the above parameters that must be specified by the `Kafka producer` client, the user can also specify multiple non-mandatory parameters for the `producer` client, covering [all the producer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#producerconfigs). |
| semantics | String | No | NON | Semantics that can be chosen EXACTLY_ONCE/AT_LEAST_ONCE/NON, default NON. |
| partition_key_fields | Array | No | - | Configure which fields are used as the key of the kafka message. |
| partition | Int | No | - | We can specify the partition, all messages will be sent to this partition. |
| assign_partitions | Array | No | - | We can decide which partition to send based on the content of the message. The function of this parameter is to distribute information. |
| transaction_prefix | String | No | - | If semantic is specified as EXACTLY_ONCE, the producer will write all messages in a Kafka transaction,kafka distinguishes different transactions by different transactionId. This parameter is prefix of kafka transactionId, make sure different job use different prefix. |
| format | String | No | json | Data format. The default format is json. Optional text format, canal-json, debezium-json and avro.If you use json or text format. The default field separator is ", ". If you customize the delimiter, add the "field_delimiter" option.If you use canal format, please refer to [canal-json](../formats/canal-json.md) for details.If you use debezium format, please refer to [debezium-json](../formats/debezium-json.md) for details. |
| field_delimiter | String | No | , | Customize the field delimiter for data format. |
| common-options | | No | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |

## Parameter Interpretation

Expand Down
Loading

0 comments on commit 40914f0

Please sign in to comment.