Skip to content

Commit

Permalink
[Improve][Jdbc] Disable upsert writes by default
Browse files Browse the repository at this point in the history
  • Loading branch information
@hailin0
hailin0 committed Jun 2, 2023
1 parent e0d8519 commit 0b921f4
Show file tree
Hide file tree
Showing 10 changed files with 88 additions and 85 deletions.
51 changes: 26 additions & 25 deletions docs/en/connector-v2/sink/Jdbc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,28 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it.

## Options

| name | type | required | default value |
|-------------------------------------------|---------|----------|---------------|
| url | String | Yes | - |
| driver | String | Yes | - |
| user | String | No | - |
| password | String | No | - |
| query | String | No | - |
| database | String | No | - |
| table | String | No | - |
| primary_keys | Array | No | - |
| support_upsert_by_query_primary_key_exist | Boolean | No | false |
| connection_check_timeout_sec | Int | No | 30 |
| max_retries | Int | No | 0 |
| batch_size | Int | No | 1000 |
| batch_interval_ms | Int | No | 1000 |
| is_exactly_once | Boolean | No | false |
| generate_sink_sql | Boolean | No | false |
| xa_data_source_class_name | String | No | - |
| max_commit_attempts | Int | No | 3 |
| transaction_timeout_sec | Int | No | -1 |
| auto_commit | Boolean | No | true |
| common-options | | no | - |
| name | type | required | default value |
|------------------------------|---------|----------|---------------|
| url | String | Yes | - |
| driver | String | Yes | - |
| user | String | No | - |
| password | String | No | - |
| query | String | No | - |
| database | String | No | - |
| table | String | No | - |
| primary_keys | Array | No | - |
| enable_upsert | Boolean | No | false |
| connection_check_timeout_sec | Int | No | 30 |
| max_retries | Int | No | 0 |
| batch_size | Int | No | 1000 |
| batch_interval_ms | Int | No | 1000 |
| is_exactly_once | Boolean | No | false |
| generate_sink_sql | Boolean | No | false |
| xa_data_source_class_name | String | No | - |
| max_commit_attempts | Int | No | 3 |
| transaction_timeout_sec | Int | No | -1 |
| auto_commit | Boolean | No | true |
| common-options | | no | - |

### driver [string]

Expand Down Expand Up @@ -85,10 +85,11 @@ This option is mutually exclusive with `query` and has a higher priority.

This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql.

### support_upsert_by_query_primary_key_exist [boolean]
### enable_upsert [boolean]

Choose to use INSERT sql, UPDATE sql to process update events(INSERT, UPDATE_AFTER) based on query primary key exists. This configuration is only used when database unsupported upsert syntax.
**Note**: that this method has low performance
Choose to use INSERT/UPDATE or UPSERT sql to process update events(INSERT, UPDATE_AFTER) based on `primary_keys` exists.

**Note**: that this method has low performance on database not support UPSERT sql

### connection_check_timeout_sec [int]

Expand Down
44 changes: 22 additions & 22 deletions docs/en/connector-v2/sink/Mysql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,28 +54,28 @@ semantics (using XA transaction guarantee).

## Sink Options

| Name | Type | Required | Default | Description |
|-------------------------------------------|---------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:mysql://localhost:3306:3306/test |
| driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use MySQL the value is `com.mysql.cj.jdbc.Driver`. |
| user | String | No | - | Connection instance user name |
| password | String | No | - | Connection instance password |
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority |
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
| primary_keys | Array | No | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql. |
| support_upsert_by_query_primary_key_exist | Boolean | No | false | Choose to use INSERT sql, UPDATE sql to process update events(INSERT, UPDATE_AFTER) based on query primary key exists. This configuration is only used when database unsupport upsert syntax. **Note**: that this method has low performance |
| connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete. |
| max_retries | Int | No | 0 | The number of retries to submit failed (executeBatch) |
| batch_size | Int | No | 1000 | For batch writing, when the number of buffered records reaches the number of `batch_size` or the time reaches `batch_interval_ms`<br/>, the data will be flushed into the database |
| batch_interval_ms | Int | No | 1000 | For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the database |
| is_exactly_once | Boolean | No | false | Whether to enable exactly-once semantics, which will use Xa transactions. If on, you need to<br/>set `xa_data_source_class_name`. |
| generate_sink_sql | Boolean | No | false | Generate sql statements based on the database table you want to write to |
| xa_data_source_class_name | String | No | - | The xa data source class name of the database Driver, for example, mysql is `com.mysql.cj.jdbc.MysqlXADataSource`, and<br/>please refer to appendix for other data sources |
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |
| Name | Type | Required | Default | Description |
|------------------------------|---------|----------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:mysql://localhost:3306:3306/test |
| driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use MySQL the value is `com.mysql.cj.jdbc.Driver`. |
| user | String | No | - | Connection instance user name |
| password | String | No | - | Connection instance password |
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority |
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
| table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
| primary_keys | Array | No | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql. |
| enable_upsert | Boolean | No | false | Choose to use INSERT/UPDATE or UPSERT sql to process update events(INSERT, UPDATE_AFTER) based on `primary_keys` exists. **Note**: that this method has low performance on database not support UPSERT sql |
| connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete. |
| max_retries | Int | No | 0 | The number of retries to submit failed (executeBatch) |
| batch_size | Int | No | 1000 | For batch writing, when the number of buffered records reaches the number of `batch_size` or the time reaches `batch_interval_ms`<br/>, the data will be flushed into the database |
| batch_interval_ms | Int | No | 1000 | For batch writing, when the number of buffers reaches the number of `batch_size` or the time reaches `batch_interval_ms`, the data will be flushed into the database |
| is_exactly_once | Boolean | No | false | Whether to enable exactly-once semantics, which will use Xa transactions. If on, you need to<br/>set `xa_data_source_class_name`. |
| generate_sink_sql | Boolean | No | false | Generate sql statements based on the database table you want to write to |
| xa_data_source_class_name | String | No | - | The xa data source class name of the database Driver, for example, mysql is `com.mysql.cj.jdbc.MysqlXADataSource`, and<br/>please refer to appendix for other data sources |
| max_commit_attempts | Int | No | 3 | The number of retries for transaction commit failures |
| transaction_timeout_sec | Int | No | -1 | The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect<br/>exactly-once semantics |
| auto_commit | Boolean | No | true | Automatic transaction commit is enabled by default |
| common-options | | no | - | Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details |

### Tips

Expand Down
Loading

0 comments on commit 0b921f4

Please sign in to comment.