[API-DRAFT] [MERGE] Merge api-draft to dev branch

.github/workflows/backend.yml

@@ -88,7 +88,7 @@ jobs:
         java: [ '8', '11' ]
         os: [ 'ubuntu-latest', 'windows-latest' ]
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 30
+    timeout-minutes: 50
     steps:
       - uses: actions/checkout@v3
         with:

.licenserc.yaml

@@ -28,7 +28,7 @@ header:
     - mvnw.cmd
     - .mvn
     - .gitattributes
-    - '**/known-dependencies.txt'
+    - '**/known-dependencies-*.txt'
     - '**/*.md'
     - '**/*.mdx'
     - '**/*.json'

LICENSE

@@ -209,6 +209,7 @@ The text of each license is the standard Apache 2.0 license.
 
 tools/dependencies/checkLicense.sh files from https://github.com/apache/skywalking
 mvnw files from https://github.com/apache/maven-wrapper Apache 2.0
+seatunnel-api/src/main/java/org/apache/seatunnel/api/table/type/RowKind.java from  https://github.com/apache/flink
 seatunnel-config/seatunnel-config-shade/src/main/java/org/apache/seatunnel/shade/com/typesafe/config/impl/ConfigNodePath.java    from  https://github.com/lightbend/config
 seatunnel-config/seatunnel-config-shade/src/main/java/org/apache/seatunnel/shade/com/typesafe/config/impl/ConfigParser.java      from  https://github.com/lightbend/config
 seatunnel-config/seatunnel-config-shade/src/main/java/org/apache/seatunnel/shade/com/typesafe/config/impl/Path.java              from  https://github.com/lightbend/config

config/flink.batch.conf.template

@@ -26,8 +26,10 @@ env {
 
 source {
   # This is a example input plugin **only for test and demonstrate the feature input plugin**
+
   FakeSource {
     result_table_name = "test"
+    field_name = "name,age"
   }
 
   # If you would like to get more information about how to configure seatunnel and see full list of input plugins,

config/spark.streaming.conf.template

@@ -31,7 +31,7 @@ env {
 
 source {
   # This is a example input plugin **only for test and demonstrate the feature input plugin**
-  fakeStream {
+  FakeStream {
     content = ["Hello World, SeaTunnel"]
   }
 

docs/en/connector/sink/Assert.md

@@ -10,51 +10,51 @@ A flink sink plugin which can assert illegal data by user defined rules
 
 Engine Supported and plugin name
 
-* [ ] Spark
-* [x] Flink: AssertSink
+* [x] Spark：Assert
+* [x] Flink: Assert
 
 :::
 
 ## Options
 
-| name                    | type     | required | default value |
-| ----------------------- | -------- | -------- | ------------- |
-| rules         		  | ConfigList | yes      | -             |
-|  field_name        | `String` | yes       | -     |
-|  field_type        | `String` | no       | -          |
-|  field_value | ConfigList | no       | -             |
-|   rule_type         | `String`    | no       | -             |
-|   rule_value         | double    | no       | -             |
+| name                          | type        | required | default value |
+| ----------------------------- | ----------  | -------- | ------------- |
+|rules                          | ConfigList  | yes      | -             |
+|rules.field_name               | string      | yes      | -             |
+|rules.field_type               | string      | no       | -             |
+|rules.field_value              | ConfigList  | no       | -             |
+|rules.field_value.rule_type    | string      | no       | -             |
+|rules.field_value.rule_value   | double      | no       | -             |
 
 
-### rules
+### rules [ConfigList]
 
 Rule definition of user's available data.  Each rule represents one field validation.
 
-### field_name
+### field_name [string]
 
 field name（string）
 
-### field_type
+### field_type [string]
 
 field type (string),  e.g. `string,boolean,byte,short,int,long,float,double,char,void,BigInteger,BigDecimal,Instant`
 
-### field_value
+### field_value [ConfigList]
 
 A list value rule define the data value validation
 
-### rule_type
+### rule_type [string]
 
 The following rules are supported for now
 `
-NOT_NULL,  
-MIN,  
-MAX,  
-MIN_LENGTH,  
-MAX_LENGTH
+NOT_NULL,   // value can't be null
+MIN,        // define the minimum value of data
+MAX,        // define the maximum value of data
+MIN_LENGTH, // define the minimum string length of a string data
+MAX_LENGTH  // define the maximum string length of a string data
 `
 
-### rule_value
+### rule_value [double]
 
 the value related to rule type
 
@@ -63,8 +63,9 @@ the value related to rule type
 the whole config obey with `hocon` style
 
 ```hocon
-AssertSink {
-    rules = 
+
+Assert {
+   rules = 
         [{
             field_name = name
             field_type = string

docs/en/new-connector/sink/Assert.md

@@ -0,0 +1,94 @@
+# Assert
+
+## Description
+
+A flink sink plugin which can assert illegal data by user defined rules
+
+## Options
+
+| name                          | type        | required | default value |
+| ----------------------------- | ----------  | -------- | ------------- |
+|rules                          | ConfigList  | yes      | -             |
+|rules.field_name               | string      | yes      | -             |
+|rules.field_type               | string      | no       | -             |
+|rules.field_value              | ConfigList  | no       | -             |
+|rules.field_value.rule_type    | string      | no       | -             |
+|rules.field_value.rule_value   | double      | no       | -             |
+
+
+### rules [ConfigList]
+
+Rule definition of user's available data.  Each rule represents one field validation.
+
+### field_name [string]
+
+field name（string）
+
+### field_type [string]
+
+field type (string),  e.g. `string,boolean,byte,short,int,long,float,double,char,void,BigInteger,BigDecimal,Instant`
+
+### field_value [ConfigList]
+
+A list value rule define the data value validation
+
+### rule_type [string]
+
+The following rules are supported for now
+`
+NOT_NULL,   // value can't be null
+MIN,        // define the minimum value of data
+MAX,        // define the maximum value of data
+MIN_LENGTH, // define the minimum string length of a string data
+MAX_LENGTH  // define the maximum string length of a string data
+`
+
+### rule_value [double]
+
+the value related to rule type
+
+
+## Example
+the whole config obey with `hocon` style
+
+```hocon
+Assert {
+    rules = 
+        [{
+            field_name = name
+            field_type = string
+            field_value = [
+                {
+                    rule_type = NOT_NULL
+                },
+                {
+                    rule_type = MIN_LENGTH
+                    rule_value = 3
+                },
+                {
+                     rule_type = MAX_LENGTH
+                     rule_value = 5
+                }
+            ]
+        },{
+            field_name = age
+            field_type = int
+            field_value = [
+                {
+                    rule_type = NOT_NULL
+                },
+                {
+                    rule_type = MIN
+                    rule_value = 10
+                },
+                {
+                     rule_type = MAX
+                     rule_value = 20
+                }
+            ]
+        }
+        ]
+
+}
+
+```

docs/en/new-connector/sink/Clickhouse.md

@@ -0,0 +1,104 @@
+# Clickhouse
+
+## Description
+
+Used to write data to Clickhouse. Supports Batch and Streaming mode.
+
+:::tip
+
+Write data to Clickhouse can also be done using JDBC
+
+:::
+
+## Options
+
+| name           | type   | required | default value |
+|----------------|--------|----------|---------------|
+| host           | string | yes      | -             |
+| database       | string | yes      | -             |
+| table          | string | yes      | -             |
+| username       | string | yes      | -             |
+| password       | string | yes      | -             |
+| clickhouse.*   | string | no       |               |
+| bulk_size      | string | no       | 20000         |
+| split_mode     | string | no       | false         |
+| sharding_key   | string | no       | -             |
+| common-options | string | no       | -             |
+
+### host [string]
+
+`ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"` .
+
+### database [string]
+
+The `ClickHouse` database
+
+### table [string]
+
+The table name
+
+### username [string]
+
+`ClickHouse` user username
+
+### password [string]
+
+`ClickHouse` user password
+
+### clickhouse [string]
+
+In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc` .
+
+The way to specify the parameter is to add the prefix `clickhouse.` to the original parameter name. For example, the way to specify `socket_timeout` is: `clickhouse.socket_timeout = 50000` . If these non-essential parameters are not specified, they will use the default values given by `clickhouse-jdbc`.
+
+### bulk_size [number]
+
+The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000` .
+
+### split_mode [boolean]
+
+This mode only support clickhouse table which engine is 'Distributed'.And `internal_replication` option
+should be `true`. They will split distributed table data in seatunnel and perform write directly on each shard. The shard weight define is clickhouse will be
+counted.
+
+### sharding_key [string]
+
+When use split_mode, which node to send data to is a problem, the default is random selection, but the
+'sharding_key' parameter can be used to specify the field for the sharding algorithm. This option only
+worked when 'split_mode' is true.
+
+### common options [string]
+
+Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
+
+## Examples
+
+```hocon
+sink {
+
+  Clickhouse {
+    host = "localhost:8123"
+    database = "default"
+    table = "fake_all"
+    username = "default"
+    password = ""
+    split_mode = true
+    sharding_key = "age"
+  }
+
+}
+```
+
+```hocon
+sink {
+
+  Clickhouse {
+    host = "localhost:8123"
+    database = "default"
+    table = "fake_all"
+    username = "default"
+    password = ""
+  }
+
+}
+```