[Feature][Zeta][REST-API]Add REST API To Submit Job #5107

zhibinF · 2023-07-18T06:59:04Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

Description

Add RestHttpPostCommandProcessor to handle post request.
Add JobImmutableInformationEnv to create jobEnv.
Add RestUtil to handle request body and request parameters.
Add e2e test.

Detail

In order to avoid the java.lang.NoClassDefFoundEror: com/sun/jersey/client/impl/Copy0nWriteHashMap, change the pipelineActions from CopyOnWriteHashMap to ConcurrentHashMap

liugddx

Please add doc for this feature. refer to rest-api.md.

liugddx · 2023-07-18T07:08:11Z

Run mvn spotless:apply before submit pr.

zhibinF · 2023-07-18T07:10:12Z

Please add doc for this feature. refer to rest-api.md.

Ok, I will provide doc.

liugddx · 2023-07-18T07:11:22Z

...ngine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointPlan.java

@@ -63,7 +64,7 @@ public class CheckpointPlan {
    public static final class Builder {
        private final Set<TaskLocation> pipelineSubtasks = new CopyOnWriteArraySet<>();
        private final Set<TaskLocation> startingSubtasks = new CopyOnWriteArraySet<>();
-        private final Map<ActionStateKey, Integer> pipelineActions = new CopyOnWriteHashMap<>();


liugddx · 2023-07-19T10:53:20Z

Test failed, please deal with it.

zhibinF · 2023-07-19T12:07:06Z

Test failed, please deal with it.

ok

liugddx

PTAL @MonsterChenzhuo

liugddx · 2023-07-31T02:21:30Z

@hailin0 @Hisoka-X @EricJoy2048 PTAL

ic4y

Great commit.

ic4y · 2023-08-02T10:15:42Z

...-server/src/main/java/org/apache/seatunnel/engine/server/job/JobImmutableInformationEnv.java

This class has too much code duplication with JobExecutionEnvironment, it needs to be refactored for maintainability.

Yes, the main reference for this class is JobExecutionEnvironment, but since JobExecutionEnvironment is under engine.client and the methods are private, I can't call them directly.
Or should I refactor two JobExecutionEnvironment and JobImmutableInformationEnv under the seatunnel.engine.core.job？

@Hisoka-X @EricJoy2048 What's your opinion?

Agree with you.

I have doubts about the class name. Why AbstractJobEnvironment implement class named JobImmutableInformationEnv and JobExecutionEnvironment? What's difference between these class? And why one class named with xxxEnvironment, and another named with xxxInformationEnv? I can't understand that these two classes inherit from the same abstract class. @zhibinF Could you add some comment for these class and change the name? cc @liugddx

JobImmutableInformationEnv is used for seatunnl reatAPI build tasks running environment, at the same time return JobImmutableInformation, So I use JobImmutableInformationEnv as the class name, and JobExecutionEnvironment is used for seatunnl client submit tasks, build tasks running environment, already exist.
Because JobImmutableInformationEnv reference JobExecutionEnvironment achieve originally, follow-up found that there are a lot of redundant code, thus abstracts the AbstractJobEnvironment to reduce redundant code. And as the following different end to create the task environment of the abstract class.
I don't know if that solves your problem.

Thanks for explain, I changed the name for this, please help to review. Thanks. #5671

EricJoy2048 · 2023-08-04T08:46:17Z

...-server/src/main/java/org/apache/seatunnel/engine/server/job/JobImmutableInformationEnv.java

+    @Override
+    protected MultipleTableJobConfigParser getJobConfigParser() {
+        return new MultipleTableJobConfigParser(
+                seaTunnelJobConfig, idGenerator, jobConfig, commonPluginJars);


Why lost the isStartWithSavePoint ?

sry, I'll make changes.

…nv and JobExecutionEnvironment

liugddx · 2023-08-07T08:05:28Z

Please merge the latest dev branch.

liugddx

+1.LGTM

* [Feature][Connector V2] expose configurable options in Cassandra (apache#3681) * [Connector-V2][Paimon] Introduce paimon connector (apache#4178) * [Improve][Zeta] Improve Zeta operation max count and ignore NPE (apache#4787) * [Improve][Zeta] Improve Zeta operation max count and ignore NPE * [Improve][Zeta] Improve Zeta operation max count and ignore NPE * [Improve][Zeta] Cancel pipeline add retry to avoid cancel failed. (apache#4792) * [Hotfix][CDC] Fix chunk start/end parameter type error (apache#4777) Incorrect wrapping as Array<Array> types, but only Array type required * [Feature][Zeta] Add OSS support for Imap storage to cluster-mode type (apache#4683) * Add OSS/S3 to cluster-mode type apache#4621 * fixed bug & add e2e test * Wait for the node to start before scheduling & Move jar to parent pom & optimize writer * update LICENSE * [Hotfix][CI] Fix error repository name in ci config files (apache#4795) * [Feature][Json-format] support read format for pulsar (apache#4111) * [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql par… (apache#4797) * [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql parameter for the jdbc sinj document * [Docs][Connector-V2][Mysql] fix Mysql sink format doc (apache#4800) * [Hotfix][Connector][Jdbc] Fix sqlserver system table case sensitivity (apache#4806) * [Hotfix][Connector][Jdbc] Fix reconnect throw close statement exception (apache#4801) * [Hotfix][Connector-V2][Jdbc] Fix the error of extracting primary key column in sink (apache#4815) * [Feature][Connector-v2] Add Snowflake Source&Sink connector (apache#4470) --------- Co-authored-by: Eric <gaojun2048@gmail.com> Co-authored-by: hailin0 <wanghailin@apache.org> * [Hotfix][CI] Fix redundant modules run e2e tests when change jdbc module (apache#4824) * fix pom.xml code style (apache#4836) * [Chore] Format the.conf file using the same style (apache#4830) * [Hotfix][Zeta] Fix cpu load problem (apache#4828) * [Improve][Zeta] Reduce the number of IMAPs used by checkpointIdCounter (apache#4832) * [Bugfix][connector-v2][rabbitmq] Fix reduplicate ack msg bug and code style (apache#4842) --------- Co-authored-by: 毕博 <bibo@mafengwo.com> * [Improve][Zeta] async execute checkpoint trigger and other block method (apache#4846) * [Improve][Zeta] async execute checkpoint trigger * [Bug][Zeta] Fix zeta cannot normally recycle thread belong to abnormal tasks * [Improve][Zeta] Move `restoreState` add `addSplitsBack` execute by TaskExecuteService * [Improve][Zeta] Move `receivedReader` execute by TaskExecuteService * [Bug][Zeta] Fix task `notifyTaskStatusToMaster` failed when job not running or failed before run (apache#4847) * [Bug][Zeta] Fix task repeat notify failed when job not running * [Bug][Zeta] Fix notifyTaskStatusToMaster not release lock and NPE * [Improve][Zeta] Reduce the frequency of fetching data from imap (apache#4851) * [Improve][Zeta] Add Metaspace size default value to config file (apache#4848) * [Improve][Zeta] Speed up listAllJob function (apache#4852) * [Bug][Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (apache#4849) * [Improve][Zeta] Fix engine runtime error (apache#4850) * [Hotfix][Zeta] Fix completePendingCheckpoint concurrent action (apache#4854) This operation does not allow concurrent execution * [Hotfix][Zeta] Fix master active bug (apache#4855) * [Bugfix][DAG] Fix the incorrect setting of transform parallelism (apache#4814) * [Hotfix][Zeta] fix pipeline state not right bug (apache#4823) * [BUG][Doris] Add a jobId to the doris label to distinguish between tasks (apache#4853) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Improve] Add a jobId to the doris label to distinguish between tasks (apache#4839) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Hotfix][Zeta] Fix IMap operation timeout bug (apache#4859) * [Bug][Zeta] Fix restoreComplete Future can't be completed when cancel task (apache#4863) * [Feature][SQL Transform]Add catalog support for SQL Transform plugin (apache#4819) * [improve][SelectDB] Add a jobId to the selectDB label to distinguish between tasks (apache#4864) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Hotfix][Connector-v2][kafka] Fix the short interval of pull data settings and revise the format (apache#4875) * [Bug][Connector-V2][Doris] update last checkpoint id when doing snapshot (apache#4881) * [Hotfix][Zeta] Fix deploy operation timeout but task already finished bug (apache#4867) * [Core][Docs]Remove incubator in README file (apache#4882) * [Bugfix][CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (apache#4877) * [improve][CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (apache#4856) * [Improve][Zeta] Reduce the operation count of imap_running_job_metrics (apache#4861) * [Bug][Zeta] Fix TaskExecutionService will return not active ExecutionContext (apache#4869) * [Hotfix][Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (apache#4866) * [Bugfix] [Connector-V2] [File] Fix read temp file (apache#4876) Co-authored-by: wantao <wantao@inmyshow.com> * [Bug][Zeta] Fix TaskExecutionService synchronized lock will not release (apache#4886) * [Improve][Zeta] Move driver into lib directory and change operation count (apache#4845) * [hotfix][kafka] Fix the problem that the partition information cannot be obtained when kafka is restored (apache#4764) * [Bugfix][zeta] Fix the deadlock issue with JDBC driver loading (apache#4878) * [Chore] update 2.3.2 release-note.md (apache#4892) * [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys (apache#4634) * [Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column apache#4634 Co-authored-by: zhilinli <lzl15844876351@163.com> * [Bug][connector-v2][doris] add streamload Content-type for doris URLdecode error (apache#4880) * [Chore] Change repository name from incubator-seatunnel to seatunnel (apache#4868) --------- Co-authored-by: Jia Fan <fanjiaeminem@qq.com> * [Improve][connector-V2-Neo4j]Supports neo4j sink batch write and update docs (apache#4841) * [Hotfix][connector-v2][e2e] Fix maven scope (apache#4901) * quick-start-seatunnel-engine.md (apache#4943) * fix error (apache#4888) * [Hotfix][Connector-V2][ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (apache#4937) * Update ClickhouseFileSinkWriter.java Bug fix: When ClikchouseFileSinkerWriter writes to a temporary file, it does not check whether the field value is empty, so an exception will be thrown. Modified to write an empty string when a null value is encountered * Update ClickhouseFileSinkWriter.java repair code style * Update ClickhouseFileSinkWriter.java code style * [Improve][Zeta] Add an interface for batch retrieval of JobMetrics (apache#4576) * [Improve] Documentation and partial word optimization. (apache#4936) * code format * add cdc feature * fix cdc can not get driver error --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [Doc][Connector-V2] StarRocks `nodeUrls` property name fix (apache#4951) node_urls -> nodeUrls node_urls doesn't work * [Feature][E2E][FtpFile] add ftp file e2e test case (apache#4647) * [WIP][Feature][Connector-e2e] add ftp e2e test * Let e2e barely execute by excluding the commons-net jar package. * Resolve the maven conflict --------- Co-authored-by: hailin0 <wanghailin@apache.org> * [Hotfix][Connector-V2][StarRocks] Fix code style (apache#4966) * [Hotfix][Connector-v2][HbaseSink]Fix default timestamp (apache#4958) * [Doc]Change the transform website url (apache#4954) * [Docs][Connector-V2][Http]Reconstruct the Http connector document (apache#4962) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Feature][connector-v2][mongodb] mongodb support cdc sink (apache#4833) * [Bug] [zeta][starter]fix bug (apache#4983) (apache#4984) Co-authored-by: wsstony <tonymao777@163.com> * fix redis nodes format error. (apache#4981) Co-authored-by: lightzhao <zhaolianyong777@gmail.com> * [Improve][CDC]Remove driver for cdc connector (apache#4952) * [Hotfix][Connector-V2][Mongodb] Fix document error content and remove redundant code (apache#4982) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Improve][Connector-V2][OSS-Jindo] Optimize jindo oss connector (apache#4964) * [Improve][Connector-V2][Jindo-Oss] Optimize jindo-oss connector * [Improve][Connector-V2][Jindo-Oss] Update module name * [Hotfix][Connector-V2][StarRocks] Fix code style * [bugfix] Upgrade the key log output level(apache#4993) * [Feature][Zeta] Configuration files support user variable replacement (apache#4969) * [Feature][Transform-V2][SQL] Support 'select *' and 'like' clause for SQL Transform plugin (apache#4991) Co-authored-by: mcy <rewrma@163.com> * [Improve][CDC]change driver scope to provider (apache#5002) * [Hotfix][Connector-V2][Hive] Support user-defined hive-site.xml (apache#4965) * [Improve][Connector-v2][Mongodb]Optimize reading logic (apache#5001) Co-authored-by: chenqqq11 <chenzy15@ziroom.com> * [Feature][Connector-V2][Clickhouse] clickhouse writes with checkpoints (apache#4999) * [Hotfix][Connector-V2][Mongodb] Compatible with historical parameters (apache#4997) * Split updated modules integration test for part 4 (apache#5028) * [Hotfix] Fix the CI Job name error (apache#5032) * [Feature][CDC] Support disable/enable exactly once for INITIAL (apache#4921) * [bugfix][zeta] Fixed multi-table job data loss and latency issues (apache#149) (apache#5031) * [Hotfix][CDC] Fix jdbc connection leak for mysql (apache#5037) * [Bugfix][zeta] Fix cdc connection does not close (apache#4922) * Fix XA Transaction bug (apache#5020) * Set Up with Kubernetes, dockerfile document error in constructing docker image (apache#5022) Co-authored-by: yctan <1417983443@qq.com> * [Improve][Connector-v2][Mongodb]sink support transaction update/writing (apache#5034) * fix:the HdfsStorage can not delete checkpoint file apache#5046 (apache#5054) * [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (apache#5033) (apache#5035) * [bugfix] change MySQL CDC serverId from int to long (apache#5033) * style: 🎨 optimize code style * [Feature][Connector-V2][cdc] Change the time zone to the default time zone (apache#5030) * [Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient reuse (apache#5011) * [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (apache#4989) --------- Co-authored-by: silenceland <silenceland23@163.com> Co-authored-by: changhuyan <877018069@qq.com> * [HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file cannot be generated apache#4985 (apache#5051) * fix after the savepoint job is restored, the checkpoint file cannot be generated * fix class not found exception (apache#5063) * [Feature] update action config to support run CI on fork repo (apache#5065) * [Bugfix]fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (apache#5080) * [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone (apache#5078) * Add clickhouse connector time zone key,default system time zone * Modify the document and add clickhouse server_time_zone configuration * [Chore] Modify repeat des (apache#5088) Co-authored-by: 80597928 <Lzl@qq.com> * [Docs] Add Value types in Java to Schema feature (apache#5087) * [Feature][Connector-V2] JDBC source support string type as partition key (apache#4947) * [HotFix] Fix code style (apache#5092) * [Docs][Zeta] Add savepoint doc (apache#5081) * [Feature][connector-v2][mongodbcdc]Support source mongodb cdc (apache#4923) * [Improve] Improve savemode api (apache#4767) * [Doc] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (apache#5102) * [Improve][Docs][Clickhouse] Reconstruct the clickhouse connector doc (apache#5085) --------- Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Pom]update version to 2.3.3-SNAPSHOT (apache#5043) * update version to 2.3.3-SNAPSHOT * update dependency version in know dependencies file * Add logs to find job restore from master active switch error * [Feature][Connector-V2][mysql cdc] Conversion of tinyint(1) to bool is supported (apache#5105) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Improve][Zeta] Add sleep for Task to reduce CPU cost (apache#5117) * [Feature][JDBC Sink] Add DM upsert support (apache#5073) --------- Co-authored-by: David Zollo <davidzollo365@gmail.com> * [Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak (apache#4802) [Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak * [Hotfix]Fix mongodb cdc e2e instability (apache#5128) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Hotfix][Zeta] Fix task state memory leak (apache#5139) * [Hotfix][Zeta] Fix checkpoint error report without msg (apache#5137) * [Improve][Zeta] Improve CheckpointCoordinator notify complete when restore (apache#5136) * [Improve] Improve CheckpointCoordinator notify complete when restore * update * [Improve][Zeta] Improve CheckpointCoordinator log error when report error from task (apache#178) (apache#5134) * [Hotfix][Zeta] Fix MultipleTableJobConfigParser ignore env option (apache#5067) * [Fix][Zeta] Fix MultipleTableJobConfigParser ignore env option * update * [Improve][Connector[File] Optimize files commit order (apache#5045) Before using `HashMap` store files path, so every checkpoint file commit is out of order. Now switch to using `LinkedHashMap` to ensure that files are commit in the generated order * [Hotfix][Mongodb cdc] Solve startup resume token is negative (apache#5143) --------- Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Feature][connector][kafka] Support read debezium format message from kafka (apache#5066) * [Feature][CDC] Support tables without primary keys (with unique keys) (apache#163) (apache#5150) * [Feature][Connector-V2][CDC] Support string type shard fields. (apache#5147) * [feature][CDC base] Supports string type shard fields * Delete invalid code * [Feature][Connector-V2][File] Add cos source&sink (apache#4979) * [Feature][Connector-V2][File] Add cos sink * update doc&e2e and add pom file header * add e2e file header and config * add file-cos module into dist pom.xml * [Feature][Connector-V2][File] Add cos source --------- Co-authored-by: dengd1937 <dengd1803@gmail.com> * [Fix][Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (apache#5152) * [Hotfix][MongodbCDC]Refine data format to adapt to universal logic (apache#5162) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Chore] Update bug-report.yml (apache#5160) * [Improve][CDC] support exactly-once of cdc and fix the BinlogOffset comparing bug (apache#5057) * [Improve][CDC] support exactly-once of cdc, fix the BinlogOffset comparing bug * [Improve][CDC] adjust code style * [Improve][CDC] fix ci error --------- Co-authored-by: happyboy1024 <296442618@qq.com> * [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document (apache#4905) * [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document --------- Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Docs][Connector-V2][Doris] Reconstruct the Doris connector document (apache#4903) * [Docs][Connector-V2][Doris] Reconstruct the Doris connector document --------- Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [improve] [CDC Base] Add some split parameters to the optionRule (apache#5161) * [bugfix] [File Base] Fix Hadoop Kerberos authentication related issues. (apache#5171) * [CI] add code style check when docs changed (apache#5183) * [Bug][Translation][Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type. (apache#5170) * [Improve][Zeta] Move checkpoint notify complete in checkpoint stage (apache#5185) * [Feature][Catalog] Add JDBC Catalog auto create table (apache#4917) * [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. (apache#5153) * [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. * [Improve][Connector-v2][Jdbc] check url not null throw friendly message (apache#5097) * check url not null throw friendly message * check jdbc source config * modify jdbc validate method --------- Co-authored-by: 80597928 <Lzl@qq.com> Co-authored-by: 80597928 <673421862@qq.com> * [bugfix][zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (apache#5191) * [Improve][Docs][Kafka]Reconstruct the kafka connector document (apache#4778) * [Docs][Connector-V2][Kafka]Reconstruct the kafka connector document --------- Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Bug][Improve][LocalFileSink]Fix LocalFile Sink file_format_type. (apache#5118) * [Bug] [connector-v2] PostgreSQL versions below 9.5 are compatible use cdc sync problem (apache#5120) * [e2e] kafka e2e error (apache#5200) * [Hotfix][Connector-V2][JindoOssFile] Fix plugin-mapping.properties (apache#5215) Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com> * [Improve][Zeta] Don't trigger handleSaveMode when restore (apache#5192) * move imap storage file dependency packages to submodules (apache#5218) * [Hotfix][CI]Declare files that will always have UNIX line endings on checkout. (apache#5221) * [Hotfix][Connector-V2][Paimon] Bump paimon-bundle version to 0.4.0-incubating (apache#5219) * [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL apache#4590 (apache#4757) * [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL * [Docs] Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (apache#4793) * [Docs] update seatunnel-flink.yaml and Dockerfile to help the demo work * [Docs] update release-note apache#4788 --------- Co-authored-by: flynnxue <flynnxue@lilithgames.com> Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com> * [feature][doris] Doris factory type (apache#5061) * [feature][doris] Web need factory and data type convertor * [Fix] Update the Readme (apache#4968) Use the better description for the SeaTunnel Project * [CI] Split updated modules integration test for part 5 (apache#5208) * [CI] Split updated modules integration test for part 5 * [CI] Split updated modules integration test for part 5 * Split e2e * update json-smart * fix dm error * fix dm error * fix dm error * fix dm error * fix dm error * fix dm error * fix dm error * revert code * revert code * revert code --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [Feature][CDC][Zeta] Support schema evolution framework(DDL) (apache#5125) * Fixed IMap file storage e2e bug (apache#5237) * [Improve] [Connector-V2] Remove scheduler in JDBC sink apache#4736 (apache#5168) --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [Doc] [JDBC Oracle] Add JDBC Oracle Documentation (apache#5239) * [Feature][Zeta][REST-API]Add REST API To Submit Job (apache#5107) * [Fix] Update the project description (apache#4967) * Update the project description * [Feature][Zeta] Support history service record job execute error (apache#5114) * fix:hdfs Checkpoint Storage management fails to delete historical files * fix:hdfs Checkpoint Storage management fails to delete historical files * fix after the savepoint job is restored, the checkpoint file cannot be generated * [Feature][Zeta] Support history service record job execute error * Improve Jobstate-related class additions add serialVersionUID * add e2e test * [hotfix]Update .asf.yaml (apache#5242) * Update .asf.yaml * [Hotfix]Fix array index anomalies caused by apache#5057 (apache#5195) * [bugfix] [savepoint test] Turn on the testSavepoint test. (apache#5199) * [BUG][Connector-V2][Jdbc] support postgresql json type (apache#5194) * add Postgresql json type Co-authored-by: 80597928 <673421862@qq.com> * [Bugfix][cdc] Fix mysql bit column to java byte (apache#4817) * [Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (apache#5146) * [Docs][Connector][Source][jdbc]Change the line boundary store value type to BigDecimal (apache#4900) * [bug][jdbc][oracle]Fix the Oracle number type mapping problem (apache#5209) * [Bugfix][zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation. (apache#5206) * [Hotfix][Zeta] Avoid Redundant Job Submissions by Checking Job Status (apache#5229) * [Bugfix][zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore. (apache#5214) * [Imporve] [CDC Base] Add a fast sampling method that supports character types (apache#5179) * fixed zeta ci error (apache#5254) * [Doc][README] Remove useless github workflow, and adjust description of 'engineering structure'. (apache#4305) * [Feature][Zeta]The expiration time of a historical Job can be config (apache#5180) * fix:hdfs Checkpoint Storage management fails to delete historical files Co-authored-by: hailin0 <wanghailin@apache.org> * [bugfix] [e2e] Fixed a minor bug (apache#5274) * [Improve][SQL] Support use catalogTableName as SQL expression (apache#5273) * [Doc] Improve S3File Source & S3File Sink document (apache#5101) * Improve S3File Source & S3File Sink document * Fix style error (apache#5280) * Fix StarRocksJsonSerializer will transform array/map/row to string (apache#5281) * [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document (apache#5129) * [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document * fix format * fix format * [Improve][API & Zeta] Using connector custom serializer encode/decode states (apache#5238) * API: Using DefaultSerializer as connector sink default serializer * Zeta: Using connector custom serializer encode/decode states * [Feature][Connector-V2] connector-kafka source support data conversion extracted by kafka connect source (apache#4516) * Compatible kafka connect json apache#4137 * [Improve][CI/CD] Remove 'paths-ignore', enable the code style check for markdown files. (apache#5286) * [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0. (apache#5263) * [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0. * remove max-concurrent * [Feature][Connector-v2][RedisSink]Support redis to set expiration time. (apache#4975) * Support redis to set expiration time. * Set redis expire default value. * add e2e test. * add e2e test. * modify config file name. --------- Co-authored-by: lightzhao <zhaolianyong777@gmail.com> * [bugfix] Fix testGetErrorInfo case error (apache#5282) * [Feature][Zeta] Checkpoint support hdfs ha mode (apache#4942) * fix browser long type intercept (apache#5267) Co-authored-by: 80597928 <673421862@qq.com> * [Docs] remove `incubating` keyword in document (apache#5257) * [feature][web] hive add option because web need (apache#5154) * [feature][web] hive add option because web need * [feature][web] hive add option read_columns * [feature][web] required update optional * [bugfix] mvn spotless * fix conf * fix conf --------- Co-authored-by: liuli <m_liuli@163.com> * [Bug][flink-runtime][connectors-v2] Flink register table Environment The running mode is set to`job.mode` (apache#4826) * [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document (apache#5132) * [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document * [Improve][Connector-v2][HiveSink]remove drop partition when abort. (apache#4940) Co-authored-by: lightzhao <zhaolianyong777@gmail.com> Co-authored-by: liuli <m_liuli@163.com> Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com> * [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document (apache#5130) * [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document * fix codestyle --------- Co-authored-by: liuli <m_liuli@163.com> * [Docs][Connector-V2][HDFS]Refactor connector-v2 docs using unified format HDFS. (apache#4871) * Refactor connector-v2 docs using unified format HDFS. * add data type. * update. * add key feature. * add hdfs_site_path * 1.add data type. 2.add hdfs_site_path conf. * add data type. * add hdfs site conf. --------- Co-authored-by: lightzhao <zhaolianyong777@gmail.com> Co-authored-by: liuli <m_liuli@163.com> * [Improve] [Connector-V2] Remove scheduler in Tablestore sink (apache#5272) --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase (apache#5184) * [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase * [Hotfix] Fix com.google.common.base.Preconditions to seatunnel shade one (apache#5284) * [Merge] Fix merge conflict and fix jdbc fieldIde with compatibleMode confusion --------- Co-authored-by: Cason-ACE <35160064+cason0126@users.noreply.github.com> Co-authored-by: Tyrantlucifer <TyrantLucifer@gmail.com> Co-authored-by: hailin0 <wanghailin@apache.org> Co-authored-by: Xiaojian Sun <sunxiaojian926@163.com> Co-authored-by: Laglangyue <35491928+laglangyue@users.noreply.github.com> Co-authored-by: ZhilinLi <zhilinli0706@gmail.com> Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com> Co-authored-by: Hao Xu <sduxuhao@gmail.com> Co-authored-by: Eric <gaojun2048@gmail.com> Co-authored-by: Bibo <33744252+531651225@users.noreply.github.com> Co-authored-by: 毕博 <bibo@mafengwo.com> Co-authored-by: Carl-Zhou-CN <67902676+Carl-Zhou-CN@users.noreply.github.com> Co-authored-by: zhouyao <yao.zhou@marketingforce.com> Co-authored-by: Marvin <29311598@qq.com> Co-authored-by: monster <60029759+MonsterChenzhuo@users.noreply.github.com> Co-authored-by: gnehil <adamlee489@gmail.com> Co-authored-by: TaoZex <45089228+TaoZex@users.noreply.github.com> Co-authored-by: xiaofan2012 <41982310+xiaofan2022@users.noreply.github.com> Co-authored-by: wantao <wantao@inmyshow.com> Co-authored-by: Guangdong Liu <804167098@qq.com> Co-authored-by: zhilinli <lzl15844876351@163.com> Co-authored-by: zhaifengbing <ainizfb@126.com> Co-authored-by: dalong <60906603+alibabaMapengfei@users.noreply.github.com> Co-authored-by: FuYouJ <1247908487@qq.com> Co-authored-by: davidfans <136911434+davidfans@users.noreply.github.com> Co-authored-by: Fan Donglai <ddna_1022@163.com> Co-authored-by: gdliu3 <gdliu3@iflytek.com> Co-authored-by: DismalSnail <yepenghaobit@gmail.com> Co-authored-by: lightzhao <40714172+lightzhao@users.noreply.github.com> Co-authored-by: chenzy15 <chenzy15@ziroom.com> Co-authored-by: wssmao <39487209+wssmao@users.noreply.github.com> Co-authored-by: wsstony <tonymao777@163.com> Co-authored-by: lightzhao <zhaolianyong777@gmail.com> Co-authored-by: XiaoJiang521 <131635688+XiaoJiang521@users.noreply.github.com> Co-authored-by: mcy <rewrma@163.com> Co-authored-by: yctanGmail <138592845+yctanGmail@users.noreply.github.com> Co-authored-by: yctan <1417983443@qq.com> Co-authored-by: wu-a-ge <wfygowxf1@163.com> Co-authored-by: 司马琦昂 <bruce.maqiang@gmail.com> Co-authored-by: happyboy1024 <137260654+happyboy1024@users.noreply.github.com> Co-authored-by: He Wang <wanghechn@qq.com> Co-authored-by: silenceland <silenceland23@163.com> Co-authored-by: changhuyan <877018069@qq.com> Co-authored-by: Jarvis <liunaijie1996@163.com> Co-authored-by: 阿丙 <50567478+gaopeng666@users.noreply.github.com> Co-authored-by: jackyyyyyssss <127465317+jackyyyyyssss@users.noreply.github.com> Co-authored-by: 80597928 <Lzl@qq.com> Co-authored-by: Chengyu Yan <cheneyyin@hotmail.com> Co-authored-by: zhangchengming601 <86779821+zhangchengming601@users.noreply.github.com> Co-authored-by: lihjChina <237206177@qq.com> Co-authored-by: David Zollo <davidzollo365@gmail.com> Co-authored-by: EchoLee5 <39044001+EchoLee5@users.noreply.github.com> Co-authored-by: dengdi <114273849+dengd1937@users.noreply.github.com> Co-authored-by: dengd1937 <dengd1803@gmail.com> Co-authored-by: happyboy1024 <296442618@qq.com> Co-authored-by: FlechazoW <35768015+FlechazoW@users.noreply.github.com> Co-authored-by: 80597928 <673421862@qq.com> Co-authored-by: kun <66303359+Lifu12@users.noreply.github.com> Co-authored-by: Volodymyr <volodymyrduke@gmail.com> Co-authored-by: javalover123 <javalover123@foxmail.com> Co-authored-by: Volodymyr <770925351@qq.com> Co-authored-by: kksxf <flynnxue02@gmail.com> Co-authored-by: flynnxue <flynnxue@lilithgames.com> Co-authored-by: fang <56808812+zhibinF@users.noreply.github.com> Co-authored-by: gejinxin <844156709@qq.com> Co-authored-by: Wenjun Ruan <wenjun@apache.org> Co-authored-by: Koyfin <1040080742@qq.com> Co-authored-by: liuli <m_liuli@163.com>

liugddx reviewed Jul 18, 2023

View reviewed changes

liugddx added Zeta Zeta Rest API First-time contributor First-time contributor labels Jul 18, 2023

liugddx reviewed Jul 18, 2023

View reviewed changes

zhibinF force-pushed the mydev branch 3 times, most recently from 33a4b75 to 971176f Compare July 21, 2023 08:11

liugddx reviewed Jul 24, 2023

View reviewed changes

ic4y reviewed Aug 2, 2023

View reviewed changes

ic4y added this to the 2.3.3 milestone Aug 2, 2023

zhibinF added 2 commits August 4, 2023 16:14

[Feature][Zeta][REST-API]Add REST API To Submit Job

48439cd

add doc for this feature

6c7ffe8

zhibinF force-pushed the mydev branch 2 times, most recently from d214815 to c28785f Compare August 4, 2023 08:24

EricJoy2048 reviewed Aug 4, 2023

View reviewed changes

refactor abstract AbstractJobEnvironment for JobImmutableInformationE…

a2a6958

…nv and JobExecutionEnvironment

zhibinF force-pushed the mydev branch from c28785f to a2a6958 Compare August 4, 2023 12:23

Merge branch 'dev' of https://github.com/zhibinF/seatunnel into mydev

8fe4689

ic4y approved these changes Aug 8, 2023

View reviewed changes

github-actions bot added approved reviewed labels Aug 8, 2023

liugddx approved these changes Aug 8, 2023

View reviewed changes

liugddx merged commit dd2fe00 into apache:dev Aug 8, 2023
56 checks passed

liunaijie pushed a commit to liunaijie/seatunnel that referenced this pull request Aug 12, 2023

[Feature][Zeta][REST-API]Add REST API To Submit Job (apache#5107)

cbe5b3f

zhibinF deleted the mydev branch September 4, 2023 06:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Zeta][REST-API]Add REST API To Submit Job #5107

[Feature][Zeta][REST-API]Add REST API To Submit Job #5107

zhibinF commented Jul 18, 2023

liugddx left a comment

liugddx commented Jul 18, 2023

zhibinF commented Jul 18, 2023

liugddx Jul 18, 2023

liugddx commented Jul 19, 2023

zhibinF commented Jul 19, 2023

liugddx left a comment

liugddx commented Jul 31, 2023

ic4y left a comment

ic4y Aug 2, 2023

zhibinF Aug 3, 2023

ic4y Aug 3, 2023

liugddx Aug 3, 2023

Hisoka-X Oct 19, 2023

zhibinF Oct 20, 2023

Hisoka-X Oct 20, 2023

zhibinF Oct 20, 2023

EricJoy2048 Aug 4, 2023

zhibinF Aug 4, 2023

liugddx commented Aug 7, 2023

liugddx left a comment

[Feature][Zeta][REST-API]Add REST API To Submit Job #5107

[Feature][Zeta][REST-API]Add REST API To Submit Job #5107

Conversation

zhibinF commented Jul 18, 2023

Purpose of this pull request

Check list

Description

Detail

liugddx left a comment

Choose a reason for hiding this comment

liugddx commented Jul 18, 2023

zhibinF commented Jul 18, 2023

Choose a reason for hiding this comment

liugddx commented Jul 19, 2023

zhibinF commented Jul 19, 2023

liugddx left a comment

Choose a reason for hiding this comment

liugddx commented Jul 31, 2023

ic4y left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liugddx commented Aug 7, 2023

liugddx left a comment

Choose a reason for hiding this comment