[Feature][Zeta][E2E] Enable zeta e2e and Fix the error when e2e enable #4615

TyrantLucifer · 2023-04-18T14:11:55Z

close #4636

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

seatunnel-api/src/main/java/org/apache/seatunnel/api/env/EnvCommonOptions.java

hailin0 · 2023-04-21T03:42:48Z

great job

...l-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/resources/log4j2.properties

Hisoka-X · 2023-04-23T02:53:50Z

...erver/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointCoordinator.java

@@ -397,7 +397,7 @@ private void startTriggerPendingCheckpoint(
                    LOG.info("wait checkpoint completed: " + pendingCheckpoint.getCheckpointId());
                    PassiveCompletableFuture<CompletedCheckpoint> completableFuture =
                            pendingCheckpoint.getCompletableFuture();
-                    completableFuture.whenComplete(
+                    completableFuture.whenCompleteAsync(


Note: make sure the TaskAcknowledgeOperation thread will be release after operation called.

Hisoka-X · 2023-04-23T02:54:51Z

...ngine-server/src/main/java/org/apache/seatunnel/engine/server/checkpoint/CheckpointPlan.java

-        private final Set<TaskLocation> pipelineSubtasks = new HashSet<>();
-        private final Set<TaskLocation> startingSubtasks = new HashSet<>();
-        private final Map<ActionStateKey, Integer> pipelineActions = new HashMap<>();
+        private final Set<TaskLocation> pipelineSubtasks = new CopyOnWriteArraySet<>();


Note: make sure Set::Stream doesn't have concurrency issues

Hisoka-X · 2023-04-23T02:56:55Z

...-server/src/main/java/org/apache/seatunnel/engine/server/task/SourceSplitEnumeratorTask.java

@@ -133,7 +135,7 @@ public ProgressState call() throws Exception {
    public void triggerBarrier(Barrier barrier) throws Exception {
        log.debug("split enumer trigger barrier [{}]", barrier);
        if (barrier.prepareClose()) {
-            this.currState = PREPARE_CLOSE;


Note: The state machine state cannot be changed outside the state machine. Just should send event.

Hisoka-X · 2023-04-23T02:57:25Z

...-server/src/main/java/org/apache/seatunnel/engine/server/task/SourceSplitEnumeratorTask.java

@@ -262,12 +264,16 @@ private void stateProcess() throws Exception {
                if (restoreComplete.isDone()) {
                    currState = READY_START;
                    reportTaskStatus(READY_START);
+                } else {
+                    Thread.sleep(100);


Avoid CPU cost too high.

Hisoka-X · 2023-04-23T02:57:59Z

...-server/src/main/java/org/apache/seatunnel/engine/server/task/SourceSplitEnumeratorTask.java

@@ -335,15 +343,15 @@ public Set<URL> getJarsUrl() {
    @Override
    public void notifyCheckpointComplete(long checkpointId) throws Exception {
        getEnumerator().notifyCheckpointComplete(checkpointId);
-        if (currState == PREPARE_CLOSE && prepareCloseBarrierId.get() == checkpointId) {
+        if (prepareCloseBarrierId.get() == checkpointId) {


Note: Unecessary check

Hisoka-X · 2023-04-23T02:59:14Z

...nel-engine-server/src/main/java/org/apache/seatunnel/engine/server/utils/NodeEngineUtil.java

@@ -37,7 +37,8 @@ public static <E> InvocationFuture<E> sendOperationToMasterNode(
                        .createInvocationBuilder(
                                SeaTunnelServer.SERVICE_NAME,
                                operation,
-                                nodeEngine.getMasterAddress());
+                                nodeEngine.getMasterAddress())
+                        .setAsync();


In task our all opeartion should be async as default .

TyrantLucifer and others added 14 commits April 18, 2023 14:23

[Feature][E2E][Zeta] Enable zeta e2e

5d220cc

[Feature][E2E][Zeta] Enable zeta e2e

0461ea5

[Feature][Zeta] Fix kafka & iceberg IT cases

fcc0600

[Feature][Zeta] Copy hadoop dependency into zeta container

fe49b24

[Feature][Zeta] Optimize CI

86641f4

[Feature][Zeta] Only enable zeta e2e to find the failed case

58ac335

[Feature][Zeta] Fix http IT case

df8b946

[Feature][Zeta] Fix jdbc-part-3 IT cases

f3073df

[Feature][CI] Improve the stability of CI

8810812

[Feature][E2E][Zeta] Remove redundant annotation

d2837f4

[Feature][Zeta][E2E] Improve CI process

efdccff

[Feature][Zeta][E2E] Change jvm options

e74e50a

add server log

876ac7d

[Feature][Zeta][E2E] Change log appender

f555cde

TyrantLucifer changed the title ~~[Feature][Zeat][E2E] Enable zeta e2e~~ [Feature][Zeta][E2E] Enable zeta e2e Apr 18, 2023

hailin0 reviewed Apr 20, 2023

View reviewed changes

seatunnel-api/src/main/java/org/apache/seatunnel/api/env/EnvCommonOptions.java Show resolved Hide resolved

TyrantLucifer added 4 commits April 21, 2023 11:20

[Feature][Zeta][E2E] Optimize CI process

9479dad

[Feature][Zeta][E2E] Fix Zeta bug

783cb71

[Feature][Zeta][E2E] Remove useless annotation

e0e44e9

[Feature][Zeta][E2E] Temporary disabled zeta engine for rocketmq e2e

170f2f7

TyrantLucifer force-pushed the enable-zeta-e2e branch from 93c2b9c to 170f2f7 Compare April 21, 2023 03:25

TyrantLucifer added 4 commits April 21, 2023 12:23

[Feature][Zeta][E2E] Enable all engine test cases

a235b8c

[Feature][Zeta][E2E] Revert log4j2.properties

0785975

[Feature][Zeta][E2E] Remove useless code & Add log4j2.properties

19b9f85

[Feature][Zeta][E2E] Improve iceberg case

073280b

hailin0 reviewed Apr 21, 2023

View reviewed changes

TyrantLucifer added 3 commits April 21, 2023 17:08

[Feature][Zeta][E2E] Fix http e2e case

4e3d647

[Feature][Zeta][E2E] Update log4j2.properties

8d41728

[Feature][Zeta][E2E] Remove useless CI check & Optimize CI check step

898cba2

[Feature][Zeta][E2E] Optimize http it cases

18ed106

TyrantLucifer mentioned this pull request Apr 22, 2023

[Hotfix][Connector-e2e][File-local] Complete the e2e test case env co… #4645

Closed

5 tasks

TyrantLucifer added 2 commits April 22, 2023 20:43

[Feature][Zeta][E2E] Fix dead links

641a330

[Feature][Zeta][E2E] Fix dead links

93368c7

hailin0 approved these changes Apr 22, 2023

View reviewed changes

github-actions bot added approved reviewed labels Apr 22, 2023

Hisoka-X reviewed Apr 23, 2023

View reviewed changes

EricJoy2048 approved these changes Apr 23, 2023

View reviewed changes

EricJoy2048 changed the title ~~[Feature][Zeta][E2E] Enable zeta e2e~~ [Feature][Zeta][E2E] Enable zeta e2e and Fix the error when e2e enable Apr 23, 2023

EricJoy2048 merged commit c8a5d52 into apache:dev Apr 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Zeta][E2E] Enable zeta e2e and Fix the error when e2e enable #4615

[Feature][Zeta][E2E] Enable zeta e2e and Fix the error when e2e enable #4615

TyrantLucifer commented Apr 18, 2023 •

edited

Loading

hailin0 commented Apr 21, 2023

Hisoka-X Apr 23, 2023

Hisoka-X Apr 23, 2023

Hisoka-X Apr 23, 2023

Hisoka-X Apr 23, 2023

Hisoka-X Apr 23, 2023

Hisoka-X Apr 23, 2023

[Feature][Zeta][E2E] Enable zeta e2e and Fix the error when e2e enable #4615

[Feature][Zeta][E2E] Enable zeta e2e and Fix the error when e2e enable #4615

Conversation

TyrantLucifer commented Apr 18, 2023 • edited Loading

Purpose of this pull request

Check list

hailin0 commented Apr 21, 2023

Hisoka-X Apr 23, 2023

Choose a reason for hiding this comment

Hisoka-X Apr 23, 2023

Choose a reason for hiding this comment

Hisoka-X Apr 23, 2023

Choose a reason for hiding this comment

Hisoka-X Apr 23, 2023

Choose a reason for hiding this comment

Hisoka-X Apr 23, 2023

Choose a reason for hiding this comment

Hisoka-X Apr 23, 2023

Choose a reason for hiding this comment

TyrantLucifer commented Apr 18, 2023 •

edited

Loading