[Feature] support `LZO` compress on File Read #5083

liunaijie · 2023-07-14T01:49:51Z

Purpose of this pull request

support compress on local text file read.

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

TyrantLucifer · 2023-07-17T11:10:24Z

Good pr, let's waiting CI/CD

TaoZex

LGTM
But the doc needs to be modified.

liunaijie · 2023-07-19T09:06:50Z

LGTM But the doc needs to be modified.

doc updated.

liunaijie · 2023-07-20T09:52:15Z

updated json reader.
updated document

liunaijie · 2023-07-21T03:14:16Z

@TaoZex @TyrantLucifer PTAL

liunaijie · 2023-07-27T02:16:35Z

pls re-trigger the CI as i update more formart reader

TyrantLucifer · 2023-07-27T03:57:30Z

pls re-trigger the CI as i update more formart reader

Done, let's waiting CI/CD.

liunaijie · 2023-07-31T02:13:35Z

pls trigger the CI

# Conflicts: # docs/en/connector-v2/source/LocalFile.md # seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSourceConfig.java # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/local/LocalFileIT.java

liunaijie · 2023-08-01T05:16:28Z

@TyrantLucifer @TaoZex PTAL

ic4y · 2023-08-05T08:07:42Z

The documentation for connectors depending on the file-base package, such as hdfsFile, oss, and s3, likely needs updating.

liunaijie · 2023-08-07T06:29:30Z

The documentation for connectors depending on the file-base package, such as hdfsFile, oss, and s3, likely needs updating.

@ic4y i will create another pr (add test case and update doc) for connectors depending on the file-base

ic4y · 2023-08-08T03:56:27Z

The documentation for connectors depending on the file-base package, such as hdfsFile, oss, and s3, likely needs updating.

@ic4y i will create another pr (add test case and update doc) for connectors depending on the file-base

It's best to complete this within this PR, as it's part of the whole.

liunaijie · 2023-08-09T02:21:01Z

@ic4y updated related connector and document, PTAL, thanks

ic4y · 2023-08-09T06:48:27Z

Waiting for CI/CD.

liunaijie · 2023-08-10T07:22:35Z

failed on ClusterFaultToleranceTwoPipelineIT.testTwoPipelineStreamJobRestoreIn2NodeMasterDown:732 » ConditionTimeout

can you help re-trigger the CI, thanks

# Conflicts: # docs/en/connector-v2/source/S3File.md

liunaijie · 2023-08-12T07:39:39Z

i can see the CI error has been fixed in another pr, can you help trigger this CI again, thanks.

liunaijie · 2023-09-04T14:04:27Z

wait for this issue fix
#5342

# Conflicts: # docs/en/connector-v2/source/HdfsFile.md # docs/en/connector-v2/source/Hive.md

...-v2-e2e/connector-file-local-e2e/src/test/resources/json/local_file_json_lzo_to_console.conf

EricJoy2048

I am sorry, In the Apache project, it is not allowed to publish binary files in the source code package (because the content of the binary file is not visible and cannot be modified. If there is a problem in the file, no one can fix it), so an alternative solution is to generate the binary file through the program (the generated code is visible, and if there is a problem, the code can also be modified to fix the problem).

Hisoka-X · 2023-10-13T09:21:04Z

@liunaijie I updated test case for this. Please take a look when you have free time.

Hisoka-X · 2023-10-13T09:29:55Z

The test case have some bug will be fixed in #5622

liunaijie · 2023-10-14T01:39:38Z

@liunaijie I updated test case for this. Please take a look when you have free time.

@Hisoka-X cool, thanks. the test case change looks good to me, but i find you change the discoverTestContainers method, look this change is only for testing, should't push.

Hisoka-X · 2023-10-14T01:52:04Z

@liunaijie I updated test case for this. Please take a look when you have free time.

@Hisoka-X cool, thanks. the test case change looks good to me, but i find you change the discoverTestContainers method, look this change is only for testing, should't push.

oh, good catch. Reverted.

Add data check complete.

Hisoka-X

Thanks @liunaijie . LGTM

--------- Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

liunaijie added 2 commits July 14, 2023 09:46

[Feature] support compress on Text File Read

40f1993

[Feature] fix code style

e0e26b3

TaoZex reviewed Jul 19, 2023

View reviewed changes

[Feature] update local file doc

21521a0

[Improve] local file read compression

9ac95ca

liunaijie changed the title ~~[Feature] support compress on Text File Read~~ [Feature] support compress on File Read Jul 20, 2023

fix code style

5ba9c26

liunaijie added 2 commits August 1, 2023 13:12

code style

f4a5d51

[Improve] add compress_codec parameter to file related connector

c2f4d6e

liunaijie force-pushed the text_file_read_compress branch from 36eed25 to c2f4d6e Compare August 9, 2023 06:35

liunaijie force-pushed the text_file_read_compress branch from c75bf4f to c2f4d6e Compare August 12, 2023 07:32

Merge branch 'up_dev' into text_file_read_compress

86cb4c6

# Conflicts: # docs/en/connector-v2/source/S3File.md

Merge branch 'dev' into text_file_read_compress

36185a1

# Conflicts: # docs/en/connector-v2/source/HdfsFile.md # docs/en/connector-v2/source/Hive.md

TyrantLucifer previously requested changes Sep 14, 2023

View reviewed changes

...-v2-e2e/connector-file-local-e2e/src/test/resources/json/local_file_json_lzo_to_console.conf Show resolved Hide resolved

github-actions bot removed approved reviewed labels Sep 14, 2023

[Feature] add assert sink check

01865f2

liunaijie dismissed EricJoy2048’s stale review via 01865f2 September 15, 2023 00:13

[Feature] update assert sink check

1af0b31

liunaijie force-pushed the text_file_read_compress branch from f341d14 to 1af0b31 Compare September 18, 2023 14:27

EricJoy2048 reviewed Sep 25, 2023

View reviewed changes

Merge branch 'dev' into text_file_read_compress

9c85559

Hisoka-X added this to the 2.3.4 milestone Oct 13, 2023

Hisoka-X added 2 commits October 13, 2023 15:03

Merge branch 'dev_' into pr/5083

48552a6

update test case

1445f96

Merge branch 'dev_' into pr/5083

cadf6a4

update

4a484ac

Hisoka-X added 3 commits October 14, 2023 10:03

update

ff83e7c

Merge branch 'dev_' into pr/5083

6b6e388

fix ci

7e5cc2e

EricJoy2048 approved these changes Oct 16, 2023

View reviewed changes

github-actions bot added approved reviewed labels Oct 16, 2023

Hisoka-X changed the title ~~[Feature] support compress on File Read~~ [Feature] support LZO compress on File Read Oct 16, 2023

Hisoka-X approved these changes Oct 16, 2023

View reviewed changes

Hisoka-X merged commit a4a1901 into apache:dev Oct 16, 2023
7 checks passed

liunaijie deleted the text_file_read_compress branch October 31, 2023 01:39

ruanwenjun pushed a commit to ruanwenjun/incubator-seatunnel that referenced this pull request Nov 14, 2023

[Feature] Support LZO compress on File Read (apache#5083)

f76971a

--------- Co-authored-by: Jia Fan <fanjiaeminem@qq.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] support `LZO` compress on File Read #5083

[Feature] support `LZO` compress on File Read #5083

liunaijie commented Jul 14, 2023 •

edited

Loading

TyrantLucifer commented Jul 17, 2023

TaoZex left a comment

liunaijie commented Jul 19, 2023

liunaijie commented Jul 20, 2023 •

edited

Loading

liunaijie commented Jul 21, 2023

liunaijie commented Jul 27, 2023

TyrantLucifer commented Jul 27, 2023

liunaijie commented Jul 31, 2023

liunaijie commented Aug 1, 2023

ic4y commented Aug 5, 2023

liunaijie commented Aug 7, 2023

ic4y commented Aug 8, 2023

liunaijie commented Aug 9, 2023

ic4y commented Aug 9, 2023

liunaijie commented Aug 10, 2023

liunaijie commented Aug 12, 2023

liunaijie commented Sep 4, 2023

EricJoy2048 left a comment

Hisoka-X commented Oct 13, 2023 •

edited

Loading

Hisoka-X commented Oct 13, 2023

liunaijie commented Oct 14, 2023

Hisoka-X commented Oct 14, 2023

Hisoka-X left a comment

[Feature] support LZO compress on File Read #5083

[Feature] support LZO compress on File Read #5083

Conversation

liunaijie commented Jul 14, 2023 • edited Loading

Purpose of this pull request

Check list

TyrantLucifer commented Jul 17, 2023

TaoZex left a comment

Choose a reason for hiding this comment

liunaijie commented Jul 19, 2023

liunaijie commented Jul 20, 2023 • edited Loading

liunaijie commented Jul 21, 2023

liunaijie commented Jul 27, 2023

TyrantLucifer commented Jul 27, 2023

liunaijie commented Jul 31, 2023

liunaijie commented Aug 1, 2023

ic4y commented Aug 5, 2023

liunaijie commented Aug 7, 2023

ic4y commented Aug 8, 2023

liunaijie commented Aug 9, 2023

ic4y commented Aug 9, 2023

liunaijie commented Aug 10, 2023

liunaijie commented Aug 12, 2023

liunaijie commented Sep 4, 2023

EricJoy2048 left a comment

Choose a reason for hiding this comment

Hisoka-X commented Oct 13, 2023 • edited Loading

Hisoka-X commented Oct 13, 2023

liunaijie commented Oct 14, 2023

Hisoka-X commented Oct 14, 2023

Hisoka-X left a comment

Choose a reason for hiding this comment

[Feature] support `LZO` compress on File Read #5083

[Feature] support `LZO` compress on File Read #5083

liunaijie commented Jul 14, 2023 •

edited

Loading

liunaijie commented Jul 20, 2023 •

edited

Loading

Hisoka-X commented Oct 13, 2023 •

edited

Loading