Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[API-DRAFT] [MERGE] Merge api-draft to dev branch #2083

Merged
merged 107 commits into from
Jul 1, 2022
Merged

[API-DRAFT] [MERGE] Merge api-draft to dev branch #2083

merged 107 commits into from
Jul 1, 2022

Conversation

Hisoka-X
Copy link
Member

Purpose of this pull request

Merge api-draft to dev branch. Let SeaTunnel new API come to main branch

Check list

ashulin and others added 30 commits April 29, 2022 11:53
@ashulin @ruanwenjun
[Feature][core] base interface.
b7d2d23
@ashulin @ruanwenjun
[Feature][core] base logic
c7f36e4
@ruanwenjun
fix build error
f844d8c
@ruanwenjun
[api-draft] Add license header (#1777)
4639ba1 
* add license header、
* remove jdk compile version
* remove duplicate log4j
@ruanwenjun
Add seatunnel datatype and convert origin value into seatunnel data t…
c340795 
…ype (#1797)

* Add seatunnel datatype
@ruanwenjun
Add flink datatype converter (#1801)
1935d71
@ruanwenjun
Add complex seatunnel datatype (#1807)
3753ed3 
* Add complex seatunnel datatype
* implemente the complex type converter
@ruanwenjun
Add flink type converter UT (#1813)
06e74eb
@ruanwenjun
Rename Row to SeaTunnelRow (#1832)
609f587
@ruanwenjun
Add spark basic type converter (#1837)
8f710f7
@Hisoka-X
[Api-Draft] Add Flink Sink Converter to support SeaTunnel transfer to…
0db318b 
… Flink engine. (#1839)

* Add Flink Sink Converter to support SeaTunnel transfer to Flink engine.
@ashulin
[Api-Draft] SeaTunnel Source support Flink engine. (#1842)
6b5136b 
Coordinated sources are not yet supported
@ruanwenjun
Add generics type about flink sink (#1844)
21e159a
@ruanwenjun
Add SeatunnelTaskExecuteCommand (#1847)
8ff90fd
@ruanwenjun
Add basic fake source (#1864)
b063bd5
@Hisoka-X
[Api-draft] Support Spark Sink convert to SeaTunnel (#1863)
48423a9 
* Add Flink Sink Converter to support SeaTunnel transfer to Flink engine.
* Support Spark sink translation
* change license
@ruanwenjun
Add SeaTunnelAPIExample (#1867)
96bd703 
* Add command implementation
* Add SeaTunnelAPIExample
@Hisoka-X
[Api-draft] Fix Spark Sink Can't support batch mode. (#1868)
af7f6e0 
* Fix Spark Sink Can't support batch mode.
@ruanwenjun
Add plugin discovery module (#1881)
5f79f45 
* Add plugin discovery module
* Remove PluginFactory
@ruanwenjun
Add boundedness to source reader context, so that we can control to s…
3abf728 
…top this reader when produce data (#1885)
@ruanwenjun
Add license header (#1886)
661f8d5
@Hisoka-X
[API-Draft] Add comment on common interface. (#1888)
fbeab83 
* Add comment on common interface.
* add serializable interface
@ruanwenjun
Add SeaTunnelPluginDiscovery to load seatunnel new api plugin (#1889)
1844a7a 
* Add SeaTunnelPluginDiscovery to load seatunnel new api plugin
@Hisoka-X
[Api-Draft] Fix Flink sink type convert error. (#1890)
dce9ec9 
* fix Flink sink type convert error
@ashulin
[Api-Draft] SeaTunnel Source support Spark engine. (#1871)
862c31e
@ruanwenjun
fix seatunnel source parallelism (#1892)
b16f382
@ruanwenjun
Add SeaTunnelRowTypeInfo to support source return record type (#1894)
b23266e 
* Add SeaTunnelRowTypeInfo to support source return record type
@ruanwenjun
Add Transform to new SeaTunnel API (#1900)
62b304a
@ruanwenjun
Add plugin processor for SeaTunnel API (#1902)
273121d
@ruanwenjun
Add SeaTunnelRowTypeInfo to SeaTunnelSink (#1904)
727c7b2
@ruanwenjun ruanwenjun assigned ruanwenjun and Hisoka-X and unassigned ruanwenjun Jun 29, 2022
@ruanwenjun ruanwenjun added API-refactor core SeaTunnel core module connectors-v1 SeaTunnel connectors, include sink, source labels Jun 29, 2022
lhyundeadsoul and others added 8 commits June 30, 2022 10:29
@lhyundeadsoul
add assert sink to Api draft (#2071)
fc640b5 
* [api-draft][connector] add assert sink
 close #1912 (comment)

* [api-draft][connector] add assert sink
 remove commons-collections4-4.2.jar in known-dependencies.txt and LICENSE

* [api-draft][connector] add doc assert plugin name

* [api-draft][connector] fix config path error when field is null
@Hisoka-X
add clickhouse source and sink docs (#2072)
032906f
@Hisoka-X
merge dev to api-draft
d265597 
merge dev to api-draft
@Hisoka-X
[API-DRAFT] [MERGE] fix merge error
736ac01
@Hisoka-X
[API-Draft] [Doc]add common option (#2095)
0656b50
@Hisoka-X
Update Clickhouse.md (#2096)
6a492d7
@Hisoka-X
[API-DRAFT] [MERGE] fix merge error
3c0e984
@Hisoka-X
Merge remote-tracking branch 'origin/api-draft' into api-draft
10ac17e
@Hisoka-X
Copy link
Member Author

Hi guys, I will merge api-draft to dev branch, So I create this PR. I resolved some conflict when merge dev to api-draft. The next step is make sure that there are no mistake created when I resolved conflict. Please check that your commit part in dev and api-draft have already complete and correct merged. If so, I will merge this PR to dev, then we can work on dev with new seatunnel api. Thanks!

ruanwenjun
ruanwenjun approved these changes Jul 1, 2022
View reviewed changes
Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I find we have moved the new connector to seatunnel-connectors-v2, this is great, so it's better to change all the new-connector to connector-v2. I find in e2e, the module name is still new-connector, we can continue to optimize this after this PR merged.

@ruanwenjun
Copy link
Member

And there are so many sonor bugs, we may need to solve this.

@Hisoka-X
Copy link
Member Author

Hisoka-X commented Jul 1, 2022

And there are so many sonor bugs, we may need to solve this.

Yes, we need solve bug and remove many duplicate code

@sonarcloud
Copy link

sonarcloud bot commented Jul 1, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug E 56 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot E 7 Security Hotspots
Code Smell A 660 Code Smells

0.0% 0.0% Coverage
6.8% 6.8% Duplication

@EricJoy2048
Copy link
Member

LGTM, Hive Sink Connector is a temporary implementation. I will rewrite this connector after this pr merged.

@CalvinKirs CalvinKirs merged commit df251bf into dev Jul 1, 2022
@EricJoy2048
Copy link
Member

EricJoy2048 commented Jul 1, 2022 •
edited
Loading

LGTM. I find we have moved the new connector to seatunnel-connectors-v2, this is great, so it's better to change all the new-connector to connector-v2. I find in e2e, the module name is still new-connector, we can continue to optimize this after this PR merged.

Yes, We should unify the names of each module of the new connector. consist of update update seatunnel-flink-new-connector-e2e to seatunnel-flink-v2-e2e , update seatunnel-spark-new-connector-e2e to seatunnel-spark-v2-e2e, update docs/en/new-connector to docs/en/connector-v2.

I will submit a pr to fix this after this pr merged.

@CalvinKirs CalvinKirs deleted the api-draft branch February 6, 2023 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruanwenjun ruanwenjun ruanwenjun approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

@Hisoka-X Hisoka-X

Labels
connectors-v1 SeaTunnel connectors, include sink, source core SeaTunnel core module
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@Hisoka-X @ruanwenjun @EricJoy2048 @CalvinKirs @ashulin @zhuangchong @ic4y @iture123 @Emor-nj @lhyundeadsoul