[Feature][Connector-V2][Slack] Add Slack sink connector #3226

lianghuan-xatu · 2022-10-29T06:50:17Z

Purpose of this pull request

Slack sink connector #3018

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

…nnel-dist

EricJoy2048 · 2022-10-29T15:24:42Z

docs/en/connector-v2/sink/Slack.md

+
+## Changelog
+
+### 2.3.0-beta 2022-10-20


### 2.3.0-beta 2022-10-20 need replace with new version, new version will be update to the real version number when we release a version.

I have done it!

EricJoy2048 · 2022-11-04T01:57:29Z

tools/dependencies/known-dependencies.txt

Why update this?

This update has been reverted.Thx

ic4y · 2022-11-04T02:51:08Z

If no e2e test is added, you need to comment your local test process and results in the form of pictures, etc. Convenient for later use.

…nector-slack-branch # Conflicts: # tools/dependencies/known-dependencies.txt

hailin0 · 2022-11-04T04:20:04Z

tools/dependencies/known-dependencies.txt

@@ -1,108 +0,0 @@
-aircompressor-0.10.jar


revert

Why delete this file?

Delete this file carelessly， which has been reverted.

hailin0 · 2022-11-04T05:25:44Z

...onnector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/SlackWriter.java

+        for (Object field : fields) {
+            stringBuffer.append(field.toString() + ",");
+        }
+        stringBuffer.deleteCharAt(fields.length - 1);
+        stringBuffer.append("\n");


suggestion

StringJoiner stringJoiner = new StringJoiner(",", "", "\n"); for (Object field : fields) { stringJoiner.add(field.toString()); } String message = stringJoiner.toString();

If you reuse StringBuffer objects, you need to clean up the previous content

Thanks for your advice. it's a great idea!

hailin0 · 2022-11-04T05:29:36Z

...onnector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/SlackWriter.java

+        stringBuffer.deleteCharAt(fields.length - 1);
+        stringBuffer.append("\n");
+        try {
+            String conversationId = slackClient.findConversation();


Does findConversation need to be executed before each message is sent?

I have fixed it! PTAL

# Conflicts: # plugin-mapping.properties # seatunnel-connectors-v2/pom.xml # seatunnel-dist/pom.xml

lianghuan-xatu · 2022-11-04T17:00:57Z

If no e2e test is added, you need to comment your local test process and results in the form of pictures, etc. Convenient for later use.

lianghuan-xatu · 2022-11-06T09:00:18Z

If no e2e test is added, you need to comment your local test process and results in the form of pictures, etc. Convenient for later use.

Spark Local Test

EricJoy2048 · 2022-11-07T03:36:10Z

...nector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/client/SlackClient.java

+            );
+            publishMessageSuccess = chatPostMessageResponse.isOk();
+        } catch (IOException | SlackApiException e) {
+            log.error("error: {}", e.getMessage(), e);


"error: {}" only can add one argu, e.getMessage(), e have two.

You can update reference this:

log.error("error: {}", ExceptionUtils.getMessage(e));

I have fixed it! Thx

EricJoy2048 · 2022-11-07T03:38:06Z

...nector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/config/SlackConfig.java

+    private String slackChannel;
+
+    public SlackConfig(@NonNull Config pluginConfig) {
+        if (pluginConfig.hasPath(WEBHOOKS_URL)) {


All of webhooks_url and oauth_token and slack_channel is required. So you use CheckConfigUtil.checkAllExists to check them.

Thanks for your advice, I have done it!

EricJoy2048 · 2022-11-07T03:41:36Z

...onnector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/SlackWriter.java

+            // One message can be sent as soon as one second
+            Thread.sleep(POST_MSG_WAITING_TIME);
+        } catch (Exception e) {
+            log.warn("Write to Slack Fail.", e);


log.warn("Write to Slack Fail: {}", ExceptionUtils.getMessage(e));

I have fixed it! Thx

…nto connector-slack-branch # Conflicts: # plugin-mapping.properties # seatunnel-dist/pom.xml

…nto connector-slack-branch # Conflicts: # plugin-mapping.properties

TyrantLucifer

BTW, please implement Factory for connectors, you can refer to #3337

TyrantLucifer · 2022-11-16T12:44:12Z

...nector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/client/SlackClient.java

+    private SlackConfig slackConfig;
+    private MethodsClient methodsClient;


Suggested change

private SlackConfig slackConfig;

private MethodsClient methodsClient;

private final SlackConfig slackConfig;

private final MethodsClient methodsClient;

TyrantLucifer · 2022-11-16T12:46:45Z

...nector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/client/SlackClient.java

+            }
+        } catch (IOException | SlackApiException e) {
+            log.warn("Find Slack Conversion Fail.", e);
+            throw new RuntimeException("Find Slack Conversion Fail.", e);


You should create a SlackConnectorException that extended SeaTunnelRuntimeException instead of it. You can refer to https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-connectors-v2/connector-cassandra/src/main/java/org/apache/seatunnel/connectors/seatunnel/cassandra/exception

TyrantLucifer · 2022-11-16T12:49:10Z

...nector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/client/SlackClient.java

+     * Close Conversion
+     */
+    public void closeMethodClient() {
+        methodsClient = null;


Is there a close() method in MethodsClient?

There is no close() method in Slack API.

TyrantLucifer · 2022-11-16T12:49:35Z

...nector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/config/SlackConfig.java

+    private String webHooksUrl;
+    private String oauthToken;
+    private String slackChannel;


Suggested change

private String webHooksUrl;

private String oauthToken;

private String slackChannel;

private final String webHooksUrl;

private final String oauthToken;

private final String slackChannel;

TyrantLucifer · 2022-11-16T12:55:28Z

.../connector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/SlackSink.java

+    public void prepare(Config pluginConfig) throws PrepareFailException {
+        CheckResult checkResult = CheckConfigUtil.checkAllExists(pluginConfig, WEBHOOKS_URL, OAUTH_TOKEN, SLACK_CHANNEL);
+        if (!checkResult.isSuccess()) {
+            throw new PrepareFailException("Slack", PluginType.SINK, checkResult.getMsg());


The same as above, please refer to https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-cassandra/src/main/java/org/apache/seatunnel/connectors/seatunnel/cassandra/source/CassandraSource.java#L65

TyrantLucifer · 2022-11-16T12:56:33Z

...onnector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/SlackWriter.java

+
+@Slf4j
+public class SlackWriter extends AbstractSinkWriter<SeaTunnelRow, Void> {
+    private SlackConfig slackConfig;


Suggested change

private SlackConfig slackConfig;

private final SlackConfig slackConfig;

TyrantLucifer · 2022-11-16T12:56:58Z

...onnector-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/sink/SlackWriter.java

+            Thread.sleep(POST_MSG_WAITING_TIME);
+        } catch (Exception e) {
+            log.warn("Write to Slack Fail.", ExceptionUtils.getMessage(e));
+            throw new RuntimeException("Write to Slack Fail.", e);


Same as above, please refer to https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/connector-cassandra/src/main/java/org/apache/seatunnel/connectors/seatunnel/cassandra/source/CassandraSource.java#L78

Thanks for your advice. I have fixed it.

…nto connector-slack-branch # Conflicts: # plugin-mapping.properties

TyrantLucifer

Please update this document: https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/Error-Quick-Reference-Manual.md

TyrantLucifer · 2022-11-19T04:23:05Z

.../java/org/apache/seatunnel/connectors/seatunnel/slack/exception/SlackConnectorErrorCode.java

+
+    @Override
+    public String getCode() {
+        return null;


return code

TyrantLucifer · 2022-11-19T04:23:14Z

.../java/org/apache/seatunnel/connectors/seatunnel/slack/exception/SlackConnectorErrorCode.java

+
+    @Override
+    public String getDescription() {
+        return null;


return description

TyrantLucifer · 2022-11-19T04:25:37Z

...or-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/slack/sink/SlackWriter.java

+            // One message can be sent as soon as one second
+            Thread.sleep(POST_MSG_WAITING_TIME);
+        } catch (Exception e) {
+            log.warn("Write to Slack Fail.", ExceptionUtils.getMessage(e));


Suggested change

log.warn("Write to Slack Fail.", ExceptionUtils.getMessage(e));

log.error("Write to Slack Fail.", ExceptionUtils.getMessage(e));

TyrantLucifer · 2022-11-19T04:26:10Z

...ctor-slack/src/main/java/org/apache/seatunnel/connectors/seatunnel/slack/sink/SlackSink.java

+public class SlackSink extends AbstractSimpleSink<SeaTunnelRow, Void> {
+
+    private Config pluginConfig;
+    private SeaTunnelRowType seaTunnelRowType;


Suggested change

private SeaTunnelRowType seaTunnelRowType;

private final SeaTunnelRowType seaTunnelRowType;

TyrantLucifer

LGTM, cc @hailin0 @EricJoy2048

EricJoy2048

LGTM, Please wait CI complete. @TyrantLucifer

TyrantLucifer

LGTM, looking forward your next contribution.

lianghuan-xatu added 5 commits October 12, 2022 19:31

[Doc]Fix typos in QuickStart Doc

7014a51

Merge branch 'apache:dev' into dev

9b9df0a

Merge branch 'apache:dev' into dev

e000574

[Feature][Connector-V2][Slack] Add Slack sink connector

8b27692

[Feature][Connector-V2][Slack] Update pom and Add dependency in seatu…

d950956

…nnel-dist

EricJoy2048 reviewed Oct 29, 2022

View reviewed changes

lianghuan-xatu added 2 commits October 30, 2022 22:24

[Feature][Connector-V2][Slack] Update change log

c550f0d

[Feature][Connector-V2][Slack] Update log info

1ba3084

lianghuan-xatu requested a review from EricJoy2048 November 2, 2022 01:11

EricJoy2048 reviewed Nov 4, 2022

View reviewed changes

lianghuan-xatu added 5 commits November 4, 2022 11:53

[Feature][Connector-V2][Slack] Remove unrelative dependience update

2d883ec

[Feature][Connector-V2][Slack] Remove unrelative dependience update

ee498f8

Merge remote-tracking branch 'origin/connector-slack-branch' into con…

952927c

…nector-slack-branch # Conflicts: # tools/dependencies/known-dependencies.txt

[Feature][Connector-V2][Slack] Upate file plugin-mapping.properties

51baf1e

[Feature][Connector-V2][Slack] Upate file plugin-mapping.properties

6385c38

hailin0 reviewed Nov 4, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/dev' into connector-slack-branch

4f4cc7a

# Conflicts: # plugin-mapping.properties # seatunnel-connectors-v2/pom.xml # seatunnel-dist/pom.xml

Hisoka-X added feature New feature connectors-v2 labels Nov 4, 2022

[Feature][Connector-V2][Slack] Fix conversationID bug

9b53e47

lianghuan-xatu added 2 commits November 5, 2022 01:09

[Feature][Connector-V2][Slack] Fix code style

7e0d71b

[Feature][Connector-V2][Slack] Fix code style

76d4610

lianghuan-xatu requested review from hailin0 and EricJoy2048 November 6, 2022 07:04

[Feature][Connector-V2][Slack] Fix NPE bug

4487f32

EricJoy2048 reviewed Nov 7, 2022

View reviewed changes

[Feature][Connector-V2][Slack] Fix Code Style

f09092b

lianghuan-xatu added 2 commits November 15, 2022 19:59

Merge branch 'dev' of https://github.com/apache/incubator-seatunnel i…

369172c

…nto connector-slack-branch # Conflicts: # plugin-mapping.properties # seatunnel-dist/pom.xml

[Feature][Connector-V2][Slack] fix code style

3900bb7

lianghuan-xatu requested a review from ic4y November 15, 2022 12:14

Merge branch 'dev' of https://github.com/apache/incubator-seatunnel i…

108c9dc

…nto connector-slack-branch # Conflicts: # plugin-mapping.properties

TyrantLucifer reviewed Nov 16, 2022

View reviewed changes

lianghuan-xatu added 4 commits November 19, 2022 10:13

[Feature][Connector-V2][Slack] Implement Factory for connectors

68dac35

Merge branch 'dev' of https://github.com/apache/incubator-seatunnel i…

7ff9f3d

…nto connector-slack-branch # Conflicts: # plugin-mapping.properties

[Feature][Connector-V2][Slack] Fix code style

6de19f1

[Feature][Connector-V2][Slack] Fix code style

7ec9f5f

lianghuan-xatu requested review from TyrantLucifer and removed request for ic4y November 19, 2022 02:47

TyrantLucifer reviewed Nov 19, 2022

View reviewed changes

[Feature][Connector-V2][Slack] Add Slack Error-Quick-Reference-Manual

a8c6423

lianghuan-xatu requested a review from TyrantLucifer November 19, 2022 08:04

TyrantLucifer previously approved these changes Nov 19, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 19, 2022

lianghuan-xatu requested review from hailin0 and EricJoy2048 and removed request for hailin0 November 19, 2022 12:28

Merge branch 'dev' into connector-slack-branch

d67353c

EricJoy2048 dismissed TyrantLucifer’s stale review via d67353c November 21, 2022 13:51

github-actions bot removed approved reviewed labels Nov 21, 2022

EricJoy2048 approved these changes Nov 21, 2022

View reviewed changes

github-actions bot added approved reviewed labels Nov 21, 2022

TyrantLucifer approved these changes Nov 21, 2022

View reviewed changes

TyrantLucifer merged commit 7a836f2 into apache:dev Nov 21, 2022

EricJoy2048 mentioned this pull request Nov 23, 2022

[Umbrella][Connector-V2] SeaTunnel Connector V2 Tracking Table #3018

Open

3 tasks

		private SlackConfig slackConfig;
		private MethodsClient methodsClient;

	private SlackConfig slackConfig;
	private final SlackConfig slackConfig;

	log.warn("Write to Slack Fail.", ExceptionUtils.getMessage(e));
	log.error("Write to Slack Fail.", ExceptionUtils.getMessage(e));

	private SeaTunnelRowType seaTunnelRowType;
	private final SeaTunnelRowType seaTunnelRowType;

[Feature][Connector-V2][Slack] Add Slack sink connector #3226

[Feature][Connector-V2][Slack] Add Slack sink connector #3226

Conversation

lianghuan-xatu commented Oct 29, 2022

Purpose of this pull request

Check list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ic4y commented Nov 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lianghuan-xatu Nov 4, 2022 • edited Loading

Choose a reason for hiding this comment

lianghuan-xatu commented Nov 4, 2022

lianghuan-xatu commented Nov 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TyrantLucifer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TyrantLucifer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TyrantLucifer left a comment

Choose a reason for hiding this comment

EricJoy2048 left a comment

Choose a reason for hiding this comment

TyrantLucifer left a comment

Choose a reason for hiding this comment

lianghuan-xatu Nov 4, 2022 •

edited

Loading