Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector V2] Add GoogleSheets Sink #3848

Closed
wants to merge 13 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/en/connector-v2/Error-Quick-Reference-Manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,12 @@ problems encountered by users.
| DINGTALK-01 | Send response to DinkTalk server failed | When users encounter this error code, it means that send response message to DinkTalk server failed, please check it |
| DINGTALK-02 | Get sign from DinkTalk server failed | When users encounter this error code, it means that get signature from DinkTalk server failed , please check it |

## GoogleSheets Connector Error Codes

| code | description | solution |
|-----------------|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
| GOOGLESHEETS-01 | Build google sheets http request exception | When users encounter this error code, it means that send http request to build google sheets failed, please check it |

## Iceberg Connector Error Codes

| code | description | solution |
Expand All @@ -218,4 +224,3 @@ problems encountered by users.
| code | description | solution |
|-------------|-------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| EMAIL-01 | Send email failed | When users encounter this error code, it means that send email to target server failed, please adjust the network environment according to the abnormal information |

55 changes: 55 additions & 0 deletions docs/en/connector-v2/sink/GoogleSheets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# GoogleSheets

> GoogleSheets sink connector
## Description

Used to write data to GoogleSheets.

## Key features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [schema projection](../../concept/connector-v2-features.md)
TaoZex marked this conversation as resolved.
Show resolved Hide resolved

## Options

| name | type | required | default value |
|------------------- |--------------|----------|---------------|
| service_account_key | string | yes | - |
| sheet_id | string | yes | - |
| sheet_name | string | yes | - |
| range | string | yes | - |

### service_account_key [string]

google cloud service account, base64 required

### sheet_id [string]

sheet id in a Google Sheets URL

### sheet_name [string]

the name of the sheet you want to output

### range [string]

the range of the sheet you want to output

## Example

simple:

```hocon
GoogleSheets {
service_account_key = "Your account key"
sheet_id = "1VI0DvyZK-NIdssSdsDSsSSSC-_-rYMi7ppJiI_jhE"
sheet_name = "sheets01"
range = "A1:C3"
}
```

## Changelog

### next version

- Add GoogleSheets Sink Connector [3848](https://github.com/apache/incubator-seatunnel/pull/3848)
1 change: 1 addition & 0 deletions plugin-mapping.properties
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ seatunnel.sink.StarRocks = connector-starrocks
seatunnel.source.MyHours = connector-http-myhours
seatunnel.sink.InfluxDB = connector-influxdb
seatunnel.source.GoogleSheets = connector-google-sheets
seatunnel.sink.GoogleSheets = connector-google-sheets
seatunnel.sink.Tablestore = connector-tablestore
seatunnel.source.Lemlist = connector-http-lemlist
seatunnel.source.Klaviyo = connector-http-klaviyo
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.seatunnel.connectors.seatunnel.google.sheets.config;

import lombok.Data;

import java.io.Serializable;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

@Data
public class RangePosition implements Serializable {

private String startX;
private Integer startY;
private String endX;
private Integer endY;

public RangePosition buildWithRange(String range) {
RangePosition rangePosition = new RangePosition();
String[] ranges = range.split(":");
Pattern xPattern = Pattern.compile("[A-Z]+");
Pattern yPattern = Pattern.compile("[0-9]+");
Matcher startXMatch = xPattern.matcher(ranges[0]);
if (startXMatch.find()) {
rangePosition.setStartX(startXMatch.group());
}
Matcher startYMatch = yPattern.matcher(ranges[0]);
if (startYMatch.find()) {
rangePosition.setStartY(Integer.parseInt(startYMatch.group()));
}
Matcher endXMatch = xPattern.matcher(ranges[1]);
if (endXMatch.find()) {
rangePosition.setEndX(endXMatch.group());
}
Matcher endYMatch = yPattern.matcher(ranges[1]);
if (endYMatch.find()) {
rangePosition.setEndY(Integer.parseInt(endYMatch.group()));
}
return rangePosition;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,19 @@ public class SheetsConfig {
.stringType()
.noDefaultValue()
.withDescription("Google Sheets login service account key");

public static final Option<String> SHEET_ID = Options.key("sheet_id")
.stringType()
.noDefaultValue()
.withDescription("Google Sheets sheet id");

public static final Option<String> SHEET_NAME = Options.key("sheet_name")
.stringType()
.noDefaultValue()
.withDescription("Google Sheets sheet name that you want to import");
.withDescription("Google Sheets sheet name that you want to input/output");

public static final Option<String> RANGE = Options.key("range")
.stringType()
.noDefaultValue()
.withDescription("Google Sheets sheet range that you want to import");
.withDescription("Google Sheets sheet range that you want to input/output");
}
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,36 @@

package org.apache.seatunnel.connectors.seatunnel.google.sheets.config;

import org.apache.seatunnel.connectors.seatunnel.google.sheets.exception.GoogleSheetsConnectorErrorCode;
import org.apache.seatunnel.connectors.seatunnel.google.sheets.exception.GoogleSheetsConnectorException;

import org.apache.seatunnel.shade.com.typesafe.config.Config;

import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.sheets.v4.Sheets;
import com.google.api.services.sheets.v4.SheetsScopes;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.ServiceAccountCredentials;
import lombok.Data;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.Serializable;
import java.security.GeneralSecurityException;
import java.util.Base64;
import java.util.Collections;

@Data
public class SheetsParameters implements Serializable {

private static final String APPLICATION_NAME = "SeaTunnel Google Sheets";

private static final JsonFactory JSON_FACTORY = GsonFactory.getDefaultInstance();

private byte[] serviceAccountKey;

private String sheetId;
Expand All @@ -42,4 +63,23 @@ public SheetsParameters buildWithConfig(Config config) {
return this;
}

public Sheets buildSheets() throws IOException {
byte[] keyBytes = Base64.getDecoder().decode(this.serviceAccountKey);
ServiceAccountCredentials sourceCredentials = ServiceAccountCredentials
.fromStream(new ByteArrayInputStream(keyBytes));
sourceCredentials = (ServiceAccountCredentials) sourceCredentials
.createScoped(Collections.singletonList(SheetsScopes.SPREADSHEETS));
HttpRequestInitializer requestInitializer = new HttpCredentialsAdapter(sourceCredentials);
NetHttpTransport httpTransport = null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
NetHttpTransport httpTransport = null;
NetHttpTransport httpTransport;

try {
httpTransport = GoogleNetHttpTransport.newTrustedTransport();
} catch (GeneralSecurityException e) {
throw new GoogleSheetsConnectorException(GoogleSheetsConnectorErrorCode.BUILD_SHEETS_REQUEST_EXCEPTION,
"Build google sheets http request exception", e);
}
return new Sheets.Builder(httpTransport, JSON_FACTORY, requestInitializer)
.setApplicationName(APPLICATION_NAME)
.build();

}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.seatunnel.connectors.seatunnel.google.sheets.exception;

import org.apache.seatunnel.common.exception.SeaTunnelErrorCode;

public enum GoogleSheetsConnectorErrorCode implements SeaTunnelErrorCode {
BUILD_SHEETS_REQUEST_EXCEPTION("GOOGLESHEETS-01", "Build google sheets http request exception");

private final String code;

private final String description;

GoogleSheetsConnectorErrorCode(String code, String description) {
this.code = code;
this.description = description;
}

@Override
public String getCode() {
return this.code;
}

@Override
public String getDescription() {
return this.description;
}

@Override
public String getErrorMessage() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove

return SeaTunnelErrorCode.super.getErrorMessage();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.seatunnel.connectors.seatunnel.google.sheets.serialize;

import org.apache.seatunnel.api.table.type.SeaTunnelRow;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class GoogleSheetsSerializer implements SeaTunnelRowSerializer {

public GoogleSheetsSerializer() {
}

@Override
public List<List<Object>> deserializeRow(List<SeaTunnelRow> input) {
List<List<Object>> result = new ArrayList<>();
for (SeaTunnelRow seaTunnelRow : input) {
List<Object> row = new ArrayList<>(Arrays.asList(seaTunnelRow.getFields()));
result.add(row);
}
return result;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.seatunnel.connectors.seatunnel.google.sheets.serialize;

import org.apache.seatunnel.api.table.type.SeaTunnelRow;

import java.util.List;

public interface SeaTunnelRowSerializer {

List<List<Object>> deserializeRow(List<SeaTunnelRow> input);
}
Loading