Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Doc] Add Config file introduce. #1740

Merged
merged 6 commits into from
Apr 29, 2022
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions docs/en/concept/config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
sidebar_position: 2
---

# Intro to config file

In SeaTunnel, the most important thing is the Config file, through which users can customize their own data
synchronization requirements to maximize the potential of SeaTunnel. So next, I will introduce you how to
configure the Config file.

## Example

Before you read on, you can find config file
examples [here](https://github.com/apache/incubator-seatunnel/tree/dev/config) and in distribute package's
config directory.

## Config file structure

The Config file will be similar to the one below.

```hocon
env {
execution.parallelism = 1
}

source {
FakeSource {
result_table_name = "fake"
field_name = "name,age"
}
}

transform {
sql {
sql = "select name,age from fake"
}
}

sink {
Clickhouse {
host = "clickhouse:8123"
database = "default"
table = "seatunnel_console"
fields = ["name"]
username = "default"
password = ""
}
}
```

As you can see, the Config file contains several sections: env, source, transform, sink. Different modules
have different functions. After you understand these modules, you will understand how SeaTunnel works.

### env

Used to add some engine optional parameters, no matter which engine (Spark or Flink), the corresponding
optional parameters should be filled in here.

<!-- TODO add supported env parameters -->

### source

source is used to define where SeaTunnel needs to fetch data, and use the fetched data for the next step.
Multiple sources can be defined at the same time. The supported source at now
check [Source of SeaTunnel](../connector/source). Each source has its own specific parameters to define how to
fetch data, and SeaTunnel also extracts the parameters that each source will use, such as
the `result_table_name` parameter, which is used to specify the name of the data generated by the current
source, which is convenient for follow-up used by other modules.

### transform

When we have the data source, we may need to further process the data, so we have the transform module. Of
course, this uses the word 'may', which means that we can also directly treat the transform as non-existent,
directly from source to sink. Like below.

```hocon
transform {
// no thing on here
}
```

Like source, transform has specific parameters that belong to each module. The supported source at now check.
The supported transform at now check [Transform of SeaTunnel](../transform)

### sink

Our purpose with SeaTunnel is to synchronize data from one place to another, so it is critical to define how
and where data is written. With the sink module provided by SeaTunnel, you can complete this operation quickly
and efficiently. Sink and source are very similar, but the difference is reading and writing. So go check out
our [supported sinks](../connector/sink).

### Other

You will find that when multiple sources and multiple sinks are defined, which data is read by each sink, and
which is the data read by each transform? We use `result_table_name` and `source_table_name` two key
configurations. Each source module will be configured with a `result_table_name` to indicate the name of the
data source generated by the data source, and other transform and sink modules can use `source_table_name` to
refer to the corresponding data source name, indicating that I want to read the data for processing. Then
transform, as an intermediate processing module, can use both `result_table_name` and `source_table_name`
configurations at the same time. But you will find that in the above example Config, not every module is
configured with these two parameters, because in SeaTunnel, there is a default convention, if these two
parameters are not configured, then the generated data from the last module of the previous node will be used.
This is much more convenient when there is only one source.

## What's More

If you want to know the details of this format configuration, Please
see [HOCON](https://github.com/lightbend/config/blob/main/HOCON.md).
8 changes: 0 additions & 8 deletions docs/en/connector/config-example.md

This file was deleted.

5 changes: 1 addition & 4 deletions docs/en/faq.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
# FAQ

why-i-should-install-computing-engine-like-spark-or-flink

## Why I should install computing engine like Spark or Flink

<!-- We should add the reason -->
TODO
Now SeaTunnel uses computing engines such as spark and flink to complete resource scheduling and node communication, so we can focus on the ease of use of data synchronization and the development of high-performance components. But this is only temporary.

## I have a question, but I can not solve it by myself

Expand Down
2 changes: 1 addition & 1 deletion docs/en/start/local.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem';

Before you getting start the local run, you need to make sure you already have installed the following software which SeaTunnel required:

* [Java](https://www.java.com/en/download/) (only JDK 8 supported by now) installed and `JAVA_HOME` set.
* [Java](https://www.java.com/en/download/) (Java 8 or 11, other versions greater than Java 8 can theoretically work as well) installed and `JAVA_HOME` set.
* Download the engine, you can choose and download one of them from below as your favour, you could see more information about [why we need engine in SeaTunnel](../faq.md#why-i-should-install-computing-engine-like-spark-or-flink)
* Spark: Please [download Spark](https://spark.apache.org/downloads.html) first(**required version >= 2** and version < 3.x). For more information you could
see [Getting Started: standalone](https://spark.apache.org/docs/latest/spark-standalone.html#installing-spark-standalone-to-a-cluster)
Expand Down
10 changes: 8 additions & 2 deletions docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -68,14 +68,20 @@ const sidebars = {
items: [
'start/local',
'start/docker',
'start/kubernetes',
'start/kubernetes'
],
},
{
zhongjiajie marked this conversation as resolved.
Show resolved Hide resolved
type: 'category',
label: 'Basic Concept',
zhongjiajie marked this conversation as resolved.
Show resolved Hide resolved
items: [
'concept/config',
],
},
{
type: 'category',
label: 'Connector',
items: [
'connector/config-example',
{
type: 'category',
label: 'Source',
Expand Down