Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(rocketmq) quick start is done. #14

Merged
merged 2 commits into from
Jul 2, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 85 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,94 @@
# Rocketmq Streams SQL
# rsqldb

Rocketmq Streams SQL 为 Rocketmq Streams 的开发提供了基于SQL的开发体验, 让基于消息队列的流式开发更加容易;
rsqldb 为 Rocketmq Streams 的开发提供了基于SQL的开发体验, 让基于消息队列的流式开发更加容易;

## Features

* 采用标准的流式SQL规范,可以与其他的流计算框架如Flink完美兼容;
* 兼容Flink自带的```udf```、```udaf```和```udtf```,除此之外,用户还可以通过实现相关接口来轻松扩展函数;


如果您希望更详细的了解Rsqldb的相关内容, 请点击[这里](docs/SUMMARY.md)
如果您希望更详细的了解rsqldb的相关内容, 请点击[这里](docs/SUMMARY.md)


## Quickstart
### 运行环境
- JDK 1.8及以上
- Maven 3.2及以上

### 下载rsqldb工程并本地构建
```xml
git clone https://github.com/alibaba/rsqldb.git

mvn clean package -DskipTest -U
```

### 拷贝安装压缩包并解压

进入rsqldb-disk模块下,将rsqldb-distribution.tar.gz安装包拷贝到任意目录,并执行命令解压并进入解压目录:
```xml
tar -zxvf rsqldb-distribution.tar.gz;cd rsqldb
```


### 启动rsqldb服务端
```shell
chmod +x bin/startAll.sh;sh bin/startAll.sh
```

### 配置sql文件
sendDataFromFile.sql中创建的任务,需要从本地文件指定位置读取数据,所以需要修改sendDataFromFile.sql中filePath变量的位置,修改为数据文件data.txt的绝对路径。


### 提交任务
执行路径依然在rsqldb解压目录下
```shell
chmod +x client/clientExector.sh;sh client/clientExector.sh submitTask sendDataFromFile.sql
```


### 启动任务
在rsqldb解压目录下执行,tail运行日志,为查看结果做准备。
```shell
tail -f log/rsqldb-runner.log
```

另开一个shell窗口,进入解压后的rsqldb目录,执行以下命令启动任务,1分钟后,查看日志输出,会将执行结果打印到日志中。
```shell
sh client/clientExector.sh startTask
```

### 查询任务
在rsqldb解压目录下执行
```shell
sh client/clientExector.sh queryTask
```
返回已经提交的任务列表。

### 停止任务
在rsqldb解压目录下执行
```shell
sh client/clientExector.sh stopTask
```

### 从RocketMQ中读取数据并处理
上述示例为从本地文件data.txt中读取数据,更为常用的用法是从RocketMQ中读取数据处理,下面给出具体步骤:

- 本地安装并启动RocketMQ,[安装文档](https://rocketmq.apache.org/docs/quick-start/)
- 启动rsqldb服务端
```shell
chmod +x bin/startAll.sh;sh bin/startAll.sh
```
- 提交任务
```shell
chmod +x client/clientExector.sh;sh client/clientExector.sh submitTask rocketmq.sql
```
- 查看输出
```shell
tail -f log/rsqldb-runner.log
```
- 另开一个窗口,启动任务
```shell
sh client/clientExector.sh startTask
```
- 向RocketMQ中生产数据:topic为rsqldb-source,与rocketmq.sql任务中的topic名称保持一致,向该topic写入data.txt文件中的数据。观察rsqldb-runner.log日志输出。
1 change: 0 additions & 1 deletion docs/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Summary

* [Introduction](README.md)
* [Quick Start](quick_start/README.md)
* [Blink SQL兼容](stream_sql/README.md)
* [创建源表](stream_source/README.md)
* [创建metaq源表](stream_source/metaq/README.md)
Expand Down
21 changes: 0 additions & 21 deletions docs/quick_start/README.md

This file was deleted.

Binary file removed docs/quick_start/pic/img.png
Binary file not shown.
5 changes: 0 additions & 5 deletions docs/quick_start/standalone/README.md

This file was deleted.

2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
<blink.version>blink-3.4.0</blink.version>
<flink.version>1.13.1</flink.version>
<scala-library.version>2.12.4</scala-library.version>
<rocketmq-streams.version>1.0.3-preview-SNAPSHOT</rocketmq-streams.version>
<rocketmq-streams.version>1.0.2-preview-SNAPSHOT</rocketmq-streams.version>
</properties>
<build>
<plugins>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.alibaba.rsqldb.client;
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import com.alibaba.rsqldb.client.http.HttpHelper;

import static com.alibaba.rsqldb.client.constant.Constants.queryTask;

public class QueryTask {
public static void main(String[] args) throws Throwable {
String result = HttpHelper.sendRequest(queryTask, "test", null);

System.out.println(result);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@

public class StartTask {
public static void main(String[] args) throws Throwable {
HttpHelper.startTask(startTask, "test", "test");
String result = HttpHelper.sendRequest(startTask, "test", "test");
System.out.println(result);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package com.alibaba.rsqldb.client;
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import com.alibaba.rsqldb.client.http.HttpHelper;

import static com.alibaba.rsqldb.client.constant.Constants.stopTask;

public class StopTask {
public static void main(String[] args) throws Throwable{
String result = HttpHelper.sendRequest(stopTask, "test", "test");
System.out.println(result);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,6 @@
*/

package com.alibaba.rsqldb.client;
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import com.alibaba.rsqldb.client.http.HttpHelper;
import com.alibaba.rsqldb.client.util.FileUtil;
Expand All @@ -44,12 +28,14 @@
public class SubmitTask {

public static void main(String[] args) throws Throwable {
if (args == null || args.length < 1) {
throw new IllegalArgumentException("home.dir is required.");
if (args == null || args.length < 2) {
throw new IllegalArgumentException("home.dir and sql file name are required.");
}

String homeDir = args[0];
String sqlFileName = args[1];

String sqlPath = homeDir + "/client/standalone.sql";
String sqlPath = homeDir + "/client/" + sqlFileName;

File file = FileUtil.getFile(sqlPath);
byte[] bytes = Files.readAllBytes(file.toPath());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,7 @@ public class Constants {

public static String startTask = "http://localhost:8080/command/task/start";

public static String queryTask = "http://localhost:8080/command/task/list";

public static String stopTask = "http://localhost:8080/command/task/stop";
}
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
import static org.apache.http.HttpHeaders.USER_AGENT;

public class HttpHelper {
public static void submitTask(String url,String namespace, String taskName, String sql) throws Exception {
public static void submitTask(String url, String namespace, String taskName, String sql) throws Exception {

HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url);
Expand Down Expand Up @@ -77,10 +77,9 @@ public static void submitTask(String url,String namespace, String taskName, Stri
}

System.out.println(result);

}

public static void startTask(String url, String namespace, String taskName) throws Throwable {
public static String sendRequest(String url, String namespace, String taskName) throws Throwable {
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(url);

Expand All @@ -89,7 +88,9 @@ public static void startTask(String url, String namespace, String taskName) thro

List<NameValuePair> urlParameters = new ArrayList<NameValuePair>();
urlParameters.add(new BasicNameValuePair("namespace", namespace));
urlParameters.add(new BasicNameValuePair("taskName", taskName));
if (taskName != null && !"".equals(taskName)) {
urlParameters.add(new BasicNameValuePair("taskName", taskName));
}

post.setEntity(new UrlEncodedFormEntity(urlParameters));

Expand All @@ -106,6 +107,6 @@ public static void startTask(String url, String namespace, String taskName) thro
result.append(line);
}

System.out.println(result);
return result.toString();
}
}
31 changes: 31 additions & 0 deletions rsqldb-disk/client/clientExector.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/bin/sh

binDir=$(cd `dirname $0`;pwd)

cd $binDir/..
homeDir=$(pwd)

cd $binDir


mainClass=()
if [ "x$1" == "xsubmitTask" ]; then
mainClass=com.alibaba.rsqldb.client.SubmitTask
java -cp rsqldb-client-1.0.0-SNAPSHOT.jar ${mainClass} ${homeDir} $2
fi

if [ "x$1" == "xstartTask" ]; then
mainClass=com.alibaba.rsqldb.client.StartTask
java -cp rsqldb-client-1.0.0-SNAPSHOT.jar ${mainClass}
fi

if [ "x$1" == "xqueryTask" ]; then
mainClass=com.alibaba.rsqldb.client.QueryTask
java -cp rsqldb-client-1.0.0-SNAPSHOT.jar ${mainClass}
fi

if [ "x$1" == "xstopTask" ]; then
mainClass=com.alibaba.rsqldb.client.StopTask
java -cp rsqldb-client-1.0.0-SNAPSHOT.jar ${mainClass}
fi

6 changes: 4 additions & 2 deletions rsqldb-disk/client/data.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
1,2,3,4,5,6,7,8,9
2,2,3,4,5,6,7,8,9
1,2,3,4
2,2,3,4
3,2,3,4
4,2,3,4
Loading