Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Playground: Restructure directories and upgrade components #212

Merged
merged 3 commits into from
Jan 31, 2023
Merged

Playground: Restructure directories and upgrade components #212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5d0dff9
Playground: Restructure directories and upgrade components
pan3793 Jan 31, 2023
6255a6d
nit
pan3793 Jan 31, 2023
e965ff4
nit
pan3793 Jan 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion docker/.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,17 @@
PROJECT_VERSION=0.5.0

CLICKHOUSE_IMAGE=clickhouse/clickhouse-server:22.3.3.44
AWS_JAVA_SDK_VERSION=1.12.239
CLICKHOUSE_JDBC_VERSION=0.3.2-patch11
HADOOP_VERSION=3.3.1
HIVE_VERSION=2.3.9
ICEBERG_VERSION=1.1.0
KYUUBI_VERSION=1.6.1-incubating
KYUUBI_HADOOP_VERSION=3.3.4
POSTGRES_VERSION=12
POSTGRES_JDBC_VERSION=42.3.4
SCALA_BINARY_VERSION=2.12
SPARK_VERSION=3.3.0
SPARK_VERSION=3.3.1
SPARK_BINARY_VERSION=3.3
SPARK_HADOOP_VERSION=3.3.2
ZOOKEEPER_VERSION=3.6.3
11 changes: 10 additions & 1 deletion docker/.env-dev
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,17 @@
PROJECT_VERSION=0.6.0-SNAPSHOT

CLICKHOUSE_IMAGE=clickhouse/clickhouse-server:22.3.3.44
AWS_JAVA_SDK_VERSION=1.12.239
CLICKHOUSE_JDBC_VERSION=0.3.2-patch11
HADOOP_VERSION=3.3.1
HIVE_VERSION=2.3.9
ICEBERG_VERSION=1.1.0
KYUUBI_VERSION=1.6.1-incubating
KYUUBI_HADOOP_VERSION=3.3.4
POSTGRES_VERSION=12
POSTGRES_JDBC_VERSION=42.3.4
SCALA_BINARY_VERSION=2.12
SPARK_VERSION=3.3.0
SPARK_VERSION=3.3.1
SPARK_BINARY_VERSION=3.3
SPARK_HADOOP_VERSION=3.3.2
ZOOKEEPER_VERSION=3.6.3
11 changes: 6 additions & 5 deletions docker/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Playground
### Setup

1. Install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/);
2. Start compose services `cd docker && docker compose up`;
2. Start compose services `docker compose --file docker/compose.yml up`;

### Play

Expand Down Expand Up @@ -42,7 +42,7 @@ Add a new Hive or Spark SQL or Kyuubi datasource with
### Shutdown

1. Stop the compose services by pressing `CTRL+C`;
2. Remove the stopped containers `docker compose rm`;
2. Remove the stopped containers `docker compose --file docker/compose.yml rm`;

## For Developers

Expand All @@ -51,12 +51,13 @@ In addition to play with the pre-build images, developers may want to build the
### Build

1. Follow [README](../README.md#build) to build the project;
2. Build images `docker/build-image.sh`;
3. Optional to use buildx to build cross-platform images `BUILDX=1 docker/build-image.sh`;
2. Build images for release `docker/build-image.sh`;
3. Build images for dev `DEV=1 docker/build-image.sh`;
4. Optional to use buildx to build cross-platform images `BUILDX=1 docker/build-image.sh`;

### Setup

1. Launch dev compose services `docker compose --file compose-dev.yml --env-file .env-dev up`;
1. Launch dev compose services `docker compose --file docker/compose-dev.yml --env-file docker/.env-dev up`;

## For Maintainers

Expand Down
28 changes: 10 additions & 18 deletions docker/build-image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,24 @@

set -e

BUILD_CMD="docker build"
APACHE_MIRROR=${APACHE_MIRROR:-https://dlcdn.apache.org}
MAVEN_MIRROR=${MAVEN_MIRROR:-https://repo1.maven.org/maven2}

BUILD_CMD="docker build"
if [ $BUILDX ]; then
echo "Using buildx to build cross-platform images"
BUILD_CMD="docker buildx build --platform=linux/amd64,linux/arm64 --push"
fi

SELF_DIR="$(cd "$(dirname "$0")"; pwd)"

PROJECT_VERSION="$(cat "${SELF_DIR}/../version.txt")"

APACHE_MIRROR=${APACHE_MIRROR:-https://dlcdn.apache.org}
MAVEN_MIRROR=${MAVEN_MIRROR:-https://repo1.maven.org/maven2}

AWS_JAVA_SDK_VERSION=1.12.239
CLICKHOUSE_JDBC_VERSION=0.3.2-patch11
HADOOP_VERSION=3.3.1
HIVE_VERSION=2.3.9
ICEBERG_VERSION=0.14.0
KYUUBI_VERSION=1.6.0-incubating
KYUUBI_HADOOP_VERSION=3.3.4
POSTGRES_JDBC_VERSION=42.3.4
SCALA_BINARY_VERSION=2.12
SPARK_VERSION=3.3.0
SPARK_BINARY_VERSION=3.3
SPARK_HADOOP_VERSION=3.3.2
if [ $DEV ]; then
echo "Loading .env-dev"
source "${SELF_DIR}/.env-dev"
else
echo "Loading .env"
source "${SELF_DIR}/.env"
fi

${BUILD_CMD} \
--build-arg APACHE_MIRROR=${APACHE_MIRROR} \
Expand Down
24 changes: 12 additions & 12 deletions docker/compose-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ services:
ports:
- 5432
volumes:
- ./image/hive-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-schema-2.3.0.postgres.sql
- ./image/hive-txn-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-txn-schema-2.3.0.postgres.sql
- ./script/hive-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-schema-2.3.0.postgres.sql
- ./script/hive-txn-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-txn-schema-2.3.0.postgres.sql

zookeeper:
image: zookeeper:${ZOOKEEPER_VERSION}
Expand Down Expand Up @@ -123,8 +123,8 @@ services:
ports:
- 9083
volumes:
- ./image/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./image/hive-site.xml:/etc/hive/conf/hive-site.xml
- ./conf/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./conf/hive-site.xml:/etc/hive/conf/hive-site.xml
depends_on:
- postgres

Expand All @@ -136,10 +136,10 @@ services:
- 4040-4050:4040-4050
- 10009:10009
volumes:
- ./image/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./image/hive-site.xml:/etc/hive/conf/hive-site.xml
- ./image/spark-defaults.conf:/etc/spark/conf/spark-defaults.conf
- ./image/kyuubi-defaults.conf:/etc/kyuubi/conf/kyuubi-defaults.conf
- ./conf/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./conf/hive-site.xml:/etc/hive/conf/hive-site.xml
- ./conf/spark-defaults.conf:/etc/spark/conf/spark-defaults.conf
- ./conf/kyuubi-defaults.conf:/etc/kyuubi/conf/kyuubi-defaults.conf
- ../spark-${SPARK_BINARY_VERSION}/clickhouse-spark-runtime/build/libs/clickhouse-spark-runtime-${SPARK_BINARY_VERSION}_${SCALA_BINARY_VERSION}-${PROJECT_VERSION}.jar:/opt/spark/jars/clickhouse-spark-runtime-${SPARK_BINARY_VERSION}_${SCALA_BINARY_VERSION}-${PROJECT_VERSION}.jar
depends_on:
- clickhouse-s1r1
Expand All @@ -162,9 +162,9 @@ services:
ports:
- 8978:8978
volumes:
- ./image/cloudbeaver-conf/cloudbeaver.conf:/opt/cloudbeaver/conf/cloudbeaver.conf
- ./image/cloudbeaver-conf/initial-data.conf:/opt/cloudbeaver/conf/initial-data.conf
- ./image/cloudbeaver-conf/initial-data-sources.conf:/opt/cloudbeaver/conf/initial-data-sources.conf
- ./image/cloudbeaver-conf/product.conf:/opt/cloudbeaver/conf/product.conf
- ./conf/cloudbeaver-conf/cloudbeaver.conf:/opt/cloudbeaver/conf/cloudbeaver.conf
- ./conf/cloudbeaver-conf/initial-data.conf:/opt/cloudbeaver/conf/initial-data.conf
- ./conf/cloudbeaver-conf/initial-data-sources.conf:/opt/cloudbeaver/conf/initial-data-sources.conf
- ./conf/cloudbeaver-conf/product.conf:/opt/cloudbeaver/conf/product.conf
depends_on:
- postgres
24 changes: 12 additions & 12 deletions docker/compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ services:
ports:
- 5432
volumes:
- ./image/hive-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-schema-2.3.0.postgres.sql
- ./image/hive-txn-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-txn-schema-2.3.0.postgres.sql
- ./script/hive-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-schema-2.3.0.postgres.sql
- ./script/hive-txn-schema-2.3.0.postgres.sql:/docker-entrypoint-initdb.d/hive-txn-schema-2.3.0.postgres.sql

zookeeper:
image: zookeeper:${ZOOKEEPER_VERSION}
Expand Down Expand Up @@ -123,8 +123,8 @@ services:
ports:
- 9083
volumes:
- ./image/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./image/hive-site.xml:/etc/hive/conf/hive-site.xml
- ./conf/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./conf/hive-site.xml:/etc/hive/conf/hive-site.xml
depends_on:
- postgres

Expand All @@ -136,10 +136,10 @@ services:
- 4040-4050:4040-4050
- 10009:10009
volumes:
- ./image/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./image/hive-site.xml:/etc/hive/conf/hive-site.xml
- ./image/spark-defaults.conf:/etc/spark/conf/spark-defaults.conf
- ./image/kyuubi-defaults.conf:/etc/kyuubi/conf/kyuubi-defaults.conf
- ./conf/core-site.xml:/etc/hadoop/conf/core-site.xml
- ./conf/hive-site.xml:/etc/hive/conf/hive-site.xml
- ./conf/spark-defaults.conf:/etc/spark/conf/spark-defaults.conf
- ./conf/kyuubi-defaults.conf:/etc/kyuubi/conf/kyuubi-defaults.conf
depends_on:
- clickhouse-s1r1
- clickhouse-s1r2
Expand All @@ -161,9 +161,9 @@ services:
ports:
- 8978:8978
volumes:
- ./image/cloudbeaver-conf/cloudbeaver.conf:/opt/cloudbeaver/conf/cloudbeaver.conf
- ./image/cloudbeaver-conf/initial-data.conf:/opt/cloudbeaver/conf/initial-data.conf
- ./image/cloudbeaver-conf/initial-data-sources.conf:/opt/cloudbeaver/conf/initial-data-sources.conf
- ./image/cloudbeaver-conf/product.conf:/opt/cloudbeaver/conf/product.conf
- ./conf/cloudbeaver-conf/cloudbeaver.conf:/opt/cloudbeaver/conf/cloudbeaver.conf
- ./conf/cloudbeaver-conf/initial-data.conf:/opt/cloudbeaver/conf/initial-data.conf
- ./conf/cloudbeaver-conf/initial-data-sources.conf:/opt/cloudbeaver/conf/initial-data-sources.conf
- ./conf/cloudbeaver-conf/product.conf:/opt/cloudbeaver/conf/product.conf
depends_on:
- postgres
0 docker/image/cloudbeaver-conf/README.md → docker/conf/cloudbeaver-conf/README.md
View file
Open in desktop
File renamed without changes.
0 ...r/image/cloudbeaver-conf/cloudbeaver.conf → ...er/conf/cloudbeaver-conf/cloudbeaver.conf
View file
Open in desktop
File renamed without changes.
0 ...loudbeaver-conf/initial-data-sources.conf → ...loudbeaver-conf/initial-data-sources.conf
View file
Open in desktop
File renamed without changes.
0 .../image/cloudbeaver-conf/initial-data.conf → ...r/conf/cloudbeaver-conf/initial-data.conf
View file
Open in desktop
File renamed without changes.
0 docker/image/cloudbeaver-conf/product.conf → docker/conf/cloudbeaver-conf/product.conf
View file
Open in desktop
File renamed without changes.
0 docker/image/core-site.xml → docker/conf/core-site.xml
View file
Open in desktop
File renamed without changes.
0 docker/image/hive-site.xml → docker/conf/hive-site.xml
View file
Open in desktop
File renamed without changes.
0 docker/image/kyuubi-defaults.conf → docker/conf/kyuubi-defaults.conf
View file
Open in desktop
File renamed without changes.
0 docker/image/spark-defaults.conf → docker/conf/spark-defaults.conf
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion docker/image/scc-base.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# limitations under the License.

FROM eclipse-temurin:8-focal
LABEL org.opencontainers.image.authors="Cheng Pan<chengpan@apache.com>"
LABEL org.opencontainers.image.authors="Cheng Pan<chengpan@apache.org>"

RUN set -x && \
ln -snf /usr/bin/bash /usr/bin/sh && \
Expand Down
2 changes: 1 addition & 1 deletion docker/image/scc-kyuubi.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ENV KYUUBI_HOME=/opt/kyuubi
ENV KYUUBI_CONF_DIR=/etc/kyuubi/conf

RUN set -x && \
wget -q ${APACHE_MIRROR}/incubator/kyuubi/kyuubi-${KYUUBI_VERSION}/apache-kyuubi-${KYUUBI_VERSION}-bin.tgz && \
wget -q ${APACHE_MIRROR}/kyuubi/kyuubi-${KYUUBI_VERSION}/apache-kyuubi-${KYUUBI_VERSION}-bin.tgz && \
tar -xzf apache-kyuubi-${KYUUBI_VERSION}-bin.tgz -C /opt && \
ln -s /opt/apache-kyuubi-${KYUUBI_VERSION}-bin ${KYUUBI_HOME} && \
rm apache-kyuubi-${KYUUBI_VERSION}-bin.tgz && \
Expand Down
0 docker/image/hive-schema-2.3.0.postgres.sql → docker/script/hive-schema-2.3.0.postgres.sql
View file
Open in desktop
File renamed without changes.
0 .../image/hive-txn-schema-2.3.0.postgres.sql → ...script/hive-txn-schema-2.3.0.postgres.sql
View file
Open in desktop
File renamed without changes.