Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Deprecate gRPC protocol #233

Merged
merged 1 commit into from
Apr 27, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Notes:
[ClickHouse Official Java Client](https://github.com/ClickHouse/clickhouse-jdbc), which brings HTTP protocol support,
extends the range of supported versions of ClickHouse Server.
2. Since 0.6.0, HTTP becomes the default protocol.
3. Since 0.7.0, gRPC is deprecated and not recommended, it may be removed in the future.

## Compatible Matrix

Expand Down
12 changes: 6 additions & 6 deletions docs/configurations/01_catalog_configurations.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,25 +49,25 @@ Then you can access ClickHouse table `<ck_db>.<ck_table>` from Spark SQL by usin

For ClickHouse cluster, give an unique catalog name for each instances.

Suppose you have two ClickHouse instances, one installed on `10.0.0.1` and exposes gRPC on port `9100` named
clickhouse1, and another installed on `10.0.0.2` and exposes gRPC on port `9100` named clickhouse2.
Suppose you have two ClickHouse instances, one installed on `10.0.0.1` and exposes HTTP on port `8123` named
clickhouse1, and another installed on `10.0.0.2` and exposes HTTP on port `8123` named clickhouse2.

Edit `$SPARK_HOME/conf/spark-defaults.conf`.

```
spark.sql.catalog.clickhouse1 xenon.clickhouse.ClickHouseCatalog
spark.sql.catalog.clickhouse1.host 10.0.0.1
spark.sql.catalog.clickhouse1.protocol grpc
spark.sql.catalog.clickhouse1.grpc_port 9100
spark.sql.catalog.clickhouse1.protocol http
spark.sql.catalog.clickhouse1.http_port 8123
spark.sql.catalog.clickhouse1.user default
spark.sql.catalog.clickhouse1.password
spark.sql.catalog.clickhouse1.database default
spark.sql.catalog.clickhouse1.option.async false

spark.sql.catalog.clickhouse2 xenon.clickhouse.ClickHouseCatalog
spark.sql.catalog.clickhouse2.host 10.0.0.2
spark.sql.catalog.clickhouse2.protocol grpc
spark.sql.catalog.clickhouse2.grpc_port 9100
spark.sql.catalog.clickhouse2.protocol http
spark.sql.catalog.clickhouse2.http_port 8123
spark.sql.catalog.clickhouse2.user default
spark.sql.catalog.clickhouse2.password
spark.sql.catalog.clickhouse2.database default
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Spark ClickHouse Connector is a high performance connector build on top of Spark
3. An available Spark cluster, and Spark version should be 3.3 or above, because we need the interfaces of Spark DataSource V2
added in 3.3.0.
4. Make sure your network policy satisfies the following requirements, both driver and executor of Spark need to access
ClickHouse HTTP/gRPC port. If you are using it to access ClickHouse cluster, ensure the connectivity between driver and
ClickHouse HTTP port. If you are using it to access ClickHouse cluster, ensure the connectivity between driver and
executor of Spark and each node of ClickHouse cluster.

## Notes
Expand Down
2 changes: 1 addition & 1 deletion docs/internals/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Overview Design
===

In high level, Spark ClickHouse Connector is a connector build on top of Spark DataSource V2 and
ClickHouse gRPC protocol.
ClickHouse HTTP protocol.

<figure markdown>
![Overview](../imgs/scc_overview.drawio.png)
Expand Down
2 changes: 1 addition & 1 deletion docs/quick_start/02_play_with_spark_sql.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ $SPARK_HOME/bin/spark-sql \
--conf spark.sql.catalog.clickhouse=xenon.clickhouse.ClickHouseCatalog \
--conf spark.sql.catalog.clickhouse.host=${CLICKHOUSE_HOST:-127.0.0.1} \
--conf spark.sql.catalog.clickhouse.protocol=http \
--conf spark.sql.catalog.clickhouse.http_port=${CLICKHOUSE_GRPC_PORT:-8123} \
--conf spark.sql.catalog.clickhouse.http_port=${CLICKHOUSE_HTTP_PORT:-8123} \
--conf spark.sql.catalog.clickhouse.user=${CLICKHOUSE_USER:-default} \
--conf spark.sql.catalog.clickhouse.password=${CLICKHOUSE_PASSWORD:-} \
--conf spark.sql.catalog.clickhouse.database=default \
Expand Down
8 changes: 4 additions & 4 deletions docs/quick_start/03_play_with_spark_shell.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ Play with Spark Shell
$SPARK_HOME/bin/spark-shell \
--conf spark.sql.catalog.clickhouse=xenon.clickhouse.ClickHouseCatalog \
--conf spark.sql.catalog.clickhouse.host=${CLICKHOUSE_HOST:-127.0.0.1} \
--conf spark.sql.catalog.clickhouse.protocol=grpc \
--conf spark.sql.catalog.clickhouse.grpc_port=${CLICKHOUSE_GRPC_PORT:-9100} \
--conf spark.sql.catalog.clickhouse.protocol=http \
--conf spark.sql.catalog.clickhouse.http_port=${CLICKHOUSE_HTTP_PORT:-8123} \
--conf spark.sql.catalog.clickhouse.user=${CLICKHOUSE_USER:-default} \
--conf spark.sql.catalog.clickhouse.password=${CLICKHOUSE_PASSWORD:-} \
--conf spark.sql.catalog.clickhouse.database=default \
Expand Down Expand Up @@ -104,8 +104,8 @@ Execute ClickHouse native SQL.
```
scala> val options = Map(
| "host" -> "clickhouse",
| "protocol" -> "grpc",
| "grpc_port" -> "9100",
| "protocol" -> "http",
| "http_port" -> "8123",
| "user" -> "default",
| "password" -> ""
| )
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

package xenon.clickhouse

import com.clickhouse.client.ClickHouseProtocol
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.clickhouse.{ExprUtils, SchemaUtils}
import org.apache.spark.sql.connector.catalog._
Expand Down Expand Up @@ -64,6 +65,9 @@ class ClickHouseCatalog extends TableCatalog
override def initialize(name: String, options: CaseInsensitiveStringMap): Unit = {
this.catalogName = name
this.nodeSpec = buildNodeSpec(options)
if (nodeSpec.protocol == ClickHouseProtocol.GRPC) {
log.warn("gPRC is deprecated and not recommended since v0.7.0, it may be removed in the future.")
}
this.currentDb = nodeSpec.database
this.nodeClient = NodeClient(nodeSpec)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

package xenon.clickhouse

import com.clickhouse.client.ClickHouseProtocol
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.clickhouse.{ExprUtils, SchemaUtils}
import org.apache.spark.sql.connector.catalog._
Expand Down Expand Up @@ -64,6 +65,9 @@ class ClickHouseCatalog extends TableCatalog
override def initialize(name: String, options: CaseInsensitiveStringMap): Unit = {
this.catalogName = name
this.nodeSpec = buildNodeSpec(options)
if (nodeSpec.protocol == ClickHouseProtocol.GRPC) {
log.warn("gPRC is deprecated and not recommended since v0.7.0, it may be removed in the future.")
}
this.currentDb = nodeSpec.database
this.nodeClient = NodeClient(nodeSpec)

Expand Down