Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-endpoint support #956

Merged
merged 10 commits into from
Jun 22, 2022
Merged

Multi-endpoint support #956

merged 10 commits into from
Jun 22, 2022

Conversation

zhicwu
Copy link
Contributor

@zhicwu zhicwu commented Jun 18, 2022

Implements basic load-balancing and failover mentioned in #894. Key changes:

  • the ability to connect to URI
  • optional health-check and node-discovery(using system.clusters table)
  • load-balancing policies: custom, firstAlive, random and roundRobin (server weight, shard num/weight, and replica are currently not considered)
  • failover for connect errors(including connect timed out)

Configuration:

Option Description Default Value Remark
auto_discovery Whether to enable node discovery false
node_discovery_interval Node discovery interval in milliseconds, 0 for one-time discovery 0
node_discovery_limit Maximum number of nodes can be discovered at a time 100
load_balancing_policy Load balancing policy '' use 'firstAlive', 'random', 'roundRobin', or full qualified class name implementing ClickHouseLoadBlanacingPolicy
load_balancing_tags Load balancing tags for filtering nodes ''
health_check_method Health check method SELECT_ONE or PING which might be faster but may not work with middleware like chproxy
health_check_interval Health check interval in milliseconds, 0 for one-time health check 0
check_all_nodes Whether all nodes should be checked or just faulty nodes false
node_check_interval Minimum interval in milliseconds required for checking node status 0
node_group_size Maximum number of nodes can be used in operation at a time 50
failover Maximum times failover allowed 0
retry Maximum times retry allowed 0

Performance:

image

Usage:

  • Java client
    try (ClickHouseClient client = ClickHouseClient.newInstance(ClickHouseProtocol.HTTP);
            ClickHouseResponse response = client.connect(
                    "http://(grpc://myoffice/mydb),localhost,{https://explorer@play.clickhouse.com:443}/system?failover=2")
                    .query("select 1").executeAndWait()) {
        Assert.assertEquals(response.firstRecord().getValue(0).asInteger(), 1);
    }
  • JDBC driver
    try (Connection conn = DriverManager.getConnection(
            "jdbc:ch://127.0.0.1:8128,(tcp://myhome?docker_cli_path=/usr/local/bin/docker),grpc://myhome/my/test/dbx1?decompress=0&failover=2", "default", "")) {
        try (Statement stmt = conn.createStatement();
            ResultSet rs = stmt.executeQuery("select 1")) {
            Assert.assertTrue(rs.next());
            Assert.assertEquals(rs.getInt(1), 1);
        }
    }

@zhicwu zhicwu requested a review from mzitnik June 18, 2022 14:18
@zhicwu zhicwu linked an issue Jun 19, 2022 that may be closed by this pull request
@mzitnik
Copy link
Contributor

mzitnik commented Jun 19, 2022

@zhicwu what was changed that the build it falling?

@zhicwu zhicwu marked this pull request as draft June 20, 2022 10:46
@zhicwu zhicwu marked this pull request as ready for review June 20, 2022 14:10
@zhicwu
Copy link
Contributor Author

zhicwu commented Jun 20, 2022

@zhicwu what was changed that the build it falling?

The initial build failure was related to how we handle null parameter for insertion. It was impacted by fundamental change s being made to pass ClickHouseConfig among ClickHouseNode, ClickHouseClient, and ClickHouseRequest. This resulted in incorrect format(TabSepartedWithNamesAndTypes instead of TabSeprated) used in testInsertDefaultValue.

@zhicwu zhicwu linked an issue Jun 20, 2022 that may be closed by this pull request
@zhicwu zhicwu marked this pull request as draft June 22, 2022 12:37
@zhicwu
Copy link
Contributor Author

zhicwu commented Jun 22, 2022

The build failure(session locked) happens occasionally in the past(see ClickHouse/ClickHouse#35948), which is irrelevant to this PR.

@Yuzer-Ly
Copy link

@zhicwu I am using the driver version 0.4.6, Why does each request use the same endpoint? and it changed when i restart my application. my conf like that:

spring:
  datasource:
     dynamic:
         datasource:
             clickhouse:
                url: jdbc:ch://ip1:8123,ip2:8123,ip3:8123?load_balancing_policy=random
                username: username
                password: password
                driver-class-name: com.clickhouse.jdbc.ClickHouseDriver
                hikari:
                   connection-timeout: 30000
                   idle-timeout: 60000
                   minimum-idle: 5
                   maximum-pool-size: 50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Question about BalancedClickhouseDataSource How to set JDBC connection string for Clickhouse cluster
3 participants