Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [clickhouse] CK连接器任务抽取完毕后无法正常停止 #7897

Closed
3 tasks done
YOMO-Lee opened this issue Oct 24, 2024 · 1 comment · Fixed by #7921
Closed
3 tasks done

[Bug] [clickhouse] CK连接器任务抽取完毕后无法正常停止 #7897

YOMO-Lee opened this issue Oct 24, 2024 · 1 comment · Fixed by #7921

Comments

@YOMO-Lee
Copy link
Contributor

YOMO-Lee commented Oct 24, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

当我通过clickhouse连接器抽取数据的时候,觉得速度比较慢,想调整并行度(虽然看到官网提示clickhouse连接器目前不支持并行),把parallelism 调整为2,这个时候当任务抽取完毕的时候,该任务并不会停止
已经验证过,必现,当我把并行度改为1的时候就没问题

虽然不支持并行度,但是期望调整 parallelism 参数并不会影响任务的正常结束
另外期望能开发该连接器,使其支持并行
可以提供思路,我来改

SeaTunnel Version

2.3.7

SeaTunnel Config

env {
  parallelism = 2
  job.mode = "BATCH"
}


source{
    Clickhouse {
    host = ""
    database = "fdc"
    sql = "select * from fdc.trace_data WHERE traceTime > '2024-10-24 04:00:00' and traceTime < '2024-10-24 04:00:20'"
    username = "reader"
    password = "reader@123"
    server_time_zone = "UTC"
    result_table_name = "fdc.trace_data"
    clickhouse.config = {
	  "max_memory_usage": "128000000000",
	  "max_bytes_before_external_group_by": "50000000000",
      "socket_timeout": "300000"
    }
  }
}


sink {
    jdbc {
        url = "jdbc:oceanbase:///ods"
        driver = "com.oceanbase.jdbc.Driver"
        user = "root"
        password = "Bh%Yac{C9l-~#?~dq[}bSQO5ng8Rr?w"
        compatible_mode = "mysql"
        generate_sink_sql = true
        database = ods
        table = trace_data
        data_save_mode = APPEND_DATA
    }
}

Running Command

./bin/seatunnel.sh --config job/ck2ob.config -e local

Error Exception

任务抽取完毕,但是停不下来

Zeta or Flink or Spark Version

zeta

Java or Scala Version

java

Screenshots

119180a7b9e22a7afeadc960f80a42a

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@YOMO-Lee YOMO-Lee added the bug label Oct 24, 2024
@YOMO-Lee YOMO-Lee changed the title [Bug] [clickhouse] Bug title [Bug] [clickhouse] CK连接器任务抽取完毕后无法正常停止 Oct 24, 2024
@zhilinli123
Copy link
Contributor

I will finish the clickhouse parallelism reading function soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants