Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

通过querySql中where条件实现增量抽取,flink 脚本支持SQL并且让用户自行修改 #160

Closed
sszc opened this issue Nov 9, 2022 · 2 comments

Comments

@sszc
Copy link

sszc commented Nov 9, 2022

因为生产环境并不会所有mysql数据库都会开通binlog, 就导致不能使用Flink CDC方式增量抽取。
如:
select a, b, c from table where update_time > ${exec_time}

@baisui1981
Copy link
Member

baisui1981 commented Nov 10, 2022

在实现hudi Sink中有一个 基于SQL表的实现 https://www.processon.com/diagraming/5f1633e41e08533a6278cbf7

在界面中用SQL的方式和用户进行交互,这样可以在sql中设置where条件(有一个分险,用户错误填写会导致增量任务启动失败)

在实现端可以 继承于该类 https://github.com/qlangtech/plugins/blob/master/tis-incr/tis-flink-extends/src/main/java/com/qlangtech/tis/realtime/TableRegisterFlinkSourceHandle.java 来实现功能

@baisui1981 baisui1981 changed the title 通过querySql中where条件实现增量抽取 通过querySql中where条件实现增量抽取,flink 脚本支持SQL并且让用户自行修改 Nov 21, 2022
@baisui1981
Copy link
Member

所有chunjun的 sink类型已经支持 flink SQL 写出

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants