Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

优化TIS Flink实时增量通道启动速度 #366

Closed
baisui1981 opened this issue Sep 27, 2024 · 1 comment
Closed

优化TIS Flink实时增量通道启动速度 #366

baisui1981 opened this issue Sep 27, 2024 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@baisui1981
Copy link
Member

baisui1981 commented Sep 27, 2024

现象

有同学反馈TIS增量启动速度比较慢,一百张表的增量执行管道需要将近20分钟。

以下是启动执行日志链接:https://github.com/qlangtech/plugins-commercial/issues/1

问题分析:

根据典型的日志:

2024-09-26 21:01:20 WARN  c.qlangtech.tis.plugin.PluginStore- target xstream file is not exist:/home/sysadm/tis-uber/data/cfg_repo/tis_plugin_config/ap/ZF_MySQL_SR_ALL/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml
2024-09-26 21:01:25 INFO  c.q.t.p.ds.BasicDataSourceFactory- tabmeta:PRI_COT_PAK_COTLOOK,colsSize:7,cols:CURDATE,COT_YEAR,PRODUCT_AREA,TYPE,NUMS,CREATE_TIME,UPDATE_TIME

下面那条通过表名获取对应的列元组信息,需要耗费5秒 时间,这个算比较耗费时间的,主要原因是:创建jdbc Connection对象费时,且没有使用dbcp的connection pool来缓存,所以当需要同步的表比较多时的确会比较耗时

优化思路:

在整个启动过程中,引入Connection缓存机制,将对应构建jdbcURL->Connection的 映射关系作为缓存。启动过程只需要构对Connection实例进行一次初始化工作,将会大大缩减启动初始化时间。

@baisui1981 baisui1981 added the enhancement New feature or request label Sep 27, 2024
@baisui1981 baisui1981 added this to the V4.0.1 milestone Sep 27, 2024
baisui1981 added a commit to qlangtech/plugins that referenced this issue Oct 5, 2024
baisui1981 added a commit to qlangtech/plugins that referenced this issue Oct 6, 2024
baisui1981 added a commit that referenced this issue Oct 6, 2024
baisui1981 added a commit to qlangtech/plugins that referenced this issue Oct 7, 2024
baisui1981 added a commit that referenced this issue Oct 7, 2024
@baisui1981
Copy link
Member Author

baisui1981 commented Oct 7, 2024

  • 引入线程绑定JDBC Connection 对象池(com.qlangtech.tis.plugin.ds.JDBCConnectionPool),实现线程级连接器复用

  • 将连接池嵌入TisExceptionInterceptor ,通过以下方式实现线程级复用
    TisExceptionInterceptor:

      try (JDBCConnectionPool jdbcConnectionPool = JDBCConnectionPool.create()) {
        JDBCConnection.connectionPool.set(jdbcConnectionPool);
       // ...
     } finally {
        JDBCConnection.connectionPool.remove();
     }

    以下是 DataSourceFactory中通过对象池获取可用的JDBCConnection实例执行逻辑

    DataSourceFactory:

        public final JDBCConnection getConnection(String jdbcUrl, boolean verify) throws SQLException {
    
          JDBCConnectionPool connectionPool = JDBCConnection.connectionPool.get();
          JDBCConnection conn = null;
          if (connectionPool != null) {
              conn = connectionPool.getConnection(jdbcUrl, verify);
              if (conn == null) {
                  return connectionPool.getConnection(jdbcUrl, verify, (url) -> {
                      try {
                          return createConnection(jdbcUrl, verify);
                      } catch (SQLException e) {
                          throw new RuntimeException(e);
                      }
                  });
              } else {
                  return conn;
              }
          } else {
              return createConnection(jdbcUrl, verify);
          }
      }

baisui1981 added a commit to qlangtech/plugins that referenced this issue Oct 7, 2024
baisui1981 added a commit that referenced this issue Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant