Retry on remote database transient issues in JDBC connectors #23302

kokosing · 2024-09-05T13:27:53Z

This PR is based on #23330

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/CachingJdbcClient.java

hashhar

In past we also discussed implementing the retry in the JdbcRecordCursor IIRC but that one would only cover retries during table scans, so metadata etc. would not be retried. I like this approach better.

re: nested retries, why do you say max time we could retry would be 30s. Can't it be the case that connection is retried and then operation is also retried?

hashhar · 2024-09-06T04:40:05Z

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

Reminder to self. Revisit this file.

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingModule.java

kokosing · 2024-09-06T07:58:34Z

re: nested retries, why do you say max time we could retry would be 30s. Can't it be the case that connection is retried and then operation is also retried?

Yes, it can be like that, but max duration is measured on the top level. I mean they should not multiply. They can however sum a bit. The most outer retry won't retry if the operation takes longer than max duration.

    public static void main(String[] args)
    {
        System.out.println(Instant.now());
        RetryPolicy<Object> policy = new RetryingModule().createRetryPolicy(ImmutableSet.of(new RetryOnSqlTransientException()));
        try {
            retry(
                    policy,
                    () -> retry(
                            policy,
                            () -> retry(
                                    policy,
                                    () -> retry(
                                            policy,
                                            retry(
                                                    policy,
                                                    () -> {throw new SQLTransientException();})))));
        } finally {
            System.out.println(Instant.now());
        }
    }

Returns:

2024-09-06T07:53:53.048912Z
2024-09-06T07:54:44.319246Z
Exception in thread "main" dev.failsafe.FailsafeException: java.sql.SQLTransientException
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:196)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.trino.plugin.jdbc.RetryingModule.retry(RetryingModule.java:85)
	...

kokosing · 2024-09-07T22:01:16Z

re: nested retries, why do you say max time we could retry would be 30s. Can't it be the case that connection is retried and then operation is also retried?

@hashhar I added explanation to the commit message:

Here we can have nested retrying. One in jdbc client and other in connection factory.
It is ok as as in both places we are using the retry policy with max duration to 30
seconds. The outer retries won't retry if the operation takes longer than 30s.

plugin/trino-base-jdbc/src/test/java/io/trino/plugin/jdbc/TestRetryingConnectionFactory.java

chenjian2664 · 2024-09-10T02:21:13Z

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/JdbcRecordSet.java

@@ -60,6 +71,6 @@ public List<Type> getColumnTypes()
    @Override
    public RecordCursor cursor()
    {
-        return new JdbcRecordCursor(jdbcClient, executor, session, split, table, columnHandles);
+        return retry(policy, () -> new JdbcRecordCursor(jdbcClient, executor, session, split, table, columnHandles));


Question here, why we do not use RetryJdbcClient directly
The JdbcRecordCursor will use the RetryJdbcClient itself.

jdbcClient is already here wrapped with RetryingJdbcClient. The issue here is that JdbcRecordCursor is using jdbcClient in a way that it is not enough. For example you cannot retry jdbc query using the same connection. The connection could be already invalid. Hence we need to retry on higher level.

hashhar · 2024-09-13T07:20:23Z

They can however sum a bit. The most outer retry won't retry if the operation takes longer than max duration.

This is basically what I was thinking of. Thanks for answering that, the updated commit message makes it clear for me.

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

hashhar

looks good % cb2cd50#r1758332145

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

hashhar · 2024-09-13T08:04:50Z

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

+        // no retrying as it could be not idempotent operation
+        delegate.setColumnType(session, handle, column, type);


setting column type should be idempotent actually since we only specify the target type

hashhar · 2024-09-13T08:05:31Z

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

+        // no retrying as it could be not idempotent operation
+        delegate.dropNotNullConstraint(session, handle, column);


safe to retry but the code needs to handle exceptions if the constraint was dropped already (e.g. timeouts)

So let leave it as follow. I will update the comment then.

github-actions · 2024-10-04T17:14:46Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

kokosing

@hashhar addressed your comments, I will be merging this once CI is passing. Please take a look at the fixup commit.

kokosing · 2024-10-10T14:22:26Z

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

+        // no retrying as it could be not idempotent operation
+        delegate.setColumnType(session, handle, column, type);


kokosing · 2024-10-10T14:31:27Z

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java

+        // no retrying as it could be not idempotent operation
+        delegate.dropNotNullConstraint(session, handle, column);


So let leave it as follow. I will update the comment then.

So far JDBC connectors were able to retry in case of transient issues when establishing connections. RetryingJdbcClient is able to retry an operation on remote database during other situations. Here we can have nested retrying. One in jdbc client and other in connection factory. It is ok as as in both places we are using the retry policy with max duration to 30 seconds. The outer retries won't retry if the operation takes longer than 30s.

Creating PreparedStatement can fail due transient issues in remote database. Let's retry it in the same as in JdbcClient.

hashhar · 2024-10-14T12:25:28Z

Thank you @kokosing

cla-bot bot added the cla-signed label Sep 5, 2024

kokosing requested review from ssheikin, ebyhr, hashhar, SemionPar and Praveen2112 September 5, 2024 13:28

kokosing force-pushed the origin/master/003_retry branch 3 times, most recently from 8a83fb8 to 935f9c7 Compare September 5, 2024 14:53

kokosing requested a review from chenjian2664 September 5, 2024 14:57

kokosing commented Sep 5, 2024

View reviewed changes

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java Show resolved Hide resolved

ssheikin reviewed Sep 5, 2024

View reviewed changes

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/CachingJdbcClient.java Outdated Show resolved Hide resolved

hashhar reviewed Sep 6, 2024

View reviewed changes

kokosing force-pushed the origin/master/003_retry branch 3 times, most recently from b989b3b to d8006b6 Compare September 7, 2024 22:00

kokosing force-pushed the origin/master/003_retry branch 5 times, most recently from bcf2be4 to 84d5ef2 Compare September 9, 2024 21:55

kokosing commented Sep 9, 2024

View reviewed changes

plugin/trino-base-jdbc/src/test/java/io/trino/plugin/jdbc/TestRetryingConnectionFactory.java Show resolved Hide resolved

chenjian2664 reviewed Sep 10, 2024

View reviewed changes

kokosing force-pushed the origin/master/003_retry branch from 84d5ef2 to eaa9544 Compare September 12, 2024 19:48

hashhar reviewed Sep 13, 2024

View reviewed changes

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java Outdated Show resolved Hide resolved

hashhar reviewed Sep 13, 2024

View reviewed changes

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java Outdated Show resolved Hide resolved

hashhar reviewed Sep 13, 2024

View reviewed changes

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java Show resolved Hide resolved

hashhar reviewed Sep 13, 2024

View reviewed changes

plugin/trino-base-jdbc/src/main/java/io/trino/plugin/jdbc/RetryingJdbcClient.java Show resolved Hide resolved

hashhar reviewed Sep 13, 2024

View reviewed changes

hashhar approved these changes Sep 13, 2024

View reviewed changes

github-actions bot added the stale label Oct 4, 2024

kokosing force-pushed the origin/master/003_retry branch from eaa9544 to 2873f5e Compare October 10, 2024 14:32

kokosing commented Oct 10, 2024

View reviewed changes

github-actions bot removed the stale label Oct 10, 2024

kokosing force-pushed the origin/master/003_retry branch from 2873f5e to 751bcf2 Compare October 11, 2024 11:49

kokosing added 2 commits October 11, 2024 15:22

Retry creating JdbcRecordCursor

87122c1

Creating PreparedStatement can fail due transient issues in remote database. Let's retry it in the same as in JdbcClient.

kokosing force-pushed the origin/master/003_retry branch from 751bcf2 to 87122c1 Compare October 11, 2024 13:22

kokosing merged commit 044f85e into trinodb:master Oct 12, 2024
61 checks passed

github-actions bot added this to the 462 milestone Oct 12, 2024

kokosing deleted the origin/master/003_retry branch October 12, 2024 17:57

mosabua mentioned this pull request Oct 15, 2024

Add Trino 462 release notes #23745

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry on remote database transient issues in JDBC connectors #23302

Retry on remote database transient issues in JDBC connectors #23302

kokosing commented Sep 5, 2024 •

edited

Loading

hashhar left a comment

hashhar Sep 6, 2024

kokosing commented Sep 6, 2024

kokosing commented Sep 7, 2024

chenjian2664 Sep 10, 2024

kokosing Sep 12, 2024

hashhar commented Sep 13, 2024

hashhar left a comment

hashhar Sep 13, 2024

kokosing Oct 10, 2024

hashhar Sep 13, 2024

kokosing Oct 10, 2024

github-actions bot commented Oct 4, 2024

kokosing left a comment

kokosing Oct 10, 2024

kokosing Oct 10, 2024

hashhar commented Oct 14, 2024

		// no retrying as it could be not idempotent operation
		delegate.setColumnType(session, handle, column, type);

		// no retrying as it could be not idempotent operation
		delegate.dropNotNullConstraint(session, handle, column);

Retry on remote database transient issues in JDBC connectors #23302

Retry on remote database transient issues in JDBC connectors #23302

Conversation

kokosing commented Sep 5, 2024 • edited Loading

hashhar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kokosing commented Sep 6, 2024

kokosing commented Sep 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashhar commented Sep 13, 2024

hashhar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 4, 2024

kokosing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashhar commented Oct 14, 2024

kokosing commented Sep 5, 2024 •

edited

Loading