[Transaction] Fix transaction buffer client handle endTxn op when topic or sub have been deleted. #11304

congbobo184 · 2021-07-13T11:56:50Z

Motivation

now, when topic have been deleted we will not handle the endTxnOnTopic and endTxnOnSub.
now, when sub have been deleted we will not handle the endTxnOnSub.

when topic or sub have been deleted, we should return success to tc on endTxnOnTopic and endTxnOnSub operation.

implement

when topic not exist in this broker, we should judge the topic if it have been created. if not created, we should return success.

Verifying this change

Add the tests for it

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

eolivelli · 2021-07-13T14:06:50Z

...r/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferClientTest.java

+        admin.topics().createSubscription(topic + "_commit_sub", subName, MessageId.earliest);
+
+
+        Awaitility.await().until(() -> {


why do we have to retry this operation (and the commit below) ?

waitting pending ack init.

codelipenghui · 2021-07-13T15:15:16Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

@@ -1930,7 +1930,7 @@ protected void handleEndTxnOnPartition(CommandEndTxnOnPartition command) {
            log.debug("[{}] handleEndTxnOnPartition txnId: [{}], txnAction: [{}]", topic,
                    txnID, txnAction);
        }
-        CompletableFuture<Optional<Topic>> topicFuture = service.getTopics().get(TopicName.get(topic).toString());
+        CompletableFuture<Optional<Topic>> topicFuture = service.getTopic(topic, true);


Users can disable the topic auto-creation, this means they want to prevent the topic auto-creation. After this change, the TC might re-create a topic even if the topic has been deleted definitely. This will break the behavior of topic auto-creation disabled.

I think the current fix is:

If the topic has been deleted, we should skip all the TC logs of this topic.

If the topic has not been deleted but only not loaded by any broker, we can load the topic and complete the transaction.

WDYT @congbobo184

@congbobo184 I am cutting 2.8.1, would you please address this comment?

congbobo184 · 2021-08-06T10:10:49Z

Hi @eolivelli @codelipenghui please review again. thanks:)

eolivelli

I am not still very comfortable with those wait loops.

Can you please check the latest utility methods we added in order to wait for the TC to be ready?

If we do not have some useful utility, can you please add a new utility method?

eolivelli · 2021-08-06T15:15:47Z

...r/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferClientTest.java

+        admin.topics().createNonPartitionedTopic(topic + "_commit_sub");
+        admin.topics().createSubscription(topic + "_commit_sub", subName, MessageId.earliest);
+
+        Awaitility.await().until(() -> {


There is some utility method to wait for the coordinator to be started.
Can we use it?

It is not clear why we need this wait loop.
It may hide many other internal errors.
This is a test case and we should have full control over what's going on

adopt your comment, thank for your comments. please review again

eolivelli

Lgtm

…op_init_topic # Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferClientTest.java

codelipenghui · 2021-08-09T03:30:14Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactory.java

+     *         an instance of {@link Boolean} is returned
+     *         if the operation succeeds.
+     */
+    CompletableFuture<Boolean> checkManagedLedgerInitializedBefore(String name);


Suggested change

CompletableFuture<Boolean> checkManagedLedgerInitializedBefore(String name);

CompletableFuture<Boolean> asyncExists(String name);

codelipenghui · 2021-08-09T03:32:16Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStore.java

+     *         an instance of {@link Boolean} is returned
+     *         if the operation succeeds.
+     */
+    CompletableFuture<Boolean> exists(String path);


Suggested change

CompletableFuture<Boolean> exists(String path);

CompletableFuture<Boolean> asyncExists(String ledgerName);

…op_init_topic

eolivelli

overall LGTM

I left some suggestions if you want to make some improvements

eolivelli · 2021-08-09T09:45:12Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

+                                        txnID, TxnAction.valueOf(txnAction));
+                                ctx.writeAndFlush(Commands.newEndTxnOnPartitionResponse(requestId,
+                                        ServerError.ServiceNotReady,
+                                        "The topic " + topic + " is not exist in broker.",


there is a typo, what about: "The topic " + topic + " does not exist in broker."

eolivelli · 2021-08-09T09:45:33Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

+                                        "The topic " + topic + " is not exist in broker.",
+                                        txnID.getMostSigBits(), txnID.getLeastSigBits()));
+                            } else {
+                                log.warn("handleEndTxnOnPartition fail ! The topic {} have not been created, "


typo: "has not been created"

eolivelli · 2021-08-09T09:46:02Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

-                            "Topic " + optionalTopic.get().getName()
-                                    + " subscription " + subName + " is not exist."));
+                    log.warn("handleEndTxnOnSubscription fail! "
+                                    + "topic {} subscription {} is not exist. txnId: [{}], txnAction: [{}]",


"is not exist" -> "does not exist"

eolivelli · 2021-08-09T09:47:14Z

...r/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferClientTest.java

+        admin.tenants().createTenant(NamespaceName.SYSTEM_NAMESPACE.getTenant(),
+                new TenantInfoImpl(Sets.newHashSet("appid1"), Sets.newHashSet(CLUSTER_NAME)));
+        admin.namespaces().createNamespace(NamespaceName.SYSTEM_NAMESPACE.toString());
+        admin.topics().createPartitionedTopic(TopicName.TRANSACTION_COORDINATOR_ASSIGN.toString(), 16);


do we need 16 partitions for this test ?

is there an utility method to setup the system topics related to Transactions ?

that's good comment, we don't need to create tc topic in this test.

…ic or sub have been deleted. (#11304) now, when topic have been deleted we will not handle the `endTxnOnTopic` and endTxnOnSub. now, when sub have been deleted we will not handle the `endTxnOnSub`. when topic or sub have been deleted, we should return success to tc on `endTxnOnTopic` and endTxnOnSub operation. when topic not exist in this broker, we should judge the topic if it have been created. if not created, we should return success. (cherry picked from commit a6e66dd)

…ic or sub have been deleted. (apache#11304) ## Motivation now, when topic have been deleted we will not handle the `endTxnOnTopic` and endTxnOnSub. now, when sub have been deleted we will not handle the `endTxnOnSub`. when topic or sub have been deleted, we should return success to tc on `endTxnOnTopic` and endTxnOnSub operation. ## implement when topic not exist in this broker, we should judge the topic if it have been created. if not created, we should return success.

[Transaction] Fix transaction buffer client init topic.

a019e62

congbobo184 requested review from codelipenghui, gaoran10, merlimat and eolivelli July 13, 2021 11:57

congbobo184 self-assigned this Jul 13, 2021

congbobo184 added cherry-picked/branch-2.8 Archived: 2.8 is end of life doc-not-needed Your PR changes do not impact docs labels Jul 13, 2021

eolivelli mentioned this pull request Jul 13, 2021

[Transaction] Fix transaction problem when sub delete. #11305

Closed

eolivelli requested changes Jul 13, 2021

View reviewed changes

codelipenghui added release/2.8.1 and removed cherry-picked/branch-2.8 Archived: 2.8 is end of life labels Jul 13, 2021

codelipenghui requested changes Jul 13, 2021

View reviewed changes

Fix some comment

f2688c6

congbobo184 changed the title ~~[Transaction] Fix transaction buffer client init topic.~~ [Transaction] Fix transaction buffer client handle endTxn op when topic or sub have been deleted. Aug 6, 2021

congbo added 2 commits August 6, 2021 15:52

Fix some test

605dbdc

fix some test

cc41bbc

eolivelli requested changes Aug 6, 2021

View reviewed changes

Fix some comment

6463d33

eolivelli approved these changes Aug 7, 2021

View reviewed changes

congbo added 2 commits August 9, 2021 09:23

Fix some test

71c370c

Merge branch 'master' into congbobo184_fix_transaction_buffer_client_…

8f431b6

…op_init_topic # Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferClientTest.java

codelipenghui reviewed Aug 9, 2021

View reviewed changes

congbo added 3 commits August 9, 2021 11:50

Fix som comment

52a0895

Merge branch 'master' into congbobo184_fix_transaction_buffer_client_…

a24f6d5

…op_init_topic

Merge branch 'master' into congbobo184_fix_transaction_buffer_client_…

957a614

…op_init_topic

eolivelli approved these changes Aug 9, 2021

View reviewed changes

Adopt some comment suggestion.

a0f5d7e

codelipenghui approved these changes Aug 9, 2021

View reviewed changes

codelipenghui merged commit a6e66dd into apache:master Aug 9, 2021

hangc0276 added cherry-picked/branch-2.8 Archived: 2.8 is end of life area/transaction labels Aug 12, 2021

hangc0276 added this to the 2.9.0 milestone Aug 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Transaction] Fix transaction buffer client handle endTxn op when topic or sub have been deleted. #11304

[Transaction] Fix transaction buffer client handle endTxn op when topic or sub have been deleted. #11304

congbobo184 commented Jul 13, 2021 •

edited

Loading

eolivelli Jul 13, 2021

congbobo184 Jul 13, 2021

codelipenghui Jul 13, 2021

hangc0276 Aug 5, 2021

congbobo184 commented Aug 6, 2021

eolivelli left a comment

eolivelli Aug 6, 2021

congbobo184 Aug 7, 2021 •

edited

Loading

eolivelli left a comment

codelipenghui Aug 9, 2021

codelipenghui Aug 9, 2021

eolivelli left a comment

eolivelli Aug 9, 2021

eolivelli Aug 9, 2021

eolivelli Aug 9, 2021

eolivelli Aug 9, 2021

congbobo184 Aug 9, 2021 •

edited

Loading

		admin.topics().createSubscription(topic + "_commit_sub", subName, MessageId.earliest);


		Awaitility.await().until(() -> {

	CompletableFuture<Boolean> checkManagedLedgerInitializedBefore(String name);
	CompletableFuture<Boolean> asyncExists(String name);

	CompletableFuture<Boolean> exists(String path);
	CompletableFuture<Boolean> asyncExists(String ledgerName);

[Transaction] Fix transaction buffer client handle endTxn op when topic or sub have been deleted. #11304

[Transaction] Fix transaction buffer client handle endTxn op when topic or sub have been deleted. #11304

Conversation

congbobo184 commented Jul 13, 2021 • edited Loading

Motivation

implement

Verifying this change

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

congbobo184 commented Aug 6, 2021

eolivelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

congbobo184 Aug 7, 2021 • edited Loading

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

congbobo184 Aug 9, 2021 • edited Loading

Choose a reason for hiding this comment

congbobo184 commented Jul 13, 2021 •

edited

Loading

congbobo184 Aug 7, 2021 •

edited

Loading

congbobo184 Aug 9, 2021 •

edited

Loading