Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support XA distributed transactions in CockroachDB #22329

Open
knz opened this issue Feb 2, 2018 · 2 comments
Open

Support XA distributed transactions in CockroachDB #22329

knz opened this issue Feb 2, 2018 · 2 comments
Labels
A-kv-client Relating to the KV client and the KV interface. A-sql-executor SQL txn logic A-sql-pgcompat Semantic compatibility with PostgreSQL C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@knz
Copy link
Contributor

knz commented Feb 2, 2018 •
edited by cockroach-jira-scripts
Loading

X/Open XA is a standard for distributed transaction processing across multiple database engines. This is used in enterprise scenarios where data is spread across different technologies but applications need to preserve ACID semantics for transactions that span multiple database engines.

PostgreSQL supports XA transactions by a combination of two mechanisms:

  • internally it supports the PREPARE TRANSACTION statement which enables a SQL transaction manager (TM) to "take over" transaction management
  • the official pg ODBC driver contains an implementation of the standard MSDTP API (Microsoft's view on what XA should look like), so that pg becomes interoperable with other MSDTP-aware databases.
  • ditto JDBC

For enterprise users CockroachDB should provide a path towards XA compatibility:

  • provide a mechanism / API so that an external TM can take over transaction control in CockroachDB
  • provide one or more driver extension that interoperates with one or more industry standard API (perhaps MSDTP or the java equivalent)

Strict compatibility with pg's PREPARE TRANSACTION would enable providing a XA API via pg's own ODBC driver.

cc @awoods187 for prioritization. cc @bdarnell for technical insights.

Also mentioned here: https://forum.cockroachlabs.com/t/xa-distributed-transactions/1336

Jira issue: CRDB-5863
@knz knz added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Feb 2, 2018
@andreimatei andreimatei mentioned this issue Feb 4, 2018
Open
@awoods187 awoods187 added this to the Later milestone Feb 5, 2018
@knz knz mentioned this issue Jul 9, 2018
Closed
@knz knz added A-kv-client Relating to the KV client and the KV interface. A-sql-executor SQL txn logic labels Jul 9, 2018
@petermattis petermattis removed this from the Later milestone Oct 5, 2018
jordanlewis added a commit to jordanlewis/cockroach that referenced this issue Oct 2, 2019
@jordanlewis
roachtest: add test that aggregates orm blacklist failures
cebc811 
The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep
up to date. If we write down blacklists in code, then we can use an approach
like this to always have an up to date aggregation.

So far it seems like there's just a lot of unknowns to categorize still.

The output today:

```
=== RUN   TestBlacklists
 648: unknown                                                (unknown)
 493: cockroachdb#5807   (sql: Add support for TEMP tables)
 151: cockroachdb#17511  (sql: support stored procedures)
  86: cockroachdb#26097  (sql: make TIMETZ more pg-compatible)
  56: cockroachdb#10735  (sql: support SQL savepoints)
  55: cockroachdb#32552  (multi-dim arrays)
  55: cockroachdb#26508  (sql: restricted DDL / DML inside transactions)
  52: cockroachdb#32565  (sql: support optional TIME precision)
  39: cockroachdb#243    (roadmap: Blob storage)
  33: cockroachdb#26725  (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea))
  31: cockroachdb#27793  (sql: support custom/user-defined base scalar (primitive) types)
  24: cockroachdb#12123  (sql: Can't drop and replace a table within a transaction)
  24: cockroachdb#26443  (sql: support user-defined schemas between database and table)
  20: cockroachdb#21286  (sql: Add support for geometric types)
  18: cockroachdb#6583   (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait}))
  17: cockroachdb#22329  (Support XA distributed transactions in CockroachDB)
  16: cockroachdb#24062  (sql: 32 bit SERIAL type)
  16: cockroachdb#30352  (roadmap:when CockroachDB  will support cursor?)
  12: cockroachdb#27791  (sql: support RANGE types)
   8: cockroachdb#40195  (pgwire: multiple active result sets (portals) not supported)
   8: cockroachdb#6130   (sql: add support for key watches with notifications of changes)
   5: Expected Failure                                       (unknown)
   5: cockroachdb#23468  (sql: support sql arrays of JSONB)
   5: cockroachdb#40854  (sql: set application_name from connection string)
   4: cockroachdb#35879  (sql: `default_transaction_read_only` should also accept 'on' and 'off')
   4: cockroachdb#32610  (sql: can't insert self reference)
   4: cockroachdb#40205  (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE)
   4: cockroachdb#35897  (sql: unknown function: pg_terminate_backend())
   4: cockroachdb#4035   (sql/pgwire: missing support for row count limits in pgwire)
   3: cockroachdb#27796  (sql: support user-defined DOMAIN types)
   3: cockroachdb#3781   (sql: Add Data Type Formatting Functions)
   3: cockroachdb#40476  (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`)
   3: cockroachdb#35882  (sql: support other character sets)
   2: cockroachdb#10028  (sql: Support view queries with star expansions)
   2: cockroachdb#35807  (sql: INTERVAL output doesn't match PG)
   2: cockroachdb#35902  (sql: large object support)
   2: cockroachdb#40474  (sql: support `SELECT ... FOR UPDATE OF` syntax)
   1: cockroachdb#18846  (sql: Support CIDR column type)
   1: cockroachdb#9682   (sql: implement computed indexes)
   1: cockroachdb#31632  (sql: FK options (deferrable, etc))
   1: cockroachdb#24897  (sql: CREATE OR REPLACE VIEW)
   1: pass?                                                  (unknown)
   1: cockroachdb#36215  (sql: enable setting standard_conforming_strings to off)
   1: cockroachdb#32562  (sql: support SET LOCAL and txn-scoped session variable changes)
   1: cockroachdb#36116  (sql: psychopg: investigate how `'infinity'::timestamp` is presented)
   1: cockroachdb#26732  (sql: support the binary operator: <int> / <float>)
   1: cockroachdb#23299  (sql: support coercing string literals to arrays)
   1: cockroachdb#36115  (sql: psychopg: investigate if datetimetz is being returned instead of datetime)
   1: cockroachdb#26925  (sql: make the CockroachDB integer types more compatible with postgres)
   1: cockroachdb#21085  (sql: WITH RECURSIVE (recursive common table expressions))
   1: cockroachdb#36179  (sql: implicity convert date to timestamp)
   1: cockroachdb#36118  (sql: Cannot parse '24:00' as type time)
   1: cockroachdb#31708  (sql: support current_time)
```

Release justification: non-production change
Release note: None
@rafiss rafiss added the A-sql-pgcompat Semantic compatibility with PostgreSQL label Oct 23, 2019
jordanlewis added a commit to jordanlewis/cockroach that referenced this issue Oct 24, 2019
@jordanlewis
roachtest: add test that aggregates orm blacklist failures
4010eac 
The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep
up to date. If we write down blacklists in code, then we can use an approach
like this to always have an up to date aggregation.

So far it seems like there's just a lot of unknowns to categorize still.

The output today:

```
=== RUN   TestBlacklists
 648: unknown                                                (unknown)
 493: cockroachdb#5807   (sql: Add support for TEMP tables)
 151: cockroachdb#17511  (sql: support stored procedures)
  86: cockroachdb#26097  (sql: make TIMETZ more pg-compatible)
  56: cockroachdb#10735  (sql: support SQL savepoints)
  55: cockroachdb#32552  (multi-dim arrays)
  55: cockroachdb#26508  (sql: restricted DDL / DML inside transactions)
  52: cockroachdb#32565  (sql: support optional TIME precision)
  39: cockroachdb#243    (roadmap: Blob storage)
  33: cockroachdb#26725  (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea))
  31: cockroachdb#27793  (sql: support custom/user-defined base scalar (primitive) types)
  24: cockroachdb#12123  (sql: Can't drop and replace a table within a transaction)
  24: cockroachdb#26443  (sql: support user-defined schemas between database and table)
  20: cockroachdb#21286  (sql: Add support for geometric types)
  18: cockroachdb#6583   (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait}))
  17: cockroachdb#22329  (Support XA distributed transactions in CockroachDB)
  16: cockroachdb#24062  (sql: 32 bit SERIAL type)
  16: cockroachdb#30352  (roadmap:when CockroachDB  will support cursor?)
  12: cockroachdb#27791  (sql: support RANGE types)
   8: cockroachdb#40195  (pgwire: multiple active result sets (portals) not supported)
   8: cockroachdb#6130   (sql: add support for key watches with notifications of changes)
   5: Expected Failure                                       (unknown)
   5: cockroachdb#23468  (sql: support sql arrays of JSONB)
   5: cockroachdb#40854  (sql: set application_name from connection string)
   4: cockroachdb#35879  (sql: `default_transaction_read_only` should also accept 'on' and 'off')
   4: cockroachdb#32610  (sql: can't insert self reference)
   4: cockroachdb#40205  (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE)
   4: cockroachdb#35897  (sql: unknown function: pg_terminate_backend())
   4: cockroachdb#4035   (sql/pgwire: missing support for row count limits in pgwire)
   3: cockroachdb#27796  (sql: support user-defined DOMAIN types)
   3: cockroachdb#3781   (sql: Add Data Type Formatting Functions)
   3: cockroachdb#40476  (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`)
   3: cockroachdb#35882  (sql: support other character sets)
   2: cockroachdb#10028  (sql: Support view queries with star expansions)
   2: cockroachdb#35807  (sql: INTERVAL output doesn't match PG)
   2: cockroachdb#35902  (sql: large object support)
   2: cockroachdb#40474  (sql: support `SELECT ... FOR UPDATE OF` syntax)
   1: cockroachdb#18846  (sql: Support CIDR column type)
   1: cockroachdb#9682   (sql: implement computed indexes)
   1: cockroachdb#31632  (sql: FK options (deferrable, etc))
   1: cockroachdb#24897  (sql: CREATE OR REPLACE VIEW)
   1: pass?                                                  (unknown)
   1: cockroachdb#36215  (sql: enable setting standard_conforming_strings to off)
   1: cockroachdb#32562  (sql: support SET LOCAL and txn-scoped session variable changes)
   1: cockroachdb#36116  (sql: psychopg: investigate how `'infinity'::timestamp` is presented)
   1: cockroachdb#26732  (sql: support the binary operator: <int> / <float>)
   1: cockroachdb#23299  (sql: support coercing string literals to arrays)
   1: cockroachdb#36115  (sql: psychopg: investigate if datetimetz is being returned instead of datetime)
   1: cockroachdb#26925  (sql: make the CockroachDB integer types more compatible with postgres)
   1: cockroachdb#21085  (sql: WITH RECURSIVE (recursive common table expressions))
   1: cockroachdb#36179  (sql: implicity convert date to timestamp)
   1: cockroachdb#36118  (sql: Cannot parse '24:00' as type time)
   1: cockroachdb#31708  (sql: support current_time)
```

Release justification: non-production change
Release note: None
craig bot pushed a commit that referenced this issue Nov 7, 2019
@jordanlewis
Merge #41252
27fd417 
41252: roachtest: add test that aggregates orm blacklist failures r=jordanlewis a=jordanlewis

The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep
up to date. If we write down blacklists in code, then we can use an approach
like this to always have an up to date aggregation.

So far it seems like there's just a lot of unknowns to categorize still.

The output today:

```
=== RUN   TestBlacklists
 648: unknown                                                (unknown)
 493: #5807   (sql: Add support for TEMP tables)
 151: #17511  (sql: support stored procedures)
  86: #26097  (sql: make TIMETZ more pg-compatible)
  56: #10735  (sql: support SQL savepoints)
  55: #32552  (multi-dim arrays)
  55: #26508  (sql: restricted DDL / DML inside transactions)
  52: #32565  (sql: support optional TIME precision)
  39: #243    (roadmap: Blob storage)
  33: #26725  (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea))
  31: #27793  (sql: support custom/user-defined base scalar (primitive) types)
  24: #12123  (sql: Can't drop and replace a table within a transaction)
  24: #26443  (sql: support user-defined schemas between database and table)
  20: #21286  (sql: Add support for geometric types)
  18: #6583   (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait}))
  17: #22329  (Support XA distributed transactions in CockroachDB)
  16: #24062  (sql: 32 bit SERIAL type)
  16: #30352  (roadmap:when CockroachDB  will support cursor?)
  12: #27791  (sql: support RANGE types)
   8: #40195  (pgwire: multiple active result sets (portals) not supported)
   8: #6130   (sql: add support for key watches with notifications of changes)
   5: Expected Failure                                       (unknown)
   5: #23468  (sql: support sql arrays of JSONB)
   5: #40854  (sql: set application_name from connection string)
   4: #35879  (sql: `default_transaction_read_only` should also accept 'on' and 'off')
   4: #32610  (sql: can't insert self reference)
   4: #40205  (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE)
   4: #35897  (sql: unknown function: pg_terminate_backend())
   4: #4035   (sql/pgwire: missing support for row count limits in pgwire)
   3: #27796  (sql: support user-defined DOMAIN types)
   3: #3781   (sql: Add Data Type Formatting Functions)
   3: #40476  (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`)
   3: #35882  (sql: support other character sets)
   2: #10028  (sql: Support view queries with star expansions)
   2: #35807  (sql: INTERVAL output doesn't match PG)
   2: #35902  (sql: large object support)
   2: #40474  (sql: support `SELECT ... FOR UPDATE OF` syntax)
   1: #18846  (sql: Support CIDR column type)
   1: #9682   (sql: implement computed indexes)
   1: #31632  (sql: FK options (deferrable, etc))
   1: #24897  (sql: CREATE OR REPLACE VIEW)
   1: pass?                                                  (unknown)
   1: #36215  (sql: enable setting standard_conforming_strings to off)
   1: #32562  (sql: support SET LOCAL and txn-scoped session variable changes)
   1: #36116  (sql: psychopg: investigate how `'infinity'::timestamp` is presented)
   1: #26732  (sql: support the binary operator: <int> / <float>)
   1: #23299  (sql: support coercing string literals to arrays)
   1: #36115  (sql: psychopg: investigate if datetimetz is being returned instead of datetime)
   1: #26925  (sql: make the CockroachDB integer types more compatible with postgres)
   1: #21085  (sql: WITH RECURSIVE (recursive common table expressions))
   1: #36179  (sql: implicity convert date to timestamp)
   1: #36118  (sql: Cannot parse '24:00' as type time)
   1: #31708  (sql: support current_time)
```

Release justification: non-production change
Release note: None

Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>
@github-actions
Copy link

github-actions bot commented Jun 7, 2021

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

@jlinder jlinder added the T-kv KV Team label Jun 16, 2021
@SebTardif
Copy link

While looking if I 'could' use CockroachDB instead of PostgreSQL, I found out this was not yet implemented. It's a big deal. Without it you end-up with convoluted solution like -> https://senoritadeveloper.medium.com/microservices-transaction-management-patterns-46ef2df9a9c4

This would probably help thinking about how to test it -> https://docs.hazelcast.com/hazelcast/5.2/pipelines/xa

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 20, 2024
@nvanbenschoten
sql: add max_prepared_transactions variable
38034c3 
Informs cockroachdb#22329.

This commit adds the max_prepared_transactions session variable, returning a
value of math.MaxInt32 to indicate that there is no limit on the number of
prepared transactions that can be created.

This change has the effect of unblocking the pgjdbc XADataSourceTest test
suite, which was previously failing during initialization. Is also includes
a small patch to work around cockroachdb#31632 (unsupported deferrable qualifier) so
that each test in the XA test suite actually runs (and fails).

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Aug 20, 2024
Draft
@nvanbenschoten nvanbenschoten mentioned this issue Aug 20, 2024
Draft
@nvanbenschoten nvanbenschoten mentioned this issue Aug 20, 2024
Draft
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 20, 2024
@nvanbenschoten
sql: add max_prepared_transactions variable
e892b8e 
Informs cockroachdb#22329.

This commit adds the max_prepared_transactions session variable, returning a
value of math.MaxInt32 to indicate that there is no limit on the number of
prepared transactions that can be created.

This change has the effect of unblocking the pgjdbc XADataSourceTest test
suite, which was previously failing during initialization. Is also includes
a small patch to work around cockroachdb#31632 (unsupported deferrable qualifier) so
that each test in the XA test suite actually runs (and fails).

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 20, 2024
@nvanbenschoten
sql: create system.prepared_transactions table
262ab50 
Informs cockroachdb#22329.

This commit adds a new `system.prepared_transactions` table, which will
be used to track prepared transactions. The table includes all of the
information needed to implement XA two-phase commit transactions, along
with additional metadata needed to implement `pg_catalog.pg_prepared_xacts`.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 20, 2024
@nvanbenschoten
sql: add parser support for XA transactions
0b32909 
Informs cockroachdb#22329.

This commit adds new syntax for `PREPARE TRANSACTION <gid>`,
`COMMIT PREPARED <gid>`, and `ROLLBACK PREPARED <gid>`, for
parity with Postgres support.

Handling of the syntax is currently unimplemented.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 21, 2024
@nvanbenschoten
kv: support two-phase commit transactions
86ea407 
Informs cockroachdb#22329.

This commit adds support for Txn.Prepare, DB.CommitPrepared, and
DB.RollbackPrepared APIs to KV. These methods allow clients to prepare a
transaction for commit, commit a prepared transaction, and rollback a
prepared transaction, respectively.

When a transaction is prepared, we send an EndTxnRequest as usual, but a
new flag indicates the transaction should be moved to status PREPARED
instead of COMMITTED. In state PREPARED, a transaction record is updated
to include all of its intents, and awaits a subsequent EndTxnRequest
which will move the transaction either to COMMITTED or ABORTED.
Transaction in state PREPARED cannot be pushed, regardless of isolation
level, ensuring that a subsequent commit or rollback is guaranteed to
succeed.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 21, 2024
@nvanbenschoten
kv/kvnemesis: add Txn.Prepare
8b61574 
Informs cockroachdb#22329.

This commit adds support for Txn.Prepare in kvnemesis.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Aug 21, 2024
Draft
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 21, 2024
@nvanbenschoten
sql: implement XA transaction SQL statements
5d92be2 
Informs cockroachdb#22329.

This commit implements the statement handling for `PREPARE TRANSACTION
<gid>`, `COMMIT PREPARED <gid>`, and `ROLLBACK PREPARED <gid>`, for
parity with Postgres support. The implementation is based on the
Postgres documentation and the Postgres source code.

Release note (sql change): XA transaction support allows CockroachDB to
participate in distributed transaction with other resources (e.g.
databases, message queues, etc) using a two phase commit protocol.
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 21, 2024
@nvanbenschoten
sql: implement pg_catalog.pg_prepared_xacts
c471eca 
Informs cockroachdb#22329.

`pg_catalog.pg_prepared_xacts` reads from `system.prepared_transactions`.

Release note: None
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 21, 2024
@nvanbenschoten
roachtest: fix pgjdbc XA test suite
367d81a 
Informs cockroachdb#22329.

This commit fixes the pgjdbc blocklist to reflect the current state of
support for XA transactions in CockroachDB. With the newly added support
for two-phase commit transactions, all tests in the XADataSourceTest
test suite now pass except for `mappingOfConstraintViolations`, which is
blocked by cockroachdb#31632 (DEFERRABLE foreign keys).

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Aug 21, 2024
Draft
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 21, 2024
@nvanbenschoten
kv/kvnemesis: add Txn.Prepare
a0184cd 
Informs cockroachdb#22329.

This commit adds support for Txn.Prepare in kvnemesis.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Aug 21, 2024
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-client Relating to the KV client and the KV interface. A-sql-executor SQL txn logic A-sql-pgcompat Semantic compatibility with PostgreSQL C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jlinder @knz @rafiss @SebTardif @petermattis @awoods187