vstreamer: include gtid events in binlog commit queries

[maxenglander]

Description

The vstreamer assumes that commit events in the binlog take the form of Xid events. This poses a problem when doing MoveTables from external datastores that produce different kinds of binlogs, such as MySQL NDB clusters.

Details

Given a schema like:

CREATE TABLE `test_innodb` (
  `id` int NOT NULL AUTO_INCREMENT,
  `data` varchar(10) DEFAULT NULL,
  `delta` varchar(10) DEFAULT NULL,
  `version` int DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

Inserting a row produces a binlog like:

#230929 16:53:07 server id 1  end_log_pos 378 CRC32 0x4c190295  Table_map: `app`.`test_innodb` mapped to number 492
# has_generated_invisible_primary_key=0
# at 378
#230929 16:53:07 server id 1  end_log_pos 434 CRC32 0xb7f9c4d4  Write_rows: table id 492 flags: STMT_END_F
# at 434
#230929 16:53:07 server id 1  end_log_pos 465 CRC32 0x72c26022  Xid = 77448

Notice the Xid event. The VTTablet vstreamer only produces GTID VEvents when it encounters these Xid binlog events:

vitess/go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

Lines 451 to 457 in 2f1fc13

	case ev.IsXID():
	vevents = append(vevents, &binlogdatapb.VEvent{
	Type: binlogdatapb.VEventType_GTID,
	Gtid: replication.EncodePosition(vs.pos),
	}, &binlogdatapb.VEvent{
	Type: binlogdatapb.VEventType_COMMIT,
	})

Meanwhile, the vplayer depends on these GTID VEvents in order to advance the vreplication.pos:

vitess/go/vt/vttablet/tabletmanager/vreplication/vplayer.go

Lines 481 to 486 in 2f1fc13

	case binlogdatapb.VEventType_GTID:
	pos, err := binlogplayer.DecodePosition(event.Gtid)
	if err != nil {
	return err
	}
	vp.pos = pos

This setup won't work for MySQL NDB clusters, which, given the same schema and insert above, produce binlogs that look like this:

# at 615
#230929 16:53:13 server id 1  end_log_pos 680 CRC32 0x6645bfe1  Table_map: `app`.`test_ndb` mapped to number 453
# has_generated_invisible_primary_key=0
# at 680
#230929 16:53:13 server id 1  end_log_pos 752 CRC32 0x91bcbe1e  Table_map: `mysql`.`ndb_apply_status` mapped to number 450
# has_generated_invisible_primary_key=0
# at 752
#230929 16:53:13 server id 1  end_log_pos 817 CRC32 0xad022d11  Write_rows: table id 450
# at 817
#230929 16:53:13 server id 1  end_log_pos 873 CRC32 0x1f9d1b61  Write_rows: table id 453 flags: STMT_END_F
# at 873
#230929 16:53:13 server id 1  end_log_pos 945 CRC32 0x6f300b74  Query   thread_id=2     exec_time=0     error_code=0

Related Issue(s)

Addresses #14166

Checklist

TODO

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on the CI
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.