Supporting START TRANSACTION WITH CONSISTENT [ROCKSDB] SNAPSHOT

Summary:
This adds two features in RocksDB.
1. Supporting START TRANSACTION WITH CONSISTENT SNAPSHOT
2. Getting current binlog position in addition to percona#1.
With these features, mysqldump can take consistent logical backup.

The second feature is done by START TRANSACTION WITH
CONSISTENT ROCKSDB SNAPSHOT. This is Facebook's extension, and
it works like existing START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT.

This diff changed some existing codebase/behaviors.

- Original Facebook-MySQL always started InnoDB transaction
regardless of engine clause. For example, START TRANSACTION WITH
CONSISTENT MYISAM SNAPSHOT was accepted but it actually started
InnoDB transaction, not MyISAM. This patch does not allow
setting engine that does not support consistent snapshot.

mysql> start transaction with consistent myisam snapshot;
ERROR 1105 (HY000): Consistent Snapshot is not supported for this engine

Currently only InnoDB and RocksDB support consistent snapshot.
To check engines, I modified sql/sql_yacc.yy, trans_begin()
and ha_start_consistent_snapshot() to pass handlerton.

- Changed constant name from
  MYSQL_START_TRANS_OPT_WITH_CONS_INNODB_SNAPSHOT to
MYSQL_START_TRANS_OPT_WITH_CONS_ENGINE_SNAPSHOT, because it's no longer
InnoDB dependent.

- When not setting engine, START TRANSACTION WITH CONSISTENT SNAPSHOT
takes both InnoDB and RocksDB snapshots, and both InnoDB and RocksDB
participate in transaction. When executing COMMIT, both InnoDB and
RocksDB modifications are committed. Remember that XA is not supported yet,
so mixing engines is not recommended anyway.

- When setting engine, START TRANSACTION WITH CONSISTENT.. takes
snapshot for the specified engine only. But it starts both
InnoDB and RocksDB transactions.

Test Plan: mtr --suite=rocksdb,rocksdb_rpl, --repeat=3

Reviewers: hermanlee4, jonahcohen, jtolmer, tian.xia, maykov, spetrunia

Reviewed By: spetrunia

Subscribers: steaphan

Differential Revision: https://reviews.facebook.net/D32355

r/consistent_snapshot_mixed_engines.result

@@ -0,0 +1,68 @@
+DROP TABLE IF EXISTS t1;
+connect  con1,localhost,root,,;
+connect  con2,localhost,root,,;
+connection con1;
+create table i1 (id int primary key , value int) engine=innodb;
+create table r1 (id int primary key , value int) engine=rocksdb;
+SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;
+START TRANSACTION WITH CONSISTENT SNAPSHOT;
+connection con2;
+insert into i1 values (1,1);
+insert into r1 values (1,1);
+connection con1;
+select * from i1;
+id	value
+select * from r1;
+id	value
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	1109	uuid:1-5
+connection con2;
+insert into i1 values (2,2);
+insert into r1 values (2,2);
+connection con1;
+select * from i1;
+id	value
+1	1
+2	2
+select * from r1;
+id	value
+1	1
+connection con2;
+insert into i1 values (3,2);
+insert into r1 values (3,2);
+connection con1;
+select * from i1;
+id	value
+1	1
+2	2
+select * from r1;
+id	value
+1	1
+START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	1997	uuid:1-9
+connection con2;
+insert into r1 values (4,4);
+connection con1;
+select * from r1;
+id	value
+1	1
+2	2
+3	2
+4	4
+connection con2;
+insert into r1 values (5,5);
+connection con1;
+select * from r1;
+id	value
+1	1
+2	2
+3	2
+4	4
+drop table i1;
+drop table r1;
+connection default;
+disconnect con1;
+disconnect con2;
+reset master;

r/rpl_rocksdb_snapshot.result

@@ -0,0 +1,222 @@
+include/master-slave.inc
+Warnings:
+Note	####	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	####	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+[connection master]
+DROP TABLE IF EXISTS t1;
+# Establish connection con1 (user=root)
+# Establish connection con2 (user=root)
+# Establish connection con3 (user=root)
+# Establish connection con4 (user=root)
+# reset replication to guarantee that master-bin.000001 is used
+include/stop_slave.inc
+RESET SLAVE;
+RESET MASTER;
+RESET MASTER;
+CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root";
+Warnings:
+Note	1759	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	1760	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+include/start_slave.inc
+# Switch to connection con1
+CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb;
+INSERT INTO t1 VALUES(1);
+SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
+START TRANSACTION WITH CONSISTENT SNAPSHOT;
+ERROR HY000: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine.
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+ERROR HY000: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine.
+ROLLBACK;
+SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	528	UUID:1-2
+# Switch to connection con2
+INSERT INTO t1 VALUES(2);
+INSERT INTO t1 VALUES(3);
+# Switch to connection con1
+SELECT * FROM t1;
+a
+1
+COMMIT;
+SELECT * FROM t1;
+a
+1
+2
+3
+DROP TABLE t1;
+# Switch to connection con1
+CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb;
+INSERT INTO t1 VALUES(1);
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	1498	UUID:1-7
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	1498	UUID:1-7
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	1498	UUID:1-7
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	1498	UUID:1-7
+# Switch to connection con2
+INSERT INTO t1 VALUES(2);
+INSERT INTO t1 VALUES(3);
+# Switch to connection con1
+SELECT * FROM t1;
+a
+1
+SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1;
+COMMIT;
+# Switch to slave
+CREATE TABLE t1_backup LIKE t1;
+INSERT INTO t1_backup SELECT * FROM t1;
+include/stop_slave.inc
+RESET SLAVE;
+RESET MASTER;
+DELETE FROM t1;
+LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1;
+SELECT * FROM t1;
+a
+1
+CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root",master_log_file="master-bin.000001",master_log_pos=binlog_pos;
+Warnings:
+Note	1759	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	1760	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+include/start_slave.inc
+SELECT * FROM t1;
+a
+1
+2
+3
+SELECT * FROM t1_backup;
+a
+1
+2
+3
+DROP TABLE t1_backup;
+DROP TABLE t1;
+# Switch to connection con1
+CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb;
+INSERT INTO t1 VALUES(1);
+# async queries from con2
+INSERT INTO t1 VALUES(2);
+# async queries from con3
+INSERT INTO t1 VALUES(21);
+# Switch to connection con1
+# Switch to connection con4
+INSERT INTO t1 VALUES(9);
+# Switch to connection con1
+SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1;
+COMMIT;
+# reap async statements
+# Switch to slave
+CREATE TABLE t1_backup LIKE t1;
+INSERT INTO t1_backup SELECT * FROM t1;
+include/stop_slave.inc
+RESET SLAVE;
+RESET MASTER;
+DELETE FROM t1;
+LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1;
+CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root",master_log_file="master-bin.000001",master_log_pos=binlog_pos;
+Warnings:
+Note	1759	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	1760	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+include/start_slave.inc
+# sync and then query slave
+ShouldBeZero
+0
+DROP TABLE t1_backup;
+DROP TABLE t1;
+# Switch to connection con1
+CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb;
+INSERT INTO t1 VALUES(1);
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	3655	UUID:1-18
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	3655	UUID:1-18
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	3655	UUID:1-18
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000001	3655	UUID:1-18
+# Switch to connection con2
+INSERT INTO t1 VALUES(2);
+INSERT INTO t1 VALUES(3);
+# Switch to connection con1
+SELECT * FROM t1;
+a
+1
+SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1;
+COMMIT;
+# Switch to slave
+CREATE TABLE t1_backup LIKE t1;
+INSERT INTO t1_backup SELECT * FROM t1;
+include/stop_slave.inc
+RESET SLAVE;
+RESET MASTER;
+SET @@global.gtid_purged='gtid_executed_from_snapshot';
+DELETE FROM t1;
+LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1;
+SELECT * FROM t1;
+a
+1
+CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root", master_auto_position=1;
+Warnings:
+Note	1759	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	1760	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+include/start_slave.inc
+SELECT * FROM t1;
+a
+1
+2
+3
+SELECT * FROM t1_backup;
+a
+1
+2
+3
+DROP TABLE t1_backup;
+DROP TABLE t1;
+# Switch to connection con1
+CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb;
+INSERT INTO t1 VALUES(1);
+# async queries from con2
+INSERT INTO t1 VALUES(2);
+# async queries from con3
+INSERT INTO t1 VALUES(21);
+# Switch to connection con1
+# Switch to connection con4
+INSERT INTO t1 VALUES(9);
+# Switch to connection con1
+SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1;
+COMMIT;
+# reap async statements
+# Switch to slave
+CREATE TABLE t1_backup LIKE t1;
+INSERT INTO t1_backup SELECT * FROM t1;
+include/stop_slave.inc
+RESET SLAVE;
+RESET MASTER;
+SET @@global.gtid_purged='gtid_executed_from_snapshot';
+DELETE FROM t1;
+LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1;
+CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root", master_auto_position=1;
+Warnings:
+Note	1759	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	1760	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+include/start_slave.inc
+# sync and then query slave
+ShouldBeZero
+0
+DROP TABLE t1_backup;
+DROP TABLE t1;
+# Switch to connection default + close connections con1 and con2
+include/stop_slave.inc
+CHANGE MASTER to master_auto_position=0;
+include/start_slave.inc
+include/rpl_end.inc

r/rpl_rocksdb_snapshot_without_gtid.result

@@ -0,0 +1,15 @@
+include/master-slave.inc
+Warnings:
+Note	####	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	####	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+[connection master]
+create table t1(a int);
+FLUSH LOGS;
+insert into t1 values(1);
+insert into t1 values(2);
+FLUSH LOGS;
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+File	Position	Gtid_executed
+master-bin.000003	120	
+drop table t1;
+include/rpl_end.inc

t/consistent_snapshot_mixed_engines-master.opt

@@ -0,0 +1 @@
+--gtid_mode=ON --enforce_gtid_consistency --log_bin --log_slave_updates

t/consistent_snapshot_mixed_engines.test

@@ -0,0 +1,79 @@
+--source include/have_log_bin.inc
+--enable_connect_log
+-- let $uuid = `select @@server_uuid;`
+
+# Save the initial number of concurrent sessions
+--source include/count_sessions.inc
+
+--disable_warnings
+DROP TABLE IF EXISTS t1;
+--enable_warnings
+
+connect (con1,localhost,root,,);
+connect (con2,localhost,root,,);
+
+connection con1;
+create table i1 (id int primary key , value int) engine=innodb;
+create table r1 (id int primary key , value int) engine=rocksdb;
+
+
+SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;
+
+# Without setting engine, this takes both InnoDB and RocksDB snapshots
+-- replace_result $uuid uuid
+START TRANSACTION WITH CONSISTENT SNAPSHOT;
+
+connection con2;
+insert into i1 values (1,1);
+insert into r1 values (1,1);
+
+connection con1;
+select * from i1;
+select * from r1;
+
+# This takes RocksDB snapshot only but both InnoDB participates in transaction.
+-- replace_result $uuid uuid
+START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT;
+
+connection con2;
+insert into i1 values (2,2);
+insert into r1 values (2,2);
+
+connection con1;
+# takes InnoDB snapshot here so changes after that not visible
+select * from i1;
+select * from r1;
+
+connection con2;
+insert into i1 values (3,2);
+insert into r1 values (3,2);
+
+connection con1;
+select * from i1;
+select * from r1;
+
+# RocksDB also partipates in transaction
+-- replace_result $uuid uuid
+START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT;
+
+connection con2;
+insert into r1 values (4,4);
+
+connection con1;
+# takes RocksDB snapshot here so changes after that are not visible
+select * from r1;
+
+connection con2;
+insert into r1 values (5,5);
+
+connection con1;
+select * from r1;
+
+drop table i1;
+drop table r1;
+
+connection default;
+disconnect con1;
+disconnect con2;
+reset master;
+--source include/wait_until_count_sessions.inc

t/rpl_rocksdb_snapshot-master.opt

@@ -0,0 +1 @@
+--gtid_mode=ON --enforce_gtid_consistency --log_bin --log_slave_updates

t/rpl_rocksdb_snapshot-slave.opt

@@ -0,0 +1 @@
+--gtid_mode=ON --enforce_gtid_consistency --log_bin --log_slave_updates