[#7889, #5326] YBASE: Implement chunking/throttling in Tablet::Backfi…

…llIndexForYSQL

Summary:
Depends on https://phabricator.dev.yugabyte.com/D12435

Implement chunking/throttling in Tablet::BackfillIndexForYSQL
* At a high level, we loop around in Tablet::BackfillIndexForYSQL to call `BACKFILL INDEX` on postgres until the backfill is done, or we are approaching the deadline.
* The loop also throttles/sleeps itself to maintain the specified rate of backfill.

Notes regarding rolling-upgrades:
  * The diff assumes that the TServer issuing the new format BACKFILL INDEX command only issues it to a postgres process, which understands the new format. This holds true, because the postgres process that the TServer talks to is the local postgres process.

  * Postgres/yb-client will read from the "leader" of the tablet. If the leadership has moved, this may be a different tablet/TServer. In this case, it may read/write from/to a TServer with a different version. In this case, the backfill_spec field is ignored by the server running the older process. This will result in the "old" behavior where the BACKFILL INDEX command will run until the end of the tablet range.

Perf notes:
  Connecting to postgres to run a BACKFILL INDEX command may have some fixed set up costs. So having a very small batch size could be inefficient.

More detailed description of the changes:

------------
Docdb:
# Implement support for backfill_spec.
# Set it accordingly while calling BACKFILL INDEX
# Parse/update backfill_spec accordingly while reading pgsql_operation for a backfill read.
# Loop around BACKFILL INDEX to implement throttling/progress-in-batches

Test Plan:
Tests added.

ybd --cxx-test pg_index_backfill-test
ybd --cxx-test integration-tests_cassandra_cpp_driver-test

Reviewers: mihnea, neil, jason

Reviewed By: jason

Subscribers: bogdan, yql, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D12425

src/yb/docdb/pgsql_operation.cc

@@ -137,6 +137,7 @@ Result<common::YQLRowwiseIteratorIf::UniPtr> CreateIterator(
     CoarseTimePoint deadline,
     const ReadHybridTime& read_time,
     bool is_explicit_request_read_time) {
+  VLOG_IF(2, request.is_for_backfill()) << "Creating iterator for " << yb::ToString(request);
 
   common::YQLRowwiseIteratorIf::UniPtr result;
   // TODO(neil) Remove the following IF block when it is completely obsolete.
@@ -167,6 +168,15 @@ Result<common::YQLRowwiseIteratorIf::UniPtr> CreateIterator(
           actual_read_time.read = start_sub_doc_key.hybrid_time();
         }
       }
+    } else if (request.is_for_backfill()) {
+      RSTATUS_DCHECK(is_explicit_request_read_time, InvalidArgument,
+                     "Backfill request should already be using explicit read times.");
+      PgsqlBackfillSpecPB spec;
+      spec.ParseFromString(a2b_hex(request.backfill_spec()));
+      if (!spec.next_row_key().empty()) {
+        KeyBytes start_key_bytes(spec.next_row_key());
+        RETURN_NOT_OK(start_sub_doc_key.FullyDecodeFrom(start_key_bytes.AsSlice()));
+      }
     }
     RETURN_NOT_OK(ql_storage.GetIterator(
         request, projection, schema, txn_op_context,