From b9b7de1f7df37f13478eda5c976aa5874464470f Mon Sep 17 00:00:00 2001 From: Nathan VanBenschoten Date: Mon, 11 Jun 2018 12:07:43 -0400 Subject: [PATCH] kv: pipeline transactional writes This change pipelines transactional writes using the approach presented in https://github.com/cockroachdb/cockroach/issues/16026#issuecomment-375904032. The change introduces transactional pipelining by creating a new `txnReqInterceptor` that hooks into the `TxnCoordSender` called `txnPipeliner`. txnPipeliner is a txnInterceptor that pipelines transactional writes by using asynchronous consensus. The interceptor then tracks all writes that have been asynchronously proposed through Raft and ensures that all interfering requests chain on to them by first proving that the async writes succeeded. The interceptor also ensures that when committing a transaction all writes that have been proposed but not proven to have succeeded are first checked before committing. These async writes are referred to as "outstanding writes" and this process of proving that an outstanding write succeeded is called "resolving" the write. Chaining on to in-flight async writes is important for two main reasons to txnPipeliner: 1. requests proposed to Raft will not necessarily succeed. For any number of reasons, the request may make it through Raft and be discarded or fail to ever even be replicated. A transaction must check that all async writes succeeded before committing. However, when these proposals do fail, their errors aren't particularly interesting to a transaction. This is because these errors are not deterministic Transaction-domain errors that a transaction must adhere to for correctness such as conditional-put errors or other symptoms of constraint violations. These kinds of errors are all discovered during write *evaluation*, which an async write will perform synchronously before consensus. Any error during consensus is outside of the Transaction-domain and can always trigger a transaction retry. 2. transport layers beneath the txnPipeliner do not provide strong enough ordering guarantees between concurrent requests in the same transaction to avoid needing explicit chaining. For instance, DistSender uses unary gRPC requests instead of gRPC streams, so it can't natively expose strong ordering guarantees. Perhaps more importantly, even when a command has entered the command queue and evaluated on a Replica, it is not guaranteed to be applied before interfering commands. This is because the command may be retried outside of the serialization of the command queue for any number of reasons, such as leaseholder changes. When the command re-enters the command queue, it's possible that interfering commands may jump ahead of it. To combat this, the txnPipeliner uses chaining to throw an error when these re-orderings would have affected the order that transactional requests evaluate in. The first sanity check benchmark was to run the change against a geo-distributed cluster running TPCC. A node was located in each of us-east1-b, us-west1-b, europe-west2-b. The load generator was located in us-east1-b and all leases were moved to this zone as well. The test first limited all operations to newOrders, but soon expanded to full TPCC after seeing desirable results. Without txn pipelining ``` _elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms) 180.0s 198.3 154.2% 916.5 486.5 2818.6 4160.7 5637.1 5905.6 ``` With txn pipelining ``` _elapsed_______tpmC____efc__avg(ms)__p50(ms)__p90(ms)__p95(ms)__p99(ms)_pMax(ms) 180.0s 200.0 155.5% 388.2 184.5 1208.0 1946.2 2684.4 2818.6 ``` One big caveat is that all foreign key checks needed to be disabled for this experiment to show a significant improvement. The reason for this is subtle. TPCC's schema is structured as a large hierarchy of tables, and most of its transactions insert down a lineage of tables. With FKs enabled, we see a query to a table immediately after it is inserted into when its child table is inserted into next. This effectively causes a pipeline stall immediately after each write, which eliminated a lot of the benefit that we expect to see here. Luckily, this is a problem that can be avoided by being a little smarter about foreign keys at the SQL-level. We should not be performing these scans over all possible column families for a row just to check for existence (resulting kv count > 0) if we know that we just inserted into that row. We need some kind of row-level existence cache in SQL to avoid these scans. This will be generally useful, but will be especially useful to avoid these pipeline stalls. Release note: None --- c-deps/libroach/protos/roachpb/data.pb.cc | 284 +++++- c-deps/libroach/protos/roachpb/data.pb.h | 240 ++++- docs/generated/settings/settings.html | 1 + pkg/internal/client/txn.go | 2 +- pkg/kv/dist_sender.go | 23 +- pkg/kv/txn_coord_sender.go | 7 +- pkg/kv/txn_coord_sender_test.go | 18 +- pkg/kv/txn_interceptor_pipeliner.go | 502 ++++++++++ pkg/kv/txn_interceptor_pipeliner_test.go | 860 ++++++++++++++++++ pkg/roachpb/api.go | 6 + pkg/roachpb/api.pb.go | 1 + pkg/roachpb/data.pb.go | 449 ++++++--- pkg/roachpb/data.proto | 17 + pkg/roachpb/errors.pb.go | 311 +++---- pkg/roachpb/errors.proto | 2 + .../logictest/testdata/logic_test/show_trace | 16 +- pkg/storage/replica_command.go | 23 +- 17 files changed, 2451 insertions(+), 311 deletions(-) create mode 100644 pkg/kv/txn_interceptor_pipeliner.go create mode 100644 pkg/kv/txn_interceptor_pipeliner_test.go diff --git a/c-deps/libroach/protos/roachpb/data.pb.cc b/c-deps/libroach/protos/roachpb/data.pb.cc index 931960d5b396..0bdcb76663c9 100644 --- a/c-deps/libroach/protos/roachpb/data.pb.cc +++ b/c-deps/libroach/protos/roachpb/data.pb.cc @@ -17,6 +17,7 @@ // @@protoc_insertion_point(includes) namespace protobuf_roachpb_2fdata_2eproto { +extern PROTOBUF_INTERNAL_EXPORT_protobuf_roachpb_2fdata_2eproto ::google::protobuf::internal::SCCInfo<0> scc_info_SequencedWrite; extern PROTOBUF_INTERNAL_EXPORT_protobuf_roachpb_2fdata_2eproto ::google::protobuf::internal::SCCInfo<0> scc_info_Span; extern PROTOBUF_INTERNAL_EXPORT_protobuf_roachpb_2fdata_2eproto ::google::protobuf::internal::SCCInfo<1> scc_info_ChangeReplicasTrigger; extern PROTOBUF_INTERNAL_EXPORT_protobuf_roachpb_2fdata_2eproto ::google::protobuf::internal::SCCInfo<1> scc_info_MergeTrigger; @@ -98,6 +99,11 @@ class IntentDefaultTypeInternal { ::google::protobuf::internal::ExplicitlyConstructed _instance; } _Intent_default_instance_; +class SequencedWriteDefaultTypeInternal { + public: + ::google::protobuf::internal::ExplicitlyConstructed + _instance; +} _SequencedWrite_default_instance_; class LeaseDefaultTypeInternal { public: ::google::protobuf::internal::ExplicitlyConstructed @@ -301,6 +307,20 @@ ::google::protobuf::internal::SCCInfo<2> scc_info_Intent = &protobuf_roachpb_2fdata_2eproto::scc_info_Span.base, &protobuf_storage_2fengine_2fenginepb_2fmvcc3_2eproto::scc_info_TxnMeta.base,}}; +static void InitDefaultsSequencedWrite() { + GOOGLE_PROTOBUF_VERIFY_VERSION; + + { + void* ptr = &::cockroach::roachpb::_SequencedWrite_default_instance_; + new (ptr) ::cockroach::roachpb::SequencedWrite(); + ::google::protobuf::internal::OnShutdownDestroyMessage(ptr); + } + ::cockroach::roachpb::SequencedWrite::InitAsDefaultInstance(); +} + +::google::protobuf::internal::SCCInfo<0> scc_info_SequencedWrite = + {{ATOMIC_VAR_INIT(::google::protobuf::internal::SCCInfoBase::kUninitialized), 0, InitDefaultsSequencedWrite}, {}}; + static void InitDefaultsLease() { GOOGLE_PROTOBUF_VERIFY_VERSION; @@ -343,10 +363,11 @@ static void InitDefaultsTxnCoordMeta() { ::cockroach::roachpb::TxnCoordMeta::InitAsDefaultInstance(); } -::google::protobuf::internal::SCCInfo<2> scc_info_TxnCoordMeta = - {{ATOMIC_VAR_INIT(::google::protobuf::internal::SCCInfoBase::kUninitialized), 2, InitDefaultsTxnCoordMeta}, { +::google::protobuf::internal::SCCInfo<3> scc_info_TxnCoordMeta = + {{ATOMIC_VAR_INIT(::google::protobuf::internal::SCCInfoBase::kUninitialized), 3, InitDefaultsTxnCoordMeta}, { &protobuf_roachpb_2fdata_2eproto::scc_info_Transaction.base, - &protobuf_roachpb_2fdata_2eproto::scc_info_Span.base,}}; + &protobuf_roachpb_2fdata_2eproto::scc_info_Span.base, + &protobuf_roachpb_2fdata_2eproto::scc_info_SequencedWrite.base,}}; void InitDefaults() { ::google::protobuf::internal::InitSCC(&scc_info_Span.base); @@ -361,6 +382,7 @@ void InitDefaults() { ::google::protobuf::internal::InitSCC(&scc_info_ObservedTimestamp.base); ::google::protobuf::internal::InitSCC(&scc_info_Transaction.base); ::google::protobuf::internal::InitSCC(&scc_info_Intent.base); + ::google::protobuf::internal::InitSCC(&scc_info_SequencedWrite.base); ::google::protobuf::internal::InitSCC(&scc_info_Lease.base); ::google::protobuf::internal::InitSCC(&scc_info_AbortSpanEntry.base); ::google::protobuf::internal::InitSCC(&scc_info_TxnCoordMeta.base); @@ -3872,6 +3894,223 @@ ::std::string Intent::GetTypeName() const { } +// =================================================================== + +void SequencedWrite::InitAsDefaultInstance() { +} +#if !defined(_MSC_VER) || _MSC_VER >= 1900 +const int SequencedWrite::kKeyFieldNumber; +const int SequencedWrite::kSequenceFieldNumber; +#endif // !defined(_MSC_VER) || _MSC_VER >= 1900 + +SequencedWrite::SequencedWrite() + : ::google::protobuf::MessageLite(), _internal_metadata_(NULL) { + ::google::protobuf::internal::InitSCC( + &protobuf_roachpb_2fdata_2eproto::scc_info_SequencedWrite.base); + SharedCtor(); + // @@protoc_insertion_point(constructor:cockroach.roachpb.SequencedWrite) +} +SequencedWrite::SequencedWrite(const SequencedWrite& from) + : ::google::protobuf::MessageLite(), + _internal_metadata_(NULL) { + _internal_metadata_.MergeFrom(from._internal_metadata_); + key_.UnsafeSetDefault(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); + if (from.key().size() > 0) { + key_.AssignWithDefault(&::google::protobuf::internal::GetEmptyStringAlreadyInited(), from.key_); + } + sequence_ = from.sequence_; + // @@protoc_insertion_point(copy_constructor:cockroach.roachpb.SequencedWrite) +} + +void SequencedWrite::SharedCtor() { + key_.UnsafeSetDefault(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); + sequence_ = 0; +} + +SequencedWrite::~SequencedWrite() { + // @@protoc_insertion_point(destructor:cockroach.roachpb.SequencedWrite) + SharedDtor(); +} + +void SequencedWrite::SharedDtor() { + key_.DestroyNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); +} + +void SequencedWrite::SetCachedSize(int size) const { + _cached_size_.Set(size); +} +const SequencedWrite& SequencedWrite::default_instance() { + ::google::protobuf::internal::InitSCC(&protobuf_roachpb_2fdata_2eproto::scc_info_SequencedWrite.base); + return *internal_default_instance(); +} + + +void SequencedWrite::Clear() { +// @@protoc_insertion_point(message_clear_start:cockroach.roachpb.SequencedWrite) + ::google::protobuf::uint32 cached_has_bits = 0; + // Prevent compiler warnings about cached_has_bits being unused + (void) cached_has_bits; + + key_.ClearToEmptyNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); + sequence_ = 0; + _internal_metadata_.Clear(); +} + +bool SequencedWrite::MergePartialFromCodedStream( + ::google::protobuf::io::CodedInputStream* input) { +#define DO_(EXPRESSION) if (!GOOGLE_PREDICT_TRUE(EXPRESSION)) goto failure + ::google::protobuf::uint32 tag; + ::google::protobuf::internal::LiteUnknownFieldSetter unknown_fields_setter( + &_internal_metadata_); + ::google::protobuf::io::StringOutputStream unknown_fields_output( + unknown_fields_setter.buffer()); + ::google::protobuf::io::CodedOutputStream unknown_fields_stream( + &unknown_fields_output, false); + // @@protoc_insertion_point(parse_start:cockroach.roachpb.SequencedWrite) + for (;;) { + ::std::pair<::google::protobuf::uint32, bool> p = input->ReadTagWithCutoffNoLastTag(127u); + tag = p.first; + if (!p.second) goto handle_unusual; + switch (::google::protobuf::internal::WireFormatLite::GetTagFieldNumber(tag)) { + case 1: { + if (static_cast< ::google::protobuf::uint8>(tag) == + static_cast< ::google::protobuf::uint8>(10u /* 10 & 0xFF */)) { + DO_(::google::protobuf::internal::WireFormatLite::ReadBytes( + input, this->mutable_key())); + } else { + goto handle_unusual; + } + break; + } + + // int32 sequence = 2; + case 2: { + if (static_cast< ::google::protobuf::uint8>(tag) == + static_cast< ::google::protobuf::uint8>(16u /* 16 & 0xFF */)) { + + DO_((::google::protobuf::internal::WireFormatLite::ReadPrimitive< + ::google::protobuf::int32, ::google::protobuf::internal::WireFormatLite::TYPE_INT32>( + input, &sequence_))); + } else { + goto handle_unusual; + } + break; + } + + default: { + handle_unusual: + if (tag == 0) { + goto success; + } + DO_(::google::protobuf::internal::WireFormatLite::SkipField( + input, tag, &unknown_fields_stream)); + break; + } + } + } +success: + // @@protoc_insertion_point(parse_success:cockroach.roachpb.SequencedWrite) + return true; +failure: + // @@protoc_insertion_point(parse_failure:cockroach.roachpb.SequencedWrite) + return false; +#undef DO_ +} + +void SequencedWrite::SerializeWithCachedSizes( + ::google::protobuf::io::CodedOutputStream* output) const { + // @@protoc_insertion_point(serialize_start:cockroach.roachpb.SequencedWrite) + ::google::protobuf::uint32 cached_has_bits = 0; + (void) cached_has_bits; + + if (this->key().size() > 0) { + ::google::protobuf::internal::WireFormatLite::WriteBytesMaybeAliased( + 1, this->key(), output); + } + + // int32 sequence = 2; + if (this->sequence() != 0) { + ::google::protobuf::internal::WireFormatLite::WriteInt32(2, this->sequence(), output); + } + + output->WriteRaw((::google::protobuf::internal::GetProto3PreserveUnknownsDefault() ? _internal_metadata_.unknown_fields() : _internal_metadata_.default_instance()).data(), + static_cast((::google::protobuf::internal::GetProto3PreserveUnknownsDefault() ? _internal_metadata_.unknown_fields() : _internal_metadata_.default_instance()).size())); + // @@protoc_insertion_point(serialize_end:cockroach.roachpb.SequencedWrite) +} + +size_t SequencedWrite::ByteSizeLong() const { +// @@protoc_insertion_point(message_byte_size_start:cockroach.roachpb.SequencedWrite) + size_t total_size = 0; + + total_size += (::google::protobuf::internal::GetProto3PreserveUnknownsDefault() ? _internal_metadata_.unknown_fields() : _internal_metadata_.default_instance()).size(); + + if (this->key().size() > 0) { + total_size += 1 + + ::google::protobuf::internal::WireFormatLite::BytesSize( + this->key()); + } + + // int32 sequence = 2; + if (this->sequence() != 0) { + total_size += 1 + + ::google::protobuf::internal::WireFormatLite::Int32Size( + this->sequence()); + } + + int cached_size = ::google::protobuf::internal::ToCachedSize(total_size); + SetCachedSize(cached_size); + return total_size; +} + +void SequencedWrite::CheckTypeAndMergeFrom( + const ::google::protobuf::MessageLite& from) { + MergeFrom(*::google::protobuf::down_cast(&from)); +} + +void SequencedWrite::MergeFrom(const SequencedWrite& from) { +// @@protoc_insertion_point(class_specific_merge_from_start:cockroach.roachpb.SequencedWrite) + GOOGLE_DCHECK_NE(&from, this); + _internal_metadata_.MergeFrom(from._internal_metadata_); + ::google::protobuf::uint32 cached_has_bits = 0; + (void) cached_has_bits; + + if (from.key().size() > 0) { + + key_.AssignWithDefault(&::google::protobuf::internal::GetEmptyStringAlreadyInited(), from.key_); + } + if (from.sequence() != 0) { + set_sequence(from.sequence()); + } +} + +void SequencedWrite::CopyFrom(const SequencedWrite& from) { +// @@protoc_insertion_point(class_specific_copy_from_start:cockroach.roachpb.SequencedWrite) + if (&from == this) return; + Clear(); + MergeFrom(from); +} + +bool SequencedWrite::IsInitialized() const { + return true; +} + +void SequencedWrite::Swap(SequencedWrite* other) { + if (other == this) return; + InternalSwap(other); +} +void SequencedWrite::InternalSwap(SequencedWrite* other) { + using std::swap; + key_.Swap(&other->key_, &::google::protobuf::internal::GetEmptyStringAlreadyInited(), + GetArenaNoVirtual()); + swap(sequence_, other->sequence_); + _internal_metadata_.Swap(&other->_internal_metadata_); +} + +::std::string SequencedWrite::GetTypeName() const { + return "cockroach.roachpb.SequencedWrite"; +} + + // =================================================================== void Lease::InitAsDefaultInstance() { @@ -4590,6 +4829,7 @@ const int TxnCoordMeta::kRefreshReadsFieldNumber; const int TxnCoordMeta::kRefreshWritesFieldNumber; const int TxnCoordMeta::kRefreshInvalidFieldNumber; const int TxnCoordMeta::kDeprecatedRefreshValidFieldNumber; +const int TxnCoordMeta::kOutstandingWritesFieldNumber; #endif // !defined(_MSC_VER) || _MSC_VER >= 1900 TxnCoordMeta::TxnCoordMeta() @@ -4604,7 +4844,8 @@ TxnCoordMeta::TxnCoordMeta(const TxnCoordMeta& from) _internal_metadata_(NULL), intents_(from.intents_), refresh_reads_(from.refresh_reads_), - refresh_writes_(from.refresh_writes_) { + refresh_writes_(from.refresh_writes_), + outstanding_writes_(from.outstanding_writes_) { _internal_metadata_.MergeFrom(from._internal_metadata_); if (from.has_txn()) { txn_ = new ::cockroach::roachpb::Transaction(*from.txn_); @@ -4650,6 +4891,7 @@ void TxnCoordMeta::Clear() { intents_.Clear(); refresh_reads_.Clear(); refresh_writes_.Clear(); + outstanding_writes_.Clear(); if (GetArenaNoVirtual() == NULL && txn_ != NULL) { delete txn_; } @@ -4762,6 +5004,17 @@ bool TxnCoordMeta::MergePartialFromCodedStream( break; } + case 8: { + if (static_cast< ::google::protobuf::uint8>(tag) == + static_cast< ::google::protobuf::uint8>(66u /* 66 & 0xFF */)) { + DO_(::google::protobuf::internal::WireFormatLite::ReadMessage( + input, add_outstanding_writes())); + } else { + goto handle_unusual; + } + break; + } + default: { handle_unusual: if (tag == 0) { @@ -4832,6 +5085,14 @@ void TxnCoordMeta::SerializeWithCachedSizes( ::google::protobuf::internal::WireFormatLite::WriteBool(7, this->refresh_invalid(), output); } + for (unsigned int i = 0, + n = static_cast(this->outstanding_writes_size()); i < n; i++) { + ::google::protobuf::internal::WireFormatLite::WriteMessage( + 8, + this->outstanding_writes(static_cast(i)), + output); + } + output->WriteRaw((::google::protobuf::internal::GetProto3PreserveUnknownsDefault() ? _internal_metadata_.unknown_fields() : _internal_metadata_.default_instance()).data(), static_cast((::google::protobuf::internal::GetProto3PreserveUnknownsDefault() ? _internal_metadata_.unknown_fields() : _internal_metadata_.default_instance()).size())); // @@protoc_insertion_point(serialize_end:cockroach.roachpb.TxnCoordMeta) @@ -4873,6 +5134,16 @@ size_t TxnCoordMeta::ByteSizeLong() const { } } + { + unsigned int count = static_cast(this->outstanding_writes_size()); + total_size += 1UL * count; + for (unsigned int i = 0; i < count; i++) { + total_size += + ::google::protobuf::internal::WireFormatLite::MessageSize( + this->outstanding_writes(static_cast(i))); + } + } + if (this->has_txn()) { total_size += 1 + ::google::protobuf::internal::WireFormatLite::MessageSize( @@ -4916,6 +5187,7 @@ void TxnCoordMeta::MergeFrom(const TxnCoordMeta& from) { intents_.MergeFrom(from.intents_); refresh_reads_.MergeFrom(from.refresh_reads_); refresh_writes_.MergeFrom(from.refresh_writes_); + outstanding_writes_.MergeFrom(from.outstanding_writes_); if (from.has_txn()) { mutable_txn()->::cockroach::roachpb::Transaction::MergeFrom(from.txn()); } @@ -4950,6 +5222,7 @@ void TxnCoordMeta::InternalSwap(TxnCoordMeta* other) { CastToBase(&intents_)->InternalSwap(CastToBase(&other->intents_)); CastToBase(&refresh_reads_)->InternalSwap(CastToBase(&other->refresh_reads_)); CastToBase(&refresh_writes_)->InternalSwap(CastToBase(&other->refresh_writes_)); + CastToBase(&outstanding_writes_)->InternalSwap(CastToBase(&other->outstanding_writes_)); swap(txn_, other->txn_); swap(command_count_, other->command_count_); swap(refresh_invalid_, other->refresh_invalid_); @@ -5003,6 +5276,9 @@ template<> GOOGLE_PROTOBUF_ATTRIBUTE_NOINLINE ::cockroach::roachpb::Transaction* template<> GOOGLE_PROTOBUF_ATTRIBUTE_NOINLINE ::cockroach::roachpb::Intent* Arena::CreateMaybeMessage< ::cockroach::roachpb::Intent >(Arena* arena) { return Arena::CreateInternal< ::cockroach::roachpb::Intent >(arena); } +template<> GOOGLE_PROTOBUF_ATTRIBUTE_NOINLINE ::cockroach::roachpb::SequencedWrite* Arena::CreateMaybeMessage< ::cockroach::roachpb::SequencedWrite >(Arena* arena) { + return Arena::CreateInternal< ::cockroach::roachpb::SequencedWrite >(arena); +} template<> GOOGLE_PROTOBUF_ATTRIBUTE_NOINLINE ::cockroach::roachpb::Lease* Arena::CreateMaybeMessage< ::cockroach::roachpb::Lease >(Arena* arena) { return Arena::CreateInternal< ::cockroach::roachpb::Lease >(arena); } diff --git a/c-deps/libroach/protos/roachpb/data.pb.h b/c-deps/libroach/protos/roachpb/data.pb.h index eb1b67584025..a7e10fbdc575 100644 --- a/c-deps/libroach/protos/roachpb/data.pb.h +++ b/c-deps/libroach/protos/roachpb/data.pb.h @@ -41,7 +41,7 @@ namespace protobuf_roachpb_2fdata_2eproto { struct TableStruct { static const ::google::protobuf::internal::ParseTableField entries[]; static const ::google::protobuf::internal::AuxillaryParseTableField aux[]; - static const ::google::protobuf::internal::ParseTable schema[15]; + static const ::google::protobuf::internal::ParseTable schema[16]; static const ::google::protobuf::internal::FieldMetadata field_metadata[]; static const ::google::protobuf::internal::SerializationTable serialization_table[]; static const ::google::protobuf::uint32 offsets[]; @@ -76,6 +76,9 @@ extern ModifiedSpanTriggerDefaultTypeInternal _ModifiedSpanTrigger_default_insta class ObservedTimestamp; class ObservedTimestampDefaultTypeInternal; extern ObservedTimestampDefaultTypeInternal _ObservedTimestamp_default_instance_; +class SequencedWrite; +class SequencedWriteDefaultTypeInternal; +extern SequencedWriteDefaultTypeInternal _SequencedWrite_default_instance_; class Span; class SpanDefaultTypeInternal; extern SpanDefaultTypeInternal _Span_default_instance_; @@ -107,6 +110,7 @@ template<> ::cockroach::roachpb::Lease* Arena::CreateMaybeMessage<::cockroach::r template<> ::cockroach::roachpb::MergeTrigger* Arena::CreateMaybeMessage<::cockroach::roachpb::MergeTrigger>(Arena*); template<> ::cockroach::roachpb::ModifiedSpanTrigger* Arena::CreateMaybeMessage<::cockroach::roachpb::ModifiedSpanTrigger>(Arena*); template<> ::cockroach::roachpb::ObservedTimestamp* Arena::CreateMaybeMessage<::cockroach::roachpb::ObservedTimestamp>(Arena*); +template<> ::cockroach::roachpb::SequencedWrite* Arena::CreateMaybeMessage<::cockroach::roachpb::SequencedWrite>(Arena*); template<> ::cockroach::roachpb::Span* Arena::CreateMaybeMessage<::cockroach::roachpb::Span>(Arena*); template<> ::cockroach::roachpb::SplitTrigger* Arena::CreateMaybeMessage<::cockroach::roachpb::SplitTrigger>(Arena*); template<> ::cockroach::roachpb::StoreIdent* Arena::CreateMaybeMessage<::cockroach::roachpb::StoreIdent>(Arena*); @@ -1765,6 +1769,121 @@ class Intent : public ::google::protobuf::MessageLite /* @@protoc_insertion_poin }; // ------------------------------------------------------------------- +class SequencedWrite : public ::google::protobuf::MessageLite /* @@protoc_insertion_point(class_definition:cockroach.roachpb.SequencedWrite) */ { + public: + SequencedWrite(); + virtual ~SequencedWrite(); + + SequencedWrite(const SequencedWrite& from); + + inline SequencedWrite& operator=(const SequencedWrite& from) { + CopyFrom(from); + return *this; + } + #if LANG_CXX11 + SequencedWrite(SequencedWrite&& from) noexcept + : SequencedWrite() { + *this = ::std::move(from); + } + + inline SequencedWrite& operator=(SequencedWrite&& from) noexcept { + if (GetArenaNoVirtual() == from.GetArenaNoVirtual()) { + if (this != &from) InternalSwap(&from); + } else { + CopyFrom(from); + } + return *this; + } + #endif + static const SequencedWrite& default_instance(); + + static void InitAsDefaultInstance(); // FOR INTERNAL USE ONLY + static inline const SequencedWrite* internal_default_instance() { + return reinterpret_cast( + &_SequencedWrite_default_instance_); + } + static constexpr int kIndexInFileMessages = + 12; + + void Swap(SequencedWrite* other); + friend void swap(SequencedWrite& a, SequencedWrite& b) { + a.Swap(&b); + } + + // implements Message ---------------------------------------------- + + inline SequencedWrite* New() const final { + return CreateMaybeMessage(NULL); + } + + SequencedWrite* New(::google::protobuf::Arena* arena) const final { + return CreateMaybeMessage(arena); + } + void CheckTypeAndMergeFrom(const ::google::protobuf::MessageLite& from) + final; + void CopyFrom(const SequencedWrite& from); + void MergeFrom(const SequencedWrite& from); + void Clear() final; + bool IsInitialized() const final; + + size_t ByteSizeLong() const final; + bool MergePartialFromCodedStream( + ::google::protobuf::io::CodedInputStream* input) final; + void SerializeWithCachedSizes( + ::google::protobuf::io::CodedOutputStream* output) const final; + void DiscardUnknownFields(); + int GetCachedSize() const final { return _cached_size_.Get(); } + + private: + void SharedCtor(); + void SharedDtor(); + void SetCachedSize(int size) const; + void InternalSwap(SequencedWrite* other); + private: + inline ::google::protobuf::Arena* GetArenaNoVirtual() const { + return NULL; + } + inline void* MaybeArenaPtr() const { + return NULL; + } + public: + + ::std::string GetTypeName() const final; + + // nested types ---------------------------------------------------- + + // accessors ------------------------------------------------------- + + void clear_key(); + static const int kKeyFieldNumber = 1; + const ::std::string& key() const; + void set_key(const ::std::string& value); + #if LANG_CXX11 + void set_key(::std::string&& value); + #endif + void set_key(const char* value); + void set_key(const void* value, size_t size); + ::std::string* mutable_key(); + ::std::string* release_key(); + void set_allocated_key(::std::string* key); + + // int32 sequence = 2; + void clear_sequence(); + static const int kSequenceFieldNumber = 2; + ::google::protobuf::int32 sequence() const; + void set_sequence(::google::protobuf::int32 value); + + // @@protoc_insertion_point(class_scope:cockroach.roachpb.SequencedWrite) + private: + + ::google::protobuf::internal::InternalMetadataWithArenaLite _internal_metadata_; + ::google::protobuf::internal::ArenaStringPtr key_; + ::google::protobuf::int32 sequence_; + mutable ::google::protobuf::internal::CachedSize _cached_size_; + friend struct ::protobuf_roachpb_2fdata_2eproto::TableStruct; +}; +// ------------------------------------------------------------------- + class Lease : public ::google::protobuf::MessageLite /* @@protoc_insertion_point(class_definition:cockroach.roachpb.Lease) */ { public: Lease(); @@ -1799,7 +1918,7 @@ class Lease : public ::google::protobuf::MessageLite /* @@protoc_insertion_point &_Lease_default_instance_); } static constexpr int kIndexInFileMessages = - 12; + 13; void Swap(Lease* other); friend void swap(Lease& a, Lease& b) { @@ -1966,7 +2085,7 @@ class AbortSpanEntry : public ::google::protobuf::MessageLite /* @@protoc_insert &_AbortSpanEntry_default_instance_); } static constexpr int kIndexInFileMessages = - 13; + 14; void Swap(AbortSpanEntry* other); friend void swap(AbortSpanEntry& a, AbortSpanEntry& b) { @@ -2093,7 +2212,7 @@ class TxnCoordMeta : public ::google::protobuf::MessageLite /* @@protoc_insertio &_TxnCoordMeta_default_instance_); } static constexpr int kIndexInFileMessages = - 14; + 15; void Swap(TxnCoordMeta* other); friend void swap(TxnCoordMeta& a, TxnCoordMeta& b) { @@ -2177,6 +2296,17 @@ class TxnCoordMeta : public ::google::protobuf::MessageLite /* @@protoc_insertio const ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::Span >& refresh_writes() const; + int outstanding_writes_size() const; + void clear_outstanding_writes(); + static const int kOutstandingWritesFieldNumber = 8; + ::cockroach::roachpb::SequencedWrite* mutable_outstanding_writes(int index); + ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::SequencedWrite >* + mutable_outstanding_writes(); + const ::cockroach::roachpb::SequencedWrite& outstanding_writes(int index) const; + ::cockroach::roachpb::SequencedWrite* add_outstanding_writes(); + const ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::SequencedWrite >& + outstanding_writes() const; + bool has_txn() const; void clear_txn(); static const int kTxnFieldNumber = 1; @@ -2213,6 +2343,7 @@ class TxnCoordMeta : public ::google::protobuf::MessageLite /* @@protoc_insertio ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::Span > intents_; ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::Span > refresh_reads_; ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::Span > refresh_writes_; + ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::SequencedWrite > outstanding_writes_; ::cockroach::roachpb::Transaction* txn_; ::google::protobuf::int32 command_count_; bool refresh_invalid_; @@ -3926,6 +4057,76 @@ inline void Intent::set_status(::cockroach::roachpb::TransactionStatus value) { // ------------------------------------------------------------------- +// SequencedWrite + +inline void SequencedWrite::clear_key() { + key_.ClearToEmptyNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); +} +inline const ::std::string& SequencedWrite::key() const { + // @@protoc_insertion_point(field_get:cockroach.roachpb.SequencedWrite.key) + return key_.GetNoArena(); +} +inline void SequencedWrite::set_key(const ::std::string& value) { + + key_.SetNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited(), value); + // @@protoc_insertion_point(field_set:cockroach.roachpb.SequencedWrite.key) +} +#if LANG_CXX11 +inline void SequencedWrite::set_key(::std::string&& value) { + + key_.SetNoArena( + &::google::protobuf::internal::GetEmptyStringAlreadyInited(), ::std::move(value)); + // @@protoc_insertion_point(field_set_rvalue:cockroach.roachpb.SequencedWrite.key) +} +#endif +inline void SequencedWrite::set_key(const char* value) { + GOOGLE_DCHECK(value != NULL); + + key_.SetNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited(), ::std::string(value)); + // @@protoc_insertion_point(field_set_char:cockroach.roachpb.SequencedWrite.key) +} +inline void SequencedWrite::set_key(const void* value, size_t size) { + + key_.SetNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited(), + ::std::string(reinterpret_cast(value), size)); + // @@protoc_insertion_point(field_set_pointer:cockroach.roachpb.SequencedWrite.key) +} +inline ::std::string* SequencedWrite::mutable_key() { + + // @@protoc_insertion_point(field_mutable:cockroach.roachpb.SequencedWrite.key) + return key_.MutableNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); +} +inline ::std::string* SequencedWrite::release_key() { + // @@protoc_insertion_point(field_release:cockroach.roachpb.SequencedWrite.key) + + return key_.ReleaseNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited()); +} +inline void SequencedWrite::set_allocated_key(::std::string* key) { + if (key != NULL) { + + } else { + + } + key_.SetAllocatedNoArena(&::google::protobuf::internal::GetEmptyStringAlreadyInited(), key); + // @@protoc_insertion_point(field_set_allocated:cockroach.roachpb.SequencedWrite.key) +} + +// int32 sequence = 2; +inline void SequencedWrite::clear_sequence() { + sequence_ = 0; +} +inline ::google::protobuf::int32 SequencedWrite::sequence() const { + // @@protoc_insertion_point(field_get:cockroach.roachpb.SequencedWrite.sequence) + return sequence_; +} +inline void SequencedWrite::set_sequence(::google::protobuf::int32 value) { + + sequence_ = value; + // @@protoc_insertion_point(field_set:cockroach.roachpb.SequencedWrite.sequence) +} + +// ------------------------------------------------------------------- + // Lease inline bool Lease::has_start() const { @@ -4493,6 +4694,35 @@ inline void TxnCoordMeta::set_deprecated_refresh_valid(bool value) { // @@protoc_insertion_point(field_set:cockroach.roachpb.TxnCoordMeta.deprecated_refresh_valid) } +inline int TxnCoordMeta::outstanding_writes_size() const { + return outstanding_writes_.size(); +} +inline void TxnCoordMeta::clear_outstanding_writes() { + outstanding_writes_.Clear(); +} +inline ::cockroach::roachpb::SequencedWrite* TxnCoordMeta::mutable_outstanding_writes(int index) { + // @@protoc_insertion_point(field_mutable:cockroach.roachpb.TxnCoordMeta.outstanding_writes) + return outstanding_writes_.Mutable(index); +} +inline ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::SequencedWrite >* +TxnCoordMeta::mutable_outstanding_writes() { + // @@protoc_insertion_point(field_mutable_list:cockroach.roachpb.TxnCoordMeta.outstanding_writes) + return &outstanding_writes_; +} +inline const ::cockroach::roachpb::SequencedWrite& TxnCoordMeta::outstanding_writes(int index) const { + // @@protoc_insertion_point(field_get:cockroach.roachpb.TxnCoordMeta.outstanding_writes) + return outstanding_writes_.Get(index); +} +inline ::cockroach::roachpb::SequencedWrite* TxnCoordMeta::add_outstanding_writes() { + // @@protoc_insertion_point(field_add:cockroach.roachpb.TxnCoordMeta.outstanding_writes) + return outstanding_writes_.Add(); +} +inline const ::google::protobuf::RepeatedPtrField< ::cockroach::roachpb::SequencedWrite >& +TxnCoordMeta::outstanding_writes() const { + // @@protoc_insertion_point(field_list:cockroach.roachpb.TxnCoordMeta.outstanding_writes) + return outstanding_writes_; +} + #ifdef __GNUC__ #pragma GCC diagnostic pop #endif // __GNUC__ @@ -4524,6 +4754,8 @@ inline void TxnCoordMeta::set_deprecated_refresh_valid(bool value) { // ------------------------------------------------------------------- +// ------------------------------------------------------------------- + // @@protoc_insertion_point(namespace_scope) diff --git a/docs/generated/settings/settings.html b/docs/generated/settings/settings.html index 52c0d3065e79..cf9017f9dbfe 100644 --- a/docs/generated/settings/settings.html +++ b/docs/generated/settings/settings.html @@ -32,6 +32,7 @@ kv.snapshot_recovery.max_ratebyte size8.0 MiBthe rate limit (bytes/sec) to use for recovery snapshots kv.transaction.max_intents_bytesinteger256000maximum number of bytes used to track write intents in transactions kv.transaction.max_refresh_spans_bytesinteger256000maximum number of bytes used to track refresh spans in serializable transactions +kv.transaction.write_pipelining_enabledbooleantrueif enabled, transactional writes are pipelined through Raft consensus rocksdb.min_wal_sync_intervalduration0sminimum duration between syncs of the RocksDB WAL server.clock.forward_jump_check_enabledbooleanfalseif enabled, forward clock jumps > max_offset/2 will cause a panic. server.clock.persist_upper_bound_intervalduration0sthe interval between persisting the wall time upper bound of the clock. The clock does not generate a wall time greater than the persisted timestamp and will panic if it sees a wall time greater than this value. When cockroach starts, it waits for the wall time to catch-up till this persisted timestamp. This guarantees monotonic wall time across server restarts. Not setting this or setting a value of 0 disables this feature. diff --git a/pkg/internal/client/txn.go b/pkg/internal/client/txn.go index cfa9c80a9cde..90b1722691c8 100644 --- a/pkg/internal/client/txn.go +++ b/pkg/internal/client/txn.go @@ -1220,7 +1220,7 @@ func (txn *Txn) GetStrippedTxnCoordMeta() roachpb.TxnCoordMeta { meta.RefreshReads = nil meta.RefreshWrites = nil case LeafTxn: - // Nothing yet. + meta.OutstandingWrites = nil } return meta } diff --git a/pkg/kv/dist_sender.go b/pkg/kv/dist_sender.go index db90b3565f4c..249c23626c40 100644 --- a/pkg/kv/dist_sender.go +++ b/pkg/kv/dist_sender.go @@ -483,7 +483,7 @@ func (ds *DistSender) initAndVerifyBatch( return roachpb.NewErrorf("batch with limit contains both forward and reverse scans") } - case *roachpb.ResolveIntentRangeRequest: + case *roachpb.QueryIntentRequest, *roachpb.ResolveIntentRangeRequest: continue case *roachpb.BeginTransactionRequest, *roachpb.EndTransactionRequest, *roachpb.ReverseScanRequest: @@ -633,11 +633,8 @@ func (ds *DistSender) Send( panic("EndTransaction not in last chunk of batch") } parts = splitBatchAndCheckForRefreshSpans(ba, true /* split ET */) - if len(parts) != 2 { - panic("split of final EndTransaction chunk resulted in != 2 parts") - } - // Restart transaction of the last chunk as two parts - // with EndTransaction in the second part. + // Restart transaction of the last chunk as multiple parts + // with EndTransaction in the last part. continue } if pErr != nil { @@ -741,7 +738,7 @@ func (ds *DistSender) divideAndSendBatchToRanges( // If we're in the middle of a panic, don't wait on responseChs. panic(r) } - var hadSuccess bool + var hadSuccessWriting bool for _, responseCh := range responseChs { resp := <-responseCh if resp.pErr != nil { @@ -750,7 +747,15 @@ func (ds *DistSender) divideAndSendBatchToRanges( } continue } - hadSuccess = true + if !hadSuccessWriting { + for _, i := range resp.positions { + req := ba.Requests[i].GetInner() + if !roachpb.IsReadOnly(req) { + hadSuccessWriting = true + break + } + } + } // Combine the new response with the existing one (including updating // the headers). @@ -775,7 +780,7 @@ func (ds *DistSender) divideAndSendBatchToRanges( // If this is a write batch with any successful responses, but // we're ultimately returning an error, wrap the error with a // MixedSuccessError. - if hadSuccess && ba.IsWrite() { + if hadSuccessWriting { pErr = roachpb.NewError(&roachpb.MixedSuccessError{Wrapped: pErr}) } } else if couldHaveSkippedResponses { diff --git a/pkg/kv/txn_coord_sender.go b/pkg/kv/txn_coord_sender.go index b0fa48b1f867..6a1db551caf1 100644 --- a/pkg/kv/txn_coord_sender.go +++ b/pkg/kv/txn_coord_sender.go @@ -142,9 +142,10 @@ type TxnCoordSender struct { // is embedded in the interceptorAlloc struct, so the entire stack is // allocated together with TxnCoordSender without any additional heap // allocations necessary. - interceptorStack [2]txnInterceptor + interceptorStack [3]txnInterceptor interceptorAlloc struct { txnIntentCollector + txnPipeliner txnSpanRefresher txnLockGatekeeper // not in interceptorStack array. } @@ -406,6 +407,9 @@ func (tcf *TxnCoordSenderFactory) TransactionalSender( st: tcf.st, ri: ri, } + tcs.interceptorAlloc.txnPipeliner = txnPipeliner{ + st: tcf.st, + } tcs.interceptorAlloc.txnSpanRefresher = txnSpanRefresher{ st: tcf.st, knobs: &tcf.testingKnobs, @@ -422,6 +426,7 @@ func (tcf *TxnCoordSenderFactory) TransactionalSender( } tcs.interceptorStack = [...]txnInterceptor{ &tcs.interceptorAlloc.txnIntentCollector, + &tcs.interceptorAlloc.txnPipeliner, &tcs.interceptorAlloc.txnSpanRefresher, } for i, reqInt := range tcs.interceptorStack { diff --git a/pkg/kv/txn_coord_sender_test.go b/pkg/kv/txn_coord_sender_test.go index 3090443ad05f..9fd4679a5ddc 100644 --- a/pkg/kv/txn_coord_sender_test.go +++ b/pkg/kv/txn_coord_sender_test.go @@ -343,7 +343,9 @@ func TestTxnCoordSenderCondenseIntentSpans(t *testing.T) { t.Errorf("expected end transaction to have intents %+v; got %+v", e, a) } } - return args.CreateReply(), nil + resp := args.CreateReply() + resp.Txn = args.Txn + return resp, nil } ambient := log.AmbientContext{Tracer: tracing.NewTracer()} ds := NewDistSender( @@ -842,6 +844,7 @@ func TestTxnCoordSenderTxnUpdatedOnError(t *testing.T) { pErr := test.pErrGen(ba.Txn) if pErr == nil { reply = ba.CreateReply() + reply.Txn = ba.Txn } return reply, pErr } @@ -1422,8 +1425,7 @@ func TestAbortTransactionOnCommitErrors(t *testing.T) { ) (*roachpb.BatchResponse, *roachpb.Error) { br := ba.CreateReply() - switch req := ba.Requests[0].GetInner().(type) { - case *roachpb.BeginTransactionRequest: + if _, hasBT := ba.GetArg(roachpb.BeginTransaction); hasBT { if _, ok := ba.Requests[1].GetInner().(*roachpb.PutRequest); !ok { t.Fatalf("expected Put") } @@ -1437,8 +1439,8 @@ func TestAbortTransactionOnCommitErrors(t *testing.T) { br.Txn.Writing = true br.Txn.Status = roachpb.PENDING } - case *roachpb.EndTransactionRequest: - if req.Commit { + } else if et, hasET := ba.GetArg(roachpb.EndTransaction); hasET { + if et.(*roachpb.EndTransactionRequest).Commit { commit.Store(true) if test.errFn != nil { return nil, test.errFn(*ba.Txn) @@ -1446,7 +1448,7 @@ func TestAbortTransactionOnCommitErrors(t *testing.T) { return nil, roachpb.NewErrorWithTxn(test.err, ba.Txn) } abort.Store(true) - default: + } else { t.Fatalf("unexpected batch: %s", ba) } return br, nil @@ -1540,7 +1542,9 @@ func TestRollbackErrorStopsHeartbeat(t *testing.T) { sender.match(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { if _, ok := ba.GetArg(roachpb.EndTransaction); !ok { - return nil, nil + resp := ba.CreateReply() + resp.Txn = ba.Txn + return resp, nil } return nil, roachpb.NewErrorf("injected err") }) diff --git a/pkg/kv/txn_interceptor_pipeliner.go b/pkg/kv/txn_interceptor_pipeliner.go new file mode 100644 index 000000000000..56309fb01bcf --- /dev/null +++ b/pkg/kv/txn_interceptor_pipeliner.go @@ -0,0 +1,502 @@ +// Copyright 2018 The Cockroach Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +// implied. See the License for the specific language governing +// permissions and limitations under the License. + +package kv + +import ( + "context" + + "github.com/google/btree" + + "github.com/cockroachdb/cockroach/pkg/roachpb" + "github.com/cockroachdb/cockroach/pkg/settings" + "github.com/cockroachdb/cockroach/pkg/settings/cluster" + "github.com/cockroachdb/cockroach/pkg/util/log" +) + +// The degree of the outstandingWrites btree. +const txnPipelinerBtreeDegree = 32 + +var pipelinedWritesEnabled = settings.RegisterBoolSetting( + "kv.transaction.write_pipelining_enabled", + "if enabled, transactional writes are pipelined through Raft consensus", + true, +) + +// txnPipeliner is a txnInterceptor that pipelines transactional writes by using +// asynchronous consensus. The interceptor then tracks all writes that have been +// asynchronously proposed through Raft and ensures that all interfering +// requests chain on to them by first proving that the async writes succeeded. +// The interceptor also ensures that when committing a transaction all writes +// that have been proposed but not proven to have succeeded are first checked +// before committing. These async writes are referred to as "outstanding writes" +// and this process of proving that an outstanding write succeeded is called +// "proving" the write. +// +// Chaining on to in-flight async writes is important for two main reasons to +// txnPipeliner: +// 1. requests proposed to Raft will not necessarily succeed. For any number of +// reasons, the request may make it through Raft and be discarded or fail to +// ever even be replicated. A transaction must check that all async writes +// succeeded before committing. However, when these proposals do fail, their +// errors aren't particularly interesting to a transaction. This is because +// these errors are not deterministic Transaction-domain errors that a +// transaction must adhere to for correctness such as conditional-put errors or +// other symptoms of constraint violations. These kinds of errors are all +// discovered during write *evaluation*, which an async write will perform +// synchronously before consensus. Any error during consensus is outside of the +// Transaction-domain and can always trigger a transaction retry. +// 2. transport layers beneath the txnPipeliner do not provide strong enough +// ordering guarantees between concurrent requests in the same transaction to +// avoid needing explicit chaining. For instance, DistSender uses unary gRPC +// requests instead of gRPC streams, so it can't natively expose strong ordering +// guarantees. Perhaps more importantly, even when a command has entered the +// command queue and evaluated on a Replica, it is not guaranteed to be applied +// before interfering commands. This is because the command may be retried +// outside of the serialization of the command queue for any number of reasons, +// such as leaseholder changes. When the command re-enters the command queue, +// it's possible that interfering commands may jump ahead of it. To combat +// this, the txnPipeliner uses chaining to throw an error when these +// re-orderings would have affected the order that transactional requests +// evaluate in. +// +// The interceptor proves all outstanding writes before committing a transaction +// by tacking on a QueryIntent request for each one to the front of an +// EndTransaction(Commit=true) requests. The result of this is that the +// EndTransaction needs to wait at the DistSender level for all of QueryIntent +// requests to succeed at before executing itself [1]. This is a little +// unfortunate because a transaction could have accumulated a large number of +// outstanding writes without proving any of them, and the more of these writes +// there are, the more chance querying one of them gets delayed and delays the +// overall transaction. +// +// Three approaches have been considered to address this, all of which revolve +// around the idea that earlier writes in a transaction may have finished +// consensus well before the EndTransaction is sent. Following this logic, it +// would be in the txnPipeliner's best interest to prove outstanding writes as +// early as possible, even if no other overlapping requests force them to be +// proven. The approaches are: +// 1. launch a background process after each successful async write to query its +// intents and wait for it to succeed. This would effectively solve the issue, +// but at the cost of many more goroutines and many more QueryIntent requests, +// most of which would be redundant because their corresponding write wouldn't +// complete until after an EndTransaction synchronously needed to prove them +// anyway. +// 2. to address the issue of an unbounded number of background goroutines +// proving writes in approach 1, a single background goroutine could be run +// that repeatedly loops over all outstanding writes and attempts to prove +// them. This approach was used in an early revision of #26599 and has the nice +// property that only one batch of QueryIntent requests is ever active at a +// given time. It may be revisited, but for now it is not used for the same +// reason as approach 1: most of its QueryIntent requests will be useless +// because a transaction will send an EndTransaction immediately after sending +// all of its writes. +// 3. turn the KV interface into a streaming protocol (#8360) that could support +// returning multiple results. This would allow clients to return immediately +// after a writes "evaluation" phase completed but hold onto a handle to the +// request and be notified immediately after its "replication" phase completes. +// This would allow txnPipeliner to prove outstanding writes immediately after +// they finish consensus without any extra RPCs. +// So far, none of these approaches have been integrated. +// +// [1] A proposal called "parallel commits" (#24194) exists that would allow all +// QueryIntent requests and the EndTransaction request that they are prepended +// to to be sent by the DistSender in parallel. This would help with this +// issue by hiding the cost of the QueryIntent requests behind the cost of the +// "staging" EndTransaction request. +// +type txnPipeliner struct { + st *cluster.Settings + wrapped lockedSender + + outstandingWrites *btree.BTree + owAlloc outstandingWriteAlloc + tmpOW1, tmpOW2 outstandingWrite // avoid allocs +} + +// outstandingWrites represent a commitment to proving (via QueryIntent) that +// a point write succeeded in replicating an intent with a specific sequence +// number. +type outstandingWrite struct { + roachpb.SequencedWrite +} + +// Less implements the btree.Item interface. +func (a *outstandingWrite) Less(b btree.Item) bool { + return a.Key.Compare(b.(*outstandingWrite).Key) < 0 +} + +// SendLocked implements the lockedSender interface. +func (tp *txnPipeliner) SendLocked( + ctx context.Context, ba roachpb.BatchRequest, +) (*roachpb.BatchResponse, *roachpb.Error) { + // Fast-path for 1PC transactions. + _, hasBT := ba.GetArg(roachpb.BeginTransaction) + _, hasET := ba.GetArg(roachpb.EndTransaction) + if hasBT && hasET { + return tp.wrapped.SendLocked(ctx, ba) + } + + // Adjust the batch so that it doesn't miss any outstanding writes. + ba = tp.chainToOutstandingWrites(ba) + + // Send through wrapped lockedSender. Unlocks while sending then re-locks. + br, pErr := tp.wrapped.SendLocked(ctx, ba) + if pErr != nil { + return nil, tp.adjustError(ctx, ba, pErr) + } + + // WIP: I think it's possible for this response to be from an earlier + // epoch. Fix that. + + // Prove any outstanding writes that we proved to exist. + br = tp.updateOutstandingWrites(ctx, ba, br) + return br, nil +} + +// chainToOutstandingWrites ensures that we "chain" on to any outstanding writes +// that overlap the keys we're trying to read/write. We do this by prepending +// QueryIntent requests with the THROW_ERROR behavior before each request that +// touches any of the outstanding writes. In effect, this allows us to prove +// that a write succeeded before depending on its existence. We later prune down +// the list of writes we proved to exist that are no longer "outstanding" in +// updateOutstandingWrites. +// +// TODO(nvanbenschoten): the number of writes that we permit to use async +// consensus in a given Batch should be bound. We'll have to prove each one +// using a QueryIntent, so there's a point where it makes more sense to just +// perform consensus for the entire batch synchronously and avoid all of the +// overhead of pipelining. We might also want to bound the size of the +// outstandingWrites tree. +func (tp *txnPipeliner) chainToOutstandingWrites(ba roachpb.BatchRequest) roachpb.BatchRequest { + asyncConsensus := tp.st.Version.IsActive(cluster.VersionAsyncConsensus) && + pipelinedWritesEnabled.Get(&tp.st.SV) + + forked := false + oldReqs := ba.Requests + // TODO(nvanbenschoten): go 1.11 includes an optimization to quickly clear + // out an entire map. That might make it cost effective to maintain a single + // chainedKeys map between calls to this function. + var chainedKeys map[string]struct{} + for i, ru := range oldReqs { + if !asyncConsensus && !forked && tp.outstandingWritesLen() == len(chainedKeys) { + // If there are no outstanding writes or all outstanding writes + // have been chained onto and async consensus is disallowed, + // short-circuit immediately. + break + } + + req := ru.GetInner() + if req.Method() == roachpb.BeginTransaction { + // Ignore BeginTransaction requests. They'll always be the first + // request in a batch and will never need to chain on any existing + // writes. + continue + } + if !roachpb.IsTransactionWrite(req) || roachpb.IsRange(req) { + // Only allow batches consisting of solely transactional point + // writes to perform consensus asynchronously. + // TODO(nvanbenschoten): We could allow batches with reads and point + // writes to perform async consensus, but this would be a bit + // tricky. Any read would need to chain on to any write that came + // before it in the batch and overlaps. For now, it doesn't seem + // worth it. + asyncConsensus = false + } + + if tp.outstandingWritesLen() > len(chainedKeys) { + // For each conflicting outstanding write, add a QueryIntent request + // to the batch to assert that it has succeeded and "chain" onto it. + itemIter := func(item btree.Item) bool { + // We don't want to modify the batch's request slice directly, + // so fork it before modifying it. + if !forked { + ba.Requests = append([]roachpb.RequestUnion(nil), ba.Requests[:i]...) + forked = true + } + + w := item.(*outstandingWrite) + if _, ok := chainedKeys[string(w.Key)]; !ok { + // The write has not already been chained onto by an earlier + // request in this batch. Add a QueryIntent request to the + // batch (before the conflicting request) to ensure that we + // chain on to the success of the outstanding write. + meta := ba.Txn.TxnMeta + meta.Sequence = w.Sequence + ba.Add(&roachpb.QueryIntentRequest{ + RequestHeader: roachpb.RequestHeader{ + Key: w.Key, + }, + Txn: meta, + // Set the IfMissing behavior to return an error if the + // outstanding write is missing. + IfMissing: roachpb.QueryIntentRequest_RETURN_ERROR, + }) + + // Record that the key has been chained onto at least once + // in this batch so that we don't chain onto it again. + if chainedKeys == nil { + chainedKeys = make(map[string]struct{}) + } + chainedKeys[string(w.Key)] = struct{}{} + } + return true + } + + if !roachpb.IsTransactional(req) { + // Non-transactional requests require that we stall the entire + // pipeline by chaining on to all outstanding writes. This is + // because their request header is often insufficient to + // determine all of the keys that they will interact with. + tp.outstandingWrites.Ascend(itemIter) + } else if et, ok := req.(*roachpb.EndTransactionRequest); ok { + if et.Commit { + // EndTransactions need to prove all outstanding writes before + // being allowed to succeed themselves. + tp.outstandingWrites.Ascend(itemIter) + } + } else { + // Transactional reads and writes needs to chain on to any + // overlapping outstanding writes. + r := req.Header().Span().AsRange() + tp.tmpOW1.Key, tp.tmpOW2.Key = roachpb.Key(r.Start), roachpb.Key(r.End) + tp.outstandingWrites.AscendRange(&tp.tmpOW1, &tp.tmpOW2, itemIter) + } + } + + // If the BatchRequest's slice of requests has been forked from the original, + // append the request to the new slice. + if forked { + ba.Add(req) + } + } + + // Set the batch's AsyncConsensus flag based on whether AsyncConsensus is + // permitted for the batch. + ba.AsyncConsensus = asyncConsensus + return ba +} + +// updateOutstandingWrites reads the response for the given request and uses +// it to update the tracked outstanding write set. It does so by performing +// two actions: +// 1. it removes all outstanding writes that the request proved to exist from +// the outstanding writes set. +// 2. it adds all async writes the the request performed to the outstanding +// write set. +// +// While doing so, the method also strips all QueryIntent responses from the +// BatchResponse, hiding the fact that they were added in the first place. +func (tp *txnPipeliner) updateOutstandingWrites( + ctx context.Context, ba roachpb.BatchRequest, br *roachpb.BatchResponse, +) *roachpb.BatchResponse { + // If the transaction is no longer pending, clear the outstanding writes + // tree. This will turn maybeRemoveProvenWriteLocked into a quick no-op. + if br.Txn != nil && br.Txn.Status != roachpb.PENDING && tp.outstandingWrites != nil { + tp.outstandingWrites.Clear(false /* addNodesToFreelist */) + } + + j := 0 + for i, ru := range ba.Requests { + req := ru.GetInner() + resp := br.Responses[i].GetInner() + + if qiReq, ok := req.(*roachpb.QueryIntentRequest); ok { + // Remove any outstanding writes that were proven to exist. + // It shouldn't be possible for a QueryIntentRequest with + // an IfMissing behavior of RETURN_ERROR to return without + // error and with with FoundIntent=false, but we handle that + // case here because it happens a lot in tests. + if resp.(*roachpb.QueryIntentResponse).FoundIntent { + tp.maybeRemoveProvenWriteLocked(qiReq.Key, qiReq.Txn.Sequence) + } + } else { + // Hide the fact that this interceptor added new requests to the batch. + br.Responses[j] = br.Responses[i] + j++ + + // Record any writes that were performed asynchronously. We'll + // need to prove that these succeeded sometime before we commit. + if ba.AsyncConsensus && req.Method() != roachpb.BeginTransaction { + header := req.Header() + tp.maybeInsertOutstandingWriteLocked(header.Key, header.Sequence) + } + } + } + // Hide the fact that this interceptor added new requests to the batch. + br.Responses = br.Responses[:j] + return br +} + +// adjustError adjusts the provided error based on the request that caused it. +// It transforms any IntentMissingError into a TransactionRetryError and fixes +// the error's index position. +func (tp *txnPipeliner) adjustError( + ctx context.Context, ba roachpb.BatchRequest, pErr *roachpb.Error, +) *roachpb.Error { + // Fix the error index to hide the impact of any QueryIntent requests. + if pErr.Index != nil { + before := int32(0) + for _, ru := range ba.Requests[:int(pErr.Index.Index)] { + req := ru.GetInner() + if req.Method() == roachpb.QueryIntent { + before++ + } + } + pErr.Index.Index -= before + } + + // Turn an IntentMissingError into a transactional retry error. + if ime, ok := pErr.GetDetail().(*roachpb.IntentMissingError); ok { + log.VEventf(ctx, 2, "transforming intent missing error into retry: %v", ime) + err := roachpb.NewTransactionRetryError(roachpb.RETRY_ASYNC_WRITE_FAILURE) + retryErr := roachpb.NewErrorWithTxn(err, pErr.GetTxn()) + retryErr.Index = pErr.Index + return retryErr + } + return pErr +} + +// setWrapped implements the txnInterceptor interface. +func (tp *txnPipeliner) setWrapped(wrapped lockedSender) { tp.wrapped = wrapped } + +// populateMetaLocked implements the txnReqInterceptor interface. +func (tp *txnPipeliner) populateMetaLocked(meta *roachpb.TxnCoordMeta) { + if l := tp.outstandingWritesLen(); l > 0 { + meta.OutstandingWrites = make([]roachpb.SequencedWrite, 0, l) + tp.outstandingWrites.Ascend(func(item btree.Item) bool { + w := item.(*outstandingWrite) + meta.OutstandingWrites = append(meta.OutstandingWrites, w.SequencedWrite) + return true + }) + } +} + +// augmentMetaLocked implements the txnReqInterceptor interface. +func (tp *txnPipeliner) augmentMetaLocked(meta roachpb.TxnCoordMeta) { + // Copy all outstanding writes into the outstandingWrite tree. + for _, w := range meta.OutstandingWrites { + tp.maybeInsertOutstandingWriteLocked(w.Key, w.Sequence) + } +} + +// epochBumpedLocked implements the txnReqInterceptor interface. +func (tp *txnPipeliner) epochBumpedLocked() { + // Clear out the outstandingWrites set and free associated memory. + if tp.outstandingWrites != nil { + // Add nodes to freelist so that next epoch can reuse btree memory. + tp.outstandingWrites.Clear(true /* addNodesToFreelist */) + tp.owAlloc.Clear() + } +} + +// closeLocked implements the txnReqInterceptor interface. +func (tp *txnPipeliner) closeLocked() {} + +// outstandingWritesLen returns the number of writes that are outstanding. +func (tp *txnPipeliner) outstandingWritesLen() int { + if tp.outstandingWrites == nil { + return 0 + } + return tp.outstandingWrites.Len() +} + +// maybeInsertOutstandingWriteLocked attempts to insert an outstanding write +// that has not been proven to have succeeded into the txnPipeliners outstanding +// write map. +func (tp *txnPipeliner) maybeInsertOutstandingWriteLocked(key roachpb.Key, seq int32) { + if tp.outstandingWrites == nil { + // Lazily initialize btree. + tp.outstandingWrites = btree.New(txnPipelinerBtreeDegree) + } + + tp.tmpOW1.Key = key + item := tp.outstandingWrites.Get(&tp.tmpOW1) + if item != nil { + otherW := item.(*outstandingWrite) + if seq > otherW.Sequence { + // Existing outstanding write has old information. + otherW.Sequence = seq + } + return + } + + w := tp.owAlloc.Alloc(key, seq) + tp.outstandingWrites.ReplaceOrInsert(w) +} + +// maybeRemoveProvenWriteLocked attempts to remove an outstanding write that +// was proven to have succeeded. The method will be a no-op if the write was +// already proved. Care is taken not to accidentally remove a write to the +// same key but at a later epoch or sequence number. +func (tp *txnPipeliner) maybeRemoveProvenWriteLocked(key roachpb.Key, seq int32) { + tp.tmpOW1.Key = key + item := tp.outstandingWrites.Get(&tp.tmpOW1) + if item == nil { + // The write was already proven or the txn epoch was incremented. + return + } + + w := item.(*outstandingWrite) + if seq < w.Sequence { + // The sequence might have changed, which means that a new write was + // sent to the same key. This write would have been forced to prove + // the existence of current write already. + return + } + + // Delete the write from the outstanding writes set. + delItem := tp.outstandingWrites.Delete(item) + if delItem != nil { + *delItem.(*outstandingWrite) = outstandingWrite{} // for GC + } +} + +// outstandingWriteAlloc provides chunk allocation of outstandingWrites, +// amortizing the overhead of each allocation. +type outstandingWriteAlloc []outstandingWrite + +// Alloc allocates a new outstandingWrite with the specified key and sequence +// number. +func (a *outstandingWriteAlloc) Alloc(key roachpb.Key, seq int32) *outstandingWrite { + // If the current alloc slice has no extra capacity, reallocate a new chunk. + if cap(*a)-len(*a) == 0 { + const chunkAllocMinSize = 4 + const chunkAllocMaxSize = 1024 + + allocSize := cap(*a) * 2 + if allocSize < chunkAllocMinSize { + allocSize = chunkAllocMinSize + } else if allocSize > chunkAllocMaxSize { + allocSize = chunkAllocMaxSize + } + *a = make([]outstandingWrite, 0, allocSize) + } + + *a = (*a)[:len(*a)+1] + w := &(*a)[len(*a)-1] + *w = outstandingWrite{ + SequencedWrite: roachpb.SequencedWrite{Key: key, Sequence: seq}, + } + return w +} + +// Clear removes all allocated outstanding writes and attempts to reclaim as +// much allocated memory as possible. +func (a *outstandingWriteAlloc) Clear() { + for i := range *a { + (*a)[i] = outstandingWrite{} // for GC + } + *a = (*a)[:0] +} diff --git a/pkg/kv/txn_interceptor_pipeliner_test.go b/pkg/kv/txn_interceptor_pipeliner_test.go new file mode 100644 index 000000000000..535c9b596ab2 --- /dev/null +++ b/pkg/kv/txn_interceptor_pipeliner_test.go @@ -0,0 +1,860 @@ +// Copyright 2018 The Cockroach Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +// implied. See the License for the specific language governing +// permissions and limitations under the License. + +package kv + +import ( + "context" + "fmt" + "strings" + "testing" + + "github.com/google/btree" + "github.com/stretchr/testify/require" + + "github.com/cockroachdb/cockroach/pkg/roachpb" + "github.com/cockroachdb/cockroach/pkg/settings/cluster" + "github.com/cockroachdb/cockroach/pkg/util/hlc" + "github.com/cockroachdb/cockroach/pkg/util/leaktest" +) + +// mockLockedSender implements the lockedSender interface and provides a way to +// mock out and adjust the SendLocked method. +type mockLockedSender struct { + mockFn func(roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) +} + +func (m *mockLockedSender) SendLocked( + ctx context.Context, ba roachpb.BatchRequest, +) (*roachpb.BatchResponse, *roachpb.Error) { + return m.mockFn(ba) +} + +func (m *mockLockedSender) MockSend( + fn func(roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error), +) { + m.mockFn = fn +} + +func makeMockTxnPipeliner() (txnPipeliner, *mockLockedSender) { + mockSender := &mockLockedSender{} + return txnPipeliner{ + st: cluster.MakeTestingClusterSettings(), + wrapped: mockSender, + }, mockSender +} + +func makeTxnProto() roachpb.Transaction { + return roachpb.MakeTransaction("test", []byte("key"), 0, 0, hlc.Timestamp{}, 0) +} + +// TestTxnPipeliner1PCTransaction tests that 1PC transactions pass through the +// txnPipeliner untouched. +func TestTxnPipeliner1PCTransaction(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA := roachpb.Key("a") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.BeginTransactionRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + ba.Add(&roachpb.EndTransactionRequest{Commit: true}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 3, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.BeginTransactionRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + require.IsType(t, &roachpb.EndTransactionRequest{}, ba.Requests[2].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + br.Txn.Status = roachpb.COMMITTED + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerTrackOutstandingWrites tests that txnPipeliner tracks writes +// that were performed with async consensus. It also tests that these writes are +// proved as requests are chained onto them. Finally, it tests that EndTxn +// requests chain on to all existing requests. +func TestTxnPipelinerTrackOutstandingWrites(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA := roachpb.Key("a") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.BeginTransactionRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + putArgs := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}} + putArgs.Sequence = 2 + ba.Add(&putArgs) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.True(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.BeginTransactionRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 1, tp.outstandingWritesLen()) + + w := tp.outstandingWrites.Min().(*outstandingWrite) + require.Equal(t, putArgs.Key, w.Key) + require.Equal(t, putArgs.Sequence, w.Sequence) + + // More writes, one that replaces the other's sequence number. + keyB, keyC := roachpb.Key("b"), roachpb.Key("c") + ba.Requests = nil + cputArgs := roachpb.ConditionalPutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}} + cputArgs.Sequence = 3 + ba.Add(&cputArgs) + initPutArgs := roachpb.InitPutRequest{RequestHeader: roachpb.RequestHeader{Key: keyB}} + initPutArgs.Sequence = 4 + ba.Add(&initPutArgs) + incArgs := roachpb.IncrementRequest{RequestHeader: roachpb.RequestHeader{Key: keyC}} + incArgs.Sequence = 5 + ba.Add(&incArgs) + // Write at the same key as another write in the same batch. Will only + // result in a single outstanding write, at the larger sequence number. + delArgs := roachpb.DeleteRequest{RequestHeader: roachpb.RequestHeader{Key: keyC}} + delArgs.Sequence = 6 + ba.Add(&delArgs) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 5, len(ba.Requests)) + require.True(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.ConditionalPutRequest{}, ba.Requests[1].GetInner()) + require.IsType(t, &roachpb.InitPutRequest{}, ba.Requests[2].GetInner()) + require.IsType(t, &roachpb.IncrementRequest{}, ba.Requests[3].GetInner()) + require.IsType(t, &roachpb.DeleteRequest{}, ba.Requests[4].GetInner()) + + qiReq := ba.Requests[0].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, keyA, qiReq.Key) + require.Equal(t, txn.ID, qiReq.Txn.ID) + require.Equal(t, txn.Timestamp, qiReq.Txn.Timestamp) + require.Equal(t, int32(2), qiReq.Txn.Sequence) + require.Equal(t, roachpb.QueryIntentRequest_RETURN_ERROR, qiReq.IfMissing) + + // No outstanding writes have been proved yet. + require.Equal(t, 1, tp.outstandingWritesLen()) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Responses[0].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Equal(t, 4, len(br.Responses)) // QueryIntent response stripped + require.IsType(t, &roachpb.ConditionalPutResponse{}, br.Responses[0].GetInner()) + require.IsType(t, &roachpb.InitPutResponse{}, br.Responses[1].GetInner()) + require.IsType(t, &roachpb.IncrementResponse{}, br.Responses[2].GetInner()) + require.IsType(t, &roachpb.DeleteResponse{}, br.Responses[3].GetInner()) + require.Nil(t, pErr) + require.Equal(t, 3, tp.outstandingWritesLen()) + + wMin := tp.outstandingWrites.Min().(*outstandingWrite) + require.Equal(t, cputArgs.Key, wMin.Key) + require.Equal(t, cputArgs.Sequence, wMin.Sequence) + wMax := tp.outstandingWrites.Max().(*outstandingWrite) + require.Equal(t, delArgs.Key, wMax.Key) + require.Equal(t, delArgs.Sequence, wMax.Sequence) + + // Send a final write, along with an EndTransaction request. Should attempt + // to prove all outstanding writes. Should NOT use async consensus. + keyD := roachpb.Key("d") + ba.Requests = nil + putArgs2 := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyD}} + putArgs2.Sequence = 7 + ba.Add(&putArgs2) + ba.Add(&roachpb.EndTransactionRequest{Commit: true}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 5, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[1].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[2].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[3].GetInner()) + require.IsType(t, &roachpb.EndTransactionRequest{}, ba.Requests[4].GetInner()) + + qiReq1 := ba.Requests[1].GetInner().(*roachpb.QueryIntentRequest) + qiReq2 := ba.Requests[2].GetInner().(*roachpb.QueryIntentRequest) + qiReq3 := ba.Requests[3].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, keyA, qiReq1.Key) + require.Equal(t, keyB, qiReq2.Key) + require.Equal(t, keyC, qiReq3.Key) + require.Equal(t, int32(3), qiReq1.Txn.Sequence) + require.Equal(t, int32(4), qiReq2.Txn.Sequence) + require.Equal(t, int32(6), qiReq3.Txn.Sequence) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Txn.Status = roachpb.COMMITTED + br.Responses[1].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + br.Responses[2].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + br.Responses[3].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Equal(t, 2, len(br.Responses)) // QueryIntent response stripped + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerReads tests that txnPipeliner will never instruct batches +// with reads in them to use async consensus. It also tests that these reading +// batches will still chain on to outstanding writers, if necessary. +func TestTxnPipelinerReads(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA, keyC := roachpb.Key("a"), roachpb.Key("c") + + // Read-only. + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.GetRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 1, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.GetRequest{}, ba.Requests[0].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + + // Read before write. + ba.Requests = nil + ba.Add(&roachpb.GetRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyC}}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.GetRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + br = ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + + // Read after write. + ba.Requests = nil + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyC}}) + ba.Add(&roachpb.GetRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.GetRequest{}, ba.Requests[1].GetInner()) + + br = ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + + // Add a key into the outstanding writes set. + tp.maybeInsertOutstandingWriteLocked(keyA, 10) + require.Equal(t, 1, tp.outstandingWritesLen()) + + // Read-only with conflicting outstanding write. + ba.Requests = nil + ba.Add(&roachpb.GetRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.GetRequest{}, ba.Requests[1].GetInner()) + + qiReq := ba.Requests[0].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, keyA, qiReq.Key) + require.Equal(t, int32(10), qiReq.Txn.Sequence) + + // No outstanding writes have been proved yet. + require.Equal(t, 1, tp.outstandingWritesLen()) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Responses[0].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerRangedWrites tests that txnPipeliner will never perform +// ranged write operations using async consensus. It also tests that ranged +// writes will correctly chain on to existing outstanding writes. +func TestTxnPipelinerRangedWrites(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA, keyD := roachpb.Key("a"), roachpb.Key("d") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + ba.Add(&roachpb.DeleteRangeRequest{RequestHeader: roachpb.RequestHeader{Key: keyA, EndKey: keyD}}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.DeleteRangeRequest{}, ba.Requests[1].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + // The PutRequest was not run asynchronously, so it is not outstanding. + require.Equal(t, 0, tp.outstandingWritesLen()) + + // Add five keys into the outstanding writes set, one of which overlaps with + // the Put request and two others which also overlap with the DeleteRange + // request. Send the batch again and assert that the Put chains onto the + // first outstanding write and the DeleteRange chains onto the second and + // third outstanding write. + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("a"), 10) + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("b"), 11) + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("c"), 12) + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("d"), 13) + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("e"), 13) + require.Equal(t, 5, tp.outstandingWritesLen()) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 5, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[2].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[3].GetInner()) + require.IsType(t, &roachpb.DeleteRangeRequest{}, ba.Requests[4].GetInner()) + + qiReq1 := ba.Requests[0].GetInner().(*roachpb.QueryIntentRequest) + qiReq2 := ba.Requests[2].GetInner().(*roachpb.QueryIntentRequest) + qiReq3 := ba.Requests[3].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, roachpb.Key("a"), qiReq1.Key) + require.Equal(t, roachpb.Key("b"), qiReq2.Key) + require.Equal(t, roachpb.Key("c"), qiReq3.Key) + require.Equal(t, txn.ID, qiReq1.Txn.ID) + require.Equal(t, txn.ID, qiReq2.Txn.ID) + require.Equal(t, txn.ID, qiReq3.Txn.ID) + require.Equal(t, int32(10), qiReq1.Txn.Sequence) + require.Equal(t, int32(11), qiReq2.Txn.Sequence) + require.Equal(t, int32(12), qiReq3.Txn.Sequence) + + // No outstanding writes have been proved yet. + require.Equal(t, 5, tp.outstandingWritesLen()) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Responses[0].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + br.Responses[2].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + br.Responses[3].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 2, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerNonTransactionalRequests tests that non-transaction requests +// cause the txnPipeliner to stall its entire pipeline. +func TestTxnPipelinerNonTransactionalRequests(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA, keyC := roachpb.Key("a"), roachpb.Key("c") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyC}}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.True(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 2, tp.outstandingWritesLen()) + + // Send a non-transactional request. Should stall pipeline and chain onto + // all outstanding writes, even if its header doesn't imply any interaction. + keyRangeDesc := roachpb.Key("rangeDesc") + ba.Requests = nil + ba.Add(&roachpb.GetSnapshotForMergeRequest{ + RequestHeader: roachpb.RequestHeader{Key: keyRangeDesc}, + }) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 3, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[1].GetInner()) + require.IsType(t, &roachpb.GetSnapshotForMergeRequest{}, ba.Requests[2].GetInner()) + + qiReq1 := ba.Requests[0].GetInner().(*roachpb.QueryIntentRequest) + qiReq2 := ba.Requests[1].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, keyA, qiReq1.Key) + require.Equal(t, keyC, qiReq2.Key) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Responses[0].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + br.Responses[1].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerManyWrites tests that a txnPipeliner behaves correctly even +// when its outstanding write tree grows to a very large size. +func TestTxnPipelinerManyWrites(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + const writes = 2048 + keyBuf := roachpb.Key(strings.Repeat("a", writes+1)) + makeKey := func(i int) roachpb.Key { return keyBuf[:i+1] } + makeSeq := func(i int) int32 { return int32(i) + 1 } + + txn := makeTxnProto() + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + for i := 0; i < writes; i++ { + key := makeKey(i) + if i == 0 { + ba.Add(&roachpb.BeginTransactionRequest{RequestHeader: roachpb.RequestHeader{Key: key}}) + } + putArgs := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: key}} + putArgs.Sequence = makeSeq(i) + ba.Add(&putArgs) + } + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, writes+1, len(ba.Requests)) + require.True(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.BeginTransactionRequest{}, ba.Requests[0].GetInner()) + for i := 0; i < writes; i++ { + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[i+1].GetInner()) + } + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, writes, tp.outstandingWritesLen()) + + // Query every other write. + ba.Requests = nil + for i := 0; i < writes; i++ { + if i%2 == 0 { + key := makeKey(i) + ba.Add(&roachpb.GetRequest{RequestHeader: roachpb.RequestHeader{Key: key}}) + } + } + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, writes, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + for i := 0; i < writes; i++ { + if i%2 == 0 { + key := makeKey(i) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[i].GetInner()) + require.IsType(t, &roachpb.GetRequest{}, ba.Requests[i+1].GetInner()) + + qiReq := ba.Requests[i].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, key, qiReq.Key) + require.Equal(t, txn.ID, qiReq.Txn.ID) + require.Equal(t, makeSeq(i), qiReq.Txn.Sequence) + + getReq := ba.Requests[i+1].GetInner().(*roachpb.GetRequest) + require.Equal(t, key, getReq.Key) + } + } + + br = ba.CreateReply() + br.Txn = ba.Txn + for i := 0; i < writes; i++ { + if i%2 == 0 { + br.Responses[i].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + } + } + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, writes/2, tp.outstandingWritesLen()) + + // Make sure the correct writes are still outstanding. + expIdx := 1 + tp.outstandingWrites.Ascend(func(i btree.Item) bool { + w := i.(*outstandingWrite) + require.Equal(t, makeKey(expIdx), w.Key) + require.Equal(t, makeSeq(expIdx), w.Sequence) + expIdx += 2 + return true + }) +} + +// TestTxnPipelinerTransactionAbort tests that a txnPipeliner allows an aborting +// EndTransactionRequest to proceed without attempting to prove all outstanding +// writes. +func TestTxnPipelinerTransactionAbort(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA := roachpb.Key("a") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.BeginTransactionRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + putArgs := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}} + putArgs.Sequence = 2 + ba.Add(&putArgs) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.True(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.BeginTransactionRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 1, tp.outstandingWritesLen()) + + // Send an EndTransaction request with commit=false. Should NOT attempt + // to prove all outstanding writes because its attempting to abort the + // txn anyway. Should NOT use async consensus. + // + // We'll unrealistically return a PENDING transaction, which won't allow + // the txnPipeliner to clean up. + ba.Requests = nil + ba.Add(&roachpb.EndTransactionRequest{Commit: false}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 1, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.EndTransactionRequest{}, ba.Requests[0].GetInner()) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Txn.Status = roachpb.PENDING // keep at PENDING + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 1, tp.outstandingWritesLen()) // nothing proven + + // Send EndTransaction request with commit=false again. Same deal. This + // time, return ABORTED transaction. This will allow the txnPipeliner to + // remove all outstanding writes because they are now uncommittable. + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 1, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.EndTransactionRequest{}, ba.Requests[0].GetInner()) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Txn.Status = roachpb.ABORTED + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerEpochIncrement tests that a txnPipeliner's outstanding write +// set is reset on an epoch increment. +func TestTxnPipelinerEpochIncrement(t *testing.T) { + defer leaktest.AfterTest(t)() + tp, _ := makeMockTxnPipeliner() + + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("b"), 10) + tp.maybeInsertOutstandingWriteLocked(roachpb.Key("d"), 11) + require.Equal(t, 2, tp.outstandingWritesLen()) + + tp.epochBumpedLocked() + require.Equal(t, 0, tp.outstandingWritesLen()) +} + +// TestTxnPipelinerIntentMissingError tests that a txnPipeliner transforms an +// IntentMissingError into a TransactionRetryError. It also ensures that it +// fixes the errors index. +func TestTxnPipelinerIntentMissingError(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + txn := makeTxnProto() + keyA, keyB := roachpb.Key("a"), roachpb.Key("b") + keyC, keyD := roachpb.Key("c"), roachpb.Key("d") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + ba.Add(&roachpb.DeleteRangeRequest{RequestHeader: roachpb.RequestHeader{Key: keyB, EndKey: keyD}}) + ba.Add(&roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyD}}) + + // Insert outstanding writes into the outstanding write set so that each + // request will need to chain on with a QueryIntent. + tp.maybeInsertOutstandingWriteLocked(keyA, 1) + tp.maybeInsertOutstandingWriteLocked(keyB, 2) + tp.maybeInsertOutstandingWriteLocked(keyC, 3) + tp.maybeInsertOutstandingWriteLocked(keyD, 4) + + for errIdx, resErrIdx := range map[int32]int32{ + 0: 0, // intent on key "a" missing + 2: 1, // intent on key "b" missing + 3: 1, // intent on key "c" missing + 5: 2, // intent on key "d" missing + } { + t.Run(fmt.Sprintf("errIdx=%d", errIdx), func(t *testing.T) { + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 7, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[2].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[3].GetInner()) + require.IsType(t, &roachpb.DeleteRangeRequest{}, ba.Requests[4].GetInner()) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[5].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[6].GetInner()) + + err := roachpb.NewIntentMissingError(nil) + pErr := roachpb.NewErrorWithTxn(err, &txn) + pErr.SetErrorIndex(errIdx) + return nil, pErr + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.Nil(t, br) + require.NotNil(t, pErr) + require.Equal(t, &txn, pErr.GetTxn()) + require.Equal(t, resErrIdx, pErr.Index.Index) + require.IsType(t, &roachpb.TransactionRetryError{}, pErr.GetDetail()) + require.Equal(t, roachpb.RETRY_ASYNC_WRITE_FAILURE, pErr.GetDetail().(*roachpb.TransactionRetryError).Reason) + }) + } +} + +// TestTxnPipelinerEnableDisableMixTxn tests that the txnPipeliner behaves +// correctly if pipelining is enabled or disabled midway through a transaction. +func TestTxnPipelinerEnableDisableMixTxn(t *testing.T) { + defer leaktest.AfterTest(t)() + ctx := context.Background() + tp, mockSender := makeMockTxnPipeliner() + + // Start with pipelining disabled. Should NOT use async consensus. + pipelinedWritesEnabled.Override(&tp.st.SV, false) + + txn := makeTxnProto() + keyA, keyC := roachpb.Key("a"), roachpb.Key("c") + + var ba roachpb.BatchRequest + ba.Header = roachpb.Header{Txn: &txn} + ba.Add(&roachpb.BeginTransactionRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}}) + putArgs := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}} + putArgs.Sequence = 1 + ba.Add(&putArgs) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.BeginTransactionRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + br := ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr := tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) + + // Enable pipelining. Should use async consensus. + pipelinedWritesEnabled.Override(&tp.st.SV, true) + + ba.Requests = nil + putArgs2 := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}} + putArgs2.Sequence = 2 + ba.Add(&putArgs2) + putArgs3 := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyC}} + putArgs3.Sequence = 3 + ba.Add(&putArgs3) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.True(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + br = ba.CreateReply() + br.Txn = ba.Txn + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 2, tp.outstandingWritesLen()) + + // Disable pipelining again. Should NOT use async consensus but should still + // make sure to chain on to any overlapping outstanding writes. + pipelinedWritesEnabled.Override(&tp.st.SV, false) + + ba.Requests = nil + putArgs4 := roachpb.PutRequest{RequestHeader: roachpb.RequestHeader{Key: keyA}} + putArgs4.Sequence = 4 + ba.Add(&putArgs4) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.PutRequest{}, ba.Requests[1].GetInner()) + + qiReq := ba.Requests[0].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, keyA, qiReq.Key) + require.Equal(t, int32(2), qiReq.Txn.Sequence) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Responses[0].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 1, tp.outstandingWritesLen()) + + // Commit the txn. Again with pipeling disabled. Again, outstanding writes + // should be proven first. + ba.Requests = nil + ba.Add(&roachpb.EndTransactionRequest{Commit: true}) + + mockSender.MockSend(func(ba roachpb.BatchRequest) (*roachpb.BatchResponse, *roachpb.Error) { + require.Equal(t, 2, len(ba.Requests)) + require.False(t, ba.AsyncConsensus) + require.IsType(t, &roachpb.QueryIntentRequest{}, ba.Requests[0].GetInner()) + require.IsType(t, &roachpb.EndTransactionRequest{}, ba.Requests[1].GetInner()) + + qiReq := ba.Requests[0].GetInner().(*roachpb.QueryIntentRequest) + require.Equal(t, keyC, qiReq.Key) + require.Equal(t, int32(3), qiReq.Txn.Sequence) + + br = ba.CreateReply() + br.Txn = ba.Txn + br.Txn.Status = roachpb.COMMITTED + br.Responses[0].GetInner().(*roachpb.QueryIntentResponse).FoundIntent = true + return br, nil + }) + + br, pErr = tp.SendLocked(ctx, ba) + require.NotNil(t, br) + require.Nil(t, pErr) + require.Equal(t, 0, tp.outstandingWritesLen()) +} diff --git a/pkg/roachpb/api.go b/pkg/roachpb/api.go index c5a555472d53..901037e58fea 100644 --- a/pkg/roachpb/api.go +++ b/pkg/roachpb/api.go @@ -129,6 +129,12 @@ func IsReadOnly(args Request) bool { return (flags&isRead) != 0 && (flags&isWrite) == 0 } +// IsTransactional returns true if the request may be part of a +// transaction. +func IsTransactional(args Request) bool { + return (args.flags() & isTxn) != 0 +} + // IsTransactionWrite returns true if the request produces write // intents when used within a transaction. func IsTransactionWrite(args Request) bool { diff --git a/pkg/roachpb/api.pb.go b/pkg/roachpb/api.pb.go index 5a1d6b033a77..c388732fb468 100644 --- a/pkg/roachpb/api.pb.go +++ b/pkg/roachpb/api.pb.go @@ -119,6 +119,7 @@ ObservedTimestamp Transaction Intent + SequencedWrite Lease AbortSpanEntry TxnCoordMeta diff --git a/pkg/roachpb/data.pb.go b/pkg/roachpb/data.pb.go index cac0e4974104..653cf4dcdb63 100644 --- a/pkg/roachpb/data.pb.go +++ b/pkg/roachpb/data.pb.go @@ -488,6 +488,23 @@ func (m *Intent) String() string { return proto.CompactTextString(m) func (*Intent) ProtoMessage() {} func (*Intent) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{11} } +// A SequencedWrite is a point write to a key with a certain sequence number. +// +// TODO(nvanbenschoten/tschottdorf): This message type can be used as the +// PromisedWrites repeated field in EndTransaction in the parallel commits +// proposal (#24194). +type SequencedWrite struct { + // The key that the write was made at. + Key Key `protobuf:"bytes,1,opt,name=key,proto3,casttype=Key" json:"key,omitempty"` + // The sequence number of the request that created the write. + Sequence int32 `protobuf:"varint,2,opt,name=sequence,proto3" json:"sequence,omitempty"` +} + +func (m *SequencedWrite) Reset() { *m = SequencedWrite{} } +func (m *SequencedWrite) String() string { return proto.CompactTextString(m) } +func (*SequencedWrite) ProtoMessage() {} +func (*SequencedWrite) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{12} } + // Lease contains information about range leases including the // expiration and lease holder. type Lease struct { @@ -526,7 +543,7 @@ type Lease struct { func (m *Lease) Reset() { *m = Lease{} } func (*Lease) ProtoMessage() {} -func (*Lease) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{12} } +func (*Lease) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{13} } // AbortSpanEntry contains information about a transaction which has // been aborted. It's written to a range's AbortSpan if the range @@ -547,7 +564,7 @@ type AbortSpanEntry struct { func (m *AbortSpanEntry) Reset() { *m = AbortSpanEntry{} } func (m *AbortSpanEntry) String() string { return proto.CompactTextString(m) } func (*AbortSpanEntry) ProtoMessage() {} -func (*AbortSpanEntry) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{13} } +func (*AbortSpanEntry) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{14} } // TxnCoordMeta is metadata held by a transaction coordinator. This // message is defined here because it is used in several layers of the @@ -587,12 +604,17 @@ type TxnCoordMeta struct { // zero value. // TODO(nvanbenschoten): Can be removed in 2.2. DeprecatedRefreshValid bool `protobuf:"varint,6,opt,name=deprecated_refresh_valid,json=deprecatedRefreshValid,proto3" json:"deprecated_refresh_valid,omitempty"` + // outstanding_writes stores all writes that are outstanding and have + // not yet been resolved. Any client wishing to send a request that + // overlaps with them must chain on to their success using a QueryIntent + // request. + OutstandingWrites []SequencedWrite `protobuf:"bytes,8,rep,name=outstanding_writes,json=outstandingWrites" json:"outstanding_writes"` } func (m *TxnCoordMeta) Reset() { *m = TxnCoordMeta{} } func (m *TxnCoordMeta) String() string { return proto.CompactTextString(m) } func (*TxnCoordMeta) ProtoMessage() {} -func (*TxnCoordMeta) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{14} } +func (*TxnCoordMeta) Descriptor() ([]byte, []int) { return fileDescriptorData, []int{15} } func init() { proto.RegisterType((*Span)(nil), "cockroach.roachpb.Span") @@ -607,6 +629,7 @@ func init() { proto.RegisterType((*ObservedTimestamp)(nil), "cockroach.roachpb.ObservedTimestamp") proto.RegisterType((*Transaction)(nil), "cockroach.roachpb.Transaction") proto.RegisterType((*Intent)(nil), "cockroach.roachpb.Intent") + proto.RegisterType((*SequencedWrite)(nil), "cockroach.roachpb.SequencedWrite") proto.RegisterType((*Lease)(nil), "cockroach.roachpb.Lease") proto.RegisterType((*AbortSpanEntry)(nil), "cockroach.roachpb.AbortSpanEntry") proto.RegisterType((*TxnCoordMeta)(nil), "cockroach.roachpb.TxnCoordMeta") @@ -1544,6 +1567,35 @@ func (m *Intent) MarshalTo(dAtA []byte) (int, error) { return i, nil } +func (m *SequencedWrite) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *SequencedWrite) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Key) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintData(dAtA, i, uint64(len(m.Key))) + i += copy(dAtA[i:], m.Key) + } + if m.Sequence != 0 { + dAtA[i] = 0x10 + i++ + i = encodeVarintData(dAtA, i, uint64(m.Sequence)) + } + return i, nil +} + func (m *Lease) Marshal() (dAtA []byte, err error) { size := m.Size() dAtA = make([]byte, size) @@ -1739,6 +1791,18 @@ func (m *TxnCoordMeta) MarshalTo(dAtA []byte) (int, error) { } i++ } + if len(m.OutstandingWrites) > 0 { + for _, msg := range m.OutstandingWrites { + dAtA[i] = 0x42 + i++ + i = encodeVarintData(dAtA, i, uint64(msg.Size())) + n, err := msg.MarshalTo(dAtA[i:]) + if err != nil { + return 0, err + } + i += n + } + } return i, nil } @@ -2144,6 +2208,19 @@ func (m *Intent) Size() (n int) { return n } +func (m *SequencedWrite) Size() (n int) { + var l int + _ = l + l = len(m.Key) + if l > 0 { + n += 1 + l + sovData(uint64(l)) + } + if m.Sequence != 0 { + n += 1 + sovData(uint64(m.Sequence)) + } + return n +} + func (m *Lease) Size() (n int) { var l int _ = l @@ -2219,6 +2296,12 @@ func (m *TxnCoordMeta) Size() (n int) { if m.RefreshInvalid { n += 2 } + if len(m.OutstandingWrites) > 0 { + for _, e := range m.OutstandingWrites { + l = e.Size() + n += 1 + l + sovData(uint64(l)) + } + } return n } @@ -4020,6 +4103,106 @@ func (m *Intent) Unmarshal(dAtA []byte) error { } return nil } +func (m *SequencedWrite) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowData + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: SequencedWrite: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: SequencedWrite: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Key", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowData + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthData + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Key = append(m.Key[:0], dAtA[iNdEx:postIndex]...) + if m.Key == nil { + m.Key = []byte{} + } + iNdEx = postIndex + case 2: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Sequence", wireType) + } + m.Sequence = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowData + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.Sequence |= (int32(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + default: + iNdEx = preIndex + skippy, err := skipData(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthData + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} func (m *Lease) Unmarshal(dAtA []byte) error { l := len(dAtA) iNdEx := 0 @@ -4608,6 +4791,37 @@ func (m *TxnCoordMeta) Unmarshal(dAtA []byte) error { } } m.RefreshInvalid = bool(v != 0) + case 8: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field OutstandingWrites", wireType) + } + var msglen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowData + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + msglen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if msglen < 0 { + return ErrInvalidLengthData + } + postIndex := iNdEx + msglen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.OutstandingWrites = append(m.OutstandingWrites, SequencedWrite{}) + if err := m.OutstandingWrites[len(m.OutstandingWrites)-1].Unmarshal(dAtA[iNdEx:postIndex]); err != nil { + return err + } + iNdEx = postIndex default: iNdEx = preIndex skippy, err := skipData(dAtA[iNdEx:]) @@ -4737,117 +4951,120 @@ var ( func init() { proto.RegisterFile("roachpb/data.proto", fileDescriptorData) } var fileDescriptorData = []byte{ - // 1788 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xcc, 0x58, 0xcd, 0x6f, 0xdb, 0xc8, - 0x15, 0x37, 0x45, 0xc9, 0xa2, 0x9e, 0x25, 0x9b, 0x9e, 0xc4, 0x89, 0xea, 0xc5, 0x4a, 0xa9, 0xb6, - 0x68, 0x83, 0xa0, 0x2b, 0xa3, 0x49, 0x3f, 0x8d, 0xb6, 0x80, 0x65, 0x69, 0x37, 0x4c, 0x6c, 0xd9, - 0xa0, 0xe8, 0x04, 0xcd, 0x62, 0xc1, 0x8e, 0xc9, 0x89, 0x44, 0x44, 0x22, 0xd9, 0x99, 0x91, 0x63, - 0xf5, 0x2f, 0xd8, 0x5b, 0xb7, 0x40, 0x0f, 0x3d, 0x06, 0xe8, 0xa9, 0xfd, 0x0b, 0x7a, 0xef, 0xa1, - 0xb9, 0x14, 0xd8, 0x5b, 0x8b, 0x1e, 0x84, 0xd6, 0xbd, 0x14, 0xe8, 0xad, 0xc7, 0x00, 0x0b, 0x14, - 0x33, 0x1c, 0x4a, 0x4c, 0xad, 0x04, 0x72, 0xb7, 0x87, 0xbd, 0xd8, 0xc3, 0xf7, 0xf1, 0x9b, 0xf7, - 0xfd, 0x48, 0x01, 0xa2, 0x11, 0xf6, 0x06, 0xf1, 0xe9, 0x8e, 0x8f, 0x39, 0x6e, 0xc6, 0x34, 0xe2, - 0x11, 0xda, 0xf4, 0x22, 0xef, 0x99, 0xa4, 0x37, 0x15, 0x77, 0xfb, 0x46, 0x2a, 0x36, 0x22, 0x1c, - 0xcf, 0x45, 0xb7, 0xdf, 0x63, 0x3c, 0xa2, 0xb8, 0x4f, 0x76, 0x48, 0xd8, 0x0f, 0xc2, 0xf4, 0x9f, - 0x90, 0x3b, 0xf3, 0xbc, 0x7b, 0x4a, 0xa8, 0x3a, 0xe6, 0xc1, 0x70, 0x67, 0x30, 0xf4, 0x76, 0x78, - 0x30, 0x22, 0x8c, 0xe3, 0x51, 0xac, 0x38, 0xd7, 0xfb, 0x51, 0x3f, 0x92, 0xc7, 0x1d, 0x71, 0x4a, - 0xa8, 0x8d, 0x8f, 0x21, 0xdf, 0x8b, 0x71, 0x88, 0xbe, 0x02, 0xfa, 0x33, 0x32, 0xa9, 0xea, 0xb7, - 0xb4, 0xdb, 0xe5, 0x56, 0xf1, 0xd5, 0xb4, 0xae, 0x3f, 0x24, 0x13, 0x5b, 0xd0, 0xd0, 0x2d, 0x28, - 0x92, 0xd0, 0x77, 0x05, 0x3b, 0xff, 0x3a, 0x7b, 0x95, 0x84, 0xfe, 0x43, 0x32, 0xd9, 0x2d, 0xff, - 0xfa, 0x45, 0x7d, 0xe5, 0xf7, 0x2f, 0xea, 0xda, 0x3f, 0x5f, 0xd4, 0xb5, 0x07, 0x79, 0x43, 0x33, - 0x73, 0x0f, 0xf2, 0x46, 0xce, 0xd4, 0x1b, 0x23, 0x28, 0x3c, 0xc2, 0xc3, 0x31, 0x41, 0xef, 0x40, - 0x89, 0xe2, 0xe7, 0xee, 0xe9, 0x84, 0x13, 0x56, 0xd5, 0x04, 0x8c, 0x6d, 0x50, 0xfc, 0xbc, 0x25, - 0x9e, 0xd1, 0x1e, 0x94, 0x66, 0xd6, 0x56, 0x73, 0xb7, 0xb4, 0xdb, 0x6b, 0x77, 0xdf, 0x6d, 0xce, - 0x03, 0x23, 0x5c, 0x6a, 0x0e, 0x86, 0x5e, 0xd3, 0x49, 0x85, 0x5a, 0xf9, 0x97, 0xd3, 0xfa, 0x8a, - 0x3d, 0xd7, 0xda, 0xcd, 0x8b, 0xab, 0x1b, 0x1f, 0x81, 0xf1, 0x90, 0x4c, 0x92, 0x1b, 0x95, 0x47, - 0xda, 0x02, 0x8f, 0xbe, 0x0d, 0x85, 0x33, 0x21, 0xa3, 0xee, 0xaa, 0x36, 0x2f, 0x25, 0xa1, 0x29, - 0x31, 0xd4, 0x35, 0x89, 0x70, 0xe3, 0xcf, 0x1a, 0x40, 0x8f, 0x47, 0x94, 0x58, 0x3e, 0x09, 0x39, - 0xea, 0x03, 0x78, 0xc3, 0x31, 0xe3, 0x84, 0xba, 0x81, 0xaf, 0xae, 0xb9, 0x2f, 0xe4, 0xff, 0x3a, - 0xad, 0xdf, 0xeb, 0x07, 0x7c, 0x30, 0x3e, 0x6d, 0x7a, 0xd1, 0x68, 0x67, 0x86, 0xed, 0x9f, 0xce, - 0xcf, 0x3b, 0xf1, 0xb3, 0xfe, 0x8e, 0x4c, 0xd5, 0x78, 0x1c, 0xf8, 0xcd, 0x93, 0x13, 0xab, 0x7d, - 0x31, 0xad, 0x97, 0xf6, 0x13, 0x40, 0xab, 0x6d, 0x97, 0x14, 0xb6, 0xe5, 0xa3, 0xf7, 0xa1, 0x18, - 0x46, 0x3e, 0x11, 0xb7, 0x08, 0x7b, 0x0b, 0xad, 0xeb, 0x17, 0xd3, 0xfa, 0x6a, 0x37, 0xf2, 0x89, - 0xd5, 0x7e, 0x35, 0x3b, 0xd9, 0xab, 0x42, 0xc8, 0xf2, 0xd1, 0xb7, 0xc0, 0x10, 0x85, 0x22, 0xe5, - 0x75, 0x29, 0x7f, 0xe3, 0x62, 0x5a, 0x2f, 0x26, 0x96, 0x0b, 0x85, 0xf4, 0x68, 0x17, 0x59, 0xe2, - 0x4d, 0xe3, 0x77, 0x1a, 0x94, 0x7b, 0xf1, 0x30, 0xe0, 0x0e, 0x0d, 0xfa, 0x7d, 0x42, 0x51, 0x07, - 0x4a, 0x43, 0xf2, 0x94, 0xbb, 0x3e, 0x61, 0x9e, 0x74, 0x6d, 0xed, 0x6e, 0x63, 0x41, 0x90, 0x6c, - 0x1c, 0xf6, 0x49, 0x9b, 0x30, 0x8f, 0x06, 0x31, 0x8f, 0xa8, 0x0a, 0x97, 0x21, 0x54, 0x05, 0x15, - 0x7d, 0x08, 0x40, 0x83, 0xfe, 0x40, 0xe1, 0xe4, 0xae, 0x88, 0x53, 0x92, 0xba, 0x82, 0x9c, 0x64, - 0xf7, 0x41, 0xde, 0xd0, 0xcd, 0x7c, 0xe3, 0x0f, 0x1a, 0x94, 0x0f, 0x09, 0xed, 0x93, 0x2f, 0xa9, - 0xb1, 0xe8, 0xdd, 0x19, 0x10, 0xe6, 0x38, 0xe9, 0xa8, 0x94, 0x8d, 0x39, 0x56, 0x95, 0xfa, 0xa7, - 0x1c, 0x6c, 0xed, 0x0f, 0x04, 0x94, 0x4d, 0xe2, 0x61, 0xe0, 0x61, 0x36, 0x77, 0x67, 0xcd, 0x93, - 0x0c, 0x97, 0x4f, 0x62, 0x22, 0x1d, 0x5a, 0xbf, 0xfb, 0xb5, 0x45, 0x86, 0x24, 0x8a, 0x09, 0x8a, - 0x33, 0x89, 0x89, 0x0d, 0xde, 0xec, 0x8c, 0xda, 0x50, 0xa4, 0x89, 0x80, 0xf2, 0xe5, 0x2d, 0x10, - 0x97, 0xbc, 0x49, 0x55, 0xd1, 0x09, 0x98, 0xe3, 0xd8, 0xc7, 0x9c, 0xf8, 0xae, 0x22, 0xb1, 0xaa, - 0x7e, 0x4b, 0xbf, 0x22, 0xdc, 0x86, 0xc2, 0x48, 0x5d, 0x45, 0x1f, 0xc0, 0x46, 0x48, 0xce, 0x79, - 0x8a, 0x29, 0x4a, 0x35, 0x2f, 0x4b, 0xb5, 0x76, 0x31, 0xad, 0x57, 0xba, 0xe4, 0x9c, 0x2b, 0x51, - 0x59, 0xb0, 0xa5, 0xd9, 0x83, 0x5d, 0x09, 0x33, 0x3c, 0x7f, 0xd7, 0x10, 0x83, 0x47, 0xc6, 0xf3, - 0x53, 0x0d, 0xae, 0x1d, 0x46, 0x7e, 0xf0, 0x34, 0x20, 0xbe, 0x18, 0x68, 0x69, 0x34, 0xbf, 0x09, - 0x88, 0x4d, 0x18, 0x27, 0x23, 0xd7, 0x8b, 0xc2, 0xa7, 0x41, 0xdf, 0x65, 0x31, 0x0e, 0x65, 0x50, - 0x0d, 0xdb, 0x4c, 0x38, 0xfb, 0x92, 0x21, 0xa7, 0x60, 0x07, 0x90, 0x6c, 0xb5, 0x61, 0x70, 0x46, - 0x42, 0xc2, 0x58, 0x22, 0x9d, 0xc4, 0xef, 0xe6, 0x02, 0x87, 0x85, 0x92, 0x6d, 0x0a, 0x95, 0x03, - 0xa5, 0x21, 0x28, 0x2a, 0xc5, 0xff, 0xca, 0xc1, 0x96, 0x15, 0x72, 0x42, 0x43, 0x3c, 0xdc, 0x8f, - 0x46, 0xa3, 0x79, 0x7b, 0xb5, 0xa1, 0xc2, 0x44, 0xbb, 0xb9, 0x3c, 0x21, 0xa8, 0xaa, 0xad, 0x2f, - 0xbc, 0x61, 0xde, 0x96, 0x76, 0x99, 0x65, 0x9b, 0xb4, 0x0d, 0x95, 0x91, 0xe8, 0x83, 0x19, 0x4a, - 0xee, 0x8d, 0x28, 0xd9, 0x7e, 0xb1, 0xcb, 0xa3, 0x6c, 0xf7, 0xfc, 0x14, 0x6e, 0xaa, 0x72, 0x4b, - 0x13, 0x3c, 0xc3, 0xd3, 0x25, 0xde, 0xed, 0x05, 0x78, 0x0b, 0x2b, 0xd7, 0xde, 0xf2, 0x16, 0x16, - 0xf4, 0x13, 0xd8, 0x1a, 0xa9, 0xcc, 0xc8, 0x78, 0xce, 0xf0, 0xf3, 0x12, 0xff, 0xeb, 0x8b, 0xec, - 0xbd, 0x9c, 0x49, 0xfb, 0xda, 0xe8, 0x32, 0x71, 0xd7, 0xf8, 0x44, 0x6d, 0x9d, 0xc6, 0x2f, 0x34, - 0xd8, 0x3c, 0x3a, 0x65, 0x84, 0x9e, 0x11, 0x7f, 0xb6, 0x27, 0xb2, 0xb3, 0x53, 0x5b, 0x62, 0x76, - 0xfe, 0x1f, 0x16, 0x91, 0x91, 0xee, 0xc1, 0xc6, 0xe7, 0xab, 0xb0, 0xe6, 0x50, 0x1c, 0x32, 0xec, - 0xf1, 0x20, 0x0a, 0xd1, 0x7d, 0xc8, 0x8b, 0x8d, 0xae, 0x92, 0x7d, 0x27, 0x83, 0xab, 0x16, 0x7b, - 0x33, 0xd9, 0xe8, 0xcd, 0x74, 0xb1, 0x37, 0x9d, 0xf3, 0xf0, 0x90, 0x70, 0xdc, 0x32, 0xc4, 0x25, - 0x9f, 0x4d, 0xeb, 0x9a, 0x2d, 0x11, 0x10, 0x82, 0x7c, 0x88, 0x47, 0xc9, 0xfa, 0x2a, 0xd9, 0xf2, - 0x8c, 0x7e, 0x08, 0xab, 0x8c, 0x63, 0x3e, 0x66, 0x32, 0xac, 0x8b, 0x27, 0x46, 0xc6, 0x9a, 0x9e, - 0x94, 0xb5, 0x95, 0x0e, 0x7a, 0x00, 0xeb, 0x43, 0xcc, 0xb8, 0x3b, 0x20, 0x98, 0xf2, 0x53, 0x82, - 0x79, 0xb5, 0xb0, 0xbc, 0xf7, 0x15, 0xa1, 0x7a, 0x3f, 0xd5, 0x14, 0x58, 0x11, 0x0d, 0xfa, 0xee, - 0x3c, 0x92, 0xab, 0x57, 0xc0, 0x12, 0xaa, 0xf3, 0xfc, 0xdd, 0x87, 0xca, 0x08, 0x9f, 0x67, 0xa0, - 0x8a, 0xcb, 0x43, 0x95, 0x47, 0xf8, 0x7c, 0x8e, 0xf4, 0x11, 0x5c, 0x8b, 0x54, 0x79, 0xcc, 0xe1, - 0x58, 0xd5, 0x78, 0xe3, 0x30, 0xbb, 0x54, 0x4c, 0x0a, 0x16, 0x45, 0xff, 0xcd, 0x60, 0xa8, 0x0a, - 0xc5, 0xe7, 0x34, 0xe0, 0x41, 0xd8, 0xaf, 0x96, 0xe4, 0x68, 0x49, 0x1f, 0xd1, 0xf7, 0xa0, 0x18, - 0x84, 0x9c, 0x84, 0x9c, 0x55, 0xd7, 0xe4, 0x55, 0x6f, 0x1a, 0x23, 0xe9, 0xe4, 0x55, 0xd2, 0xa8, - 0x01, 0x15, 0x81, 0x41, 0x5c, 0x1e, 0x45, 0x6e, 0x34, 0xf4, 0xab, 0x65, 0x09, 0xbc, 0x26, 0x89, - 0x4e, 0x14, 0x1d, 0x0d, 0x7d, 0x21, 0x43, 0x09, 0xa7, 0x13, 0x37, 0x0a, 0xdd, 0x78, 0xcc, 0x06, - 0xd5, 0x4a, 0x22, 0x23, 0x89, 0x47, 0xe1, 0xf1, 0x98, 0x0d, 0xd0, 0x09, 0x5c, 0x27, 0x71, 0xe4, - 0x0d, 0xdc, 0x9f, 0x13, 0x1a, 0x65, 0x02, 0xb9, 0xbe, 0x7c, 0x20, 0x91, 0x04, 0x78, 0x42, 0x68, - 0x34, 0x0f, 0xa7, 0x03, 0xd7, 0x28, 0x79, 0x4a, 0x09, 0x1b, 0x64, 0xe3, 0x59, 0xdd, 0xb8, 0x02, - 0xea, 0x4c, 0x7f, 0x8e, 0xfa, 0x23, 0x78, 0xe7, 0xf5, 0xd2, 0x71, 0x9f, 0x63, 0xe6, 0xa6, 0x01, - 0xaf, 0x9a, 0xd2, 0xbd, 0xea, 0x6b, 0x25, 0xf2, 0x18, 0xb3, 0x34, 0x53, 0xaf, 0xbf, 0x87, 0x36, - 0xfe, 0xa8, 0xc1, 0xaa, 0x25, 0xa3, 0x89, 0xbe, 0x03, 0xf9, 0xd9, 0xdc, 0x7f, 0x4b, 0x0a, 0x32, - 0x7d, 0x26, 0xc4, 0x51, 0x0b, 0x74, 0x7e, 0x9e, 0xce, 0xff, 0xab, 0x34, 0x6c, 0xe2, 0xa1, 0x50, - 0xce, 0xf4, 0xa5, 0x7e, 0xf5, 0xbe, 0x54, 0x9b, 0xe4, 0x73, 0x1d, 0x0a, 0x07, 0x04, 0x33, 0x82, - 0x7e, 0x00, 0x05, 0xc6, 0x31, 0xe5, 0xca, 0x93, 0xa5, 0x02, 0x9d, 0x68, 0xa0, 0x8f, 0x01, 0xc8, - 0x79, 0x1c, 0x50, 0x2c, 0xae, 0x59, 0x6e, 0xb8, 0xd5, 0xfe, 0x3d, 0xad, 0x6f, 0x67, 0xde, 0x5f, - 0x77, 0x1b, 0x14, 0x87, 0x7e, 0x38, 0x1e, 0x0e, 0xf1, 0xe9, 0x90, 0x34, 0xec, 0x0c, 0x60, 0xf6, - 0x7d, 0x43, 0xff, 0xdf, 0xdf, 0x37, 0xc6, 0x70, 0xd3, 0x27, 0x31, 0x25, 0x9e, 0x7c, 0xe5, 0x90, - 0x86, 0x8b, 0xbf, 0x2c, 0x60, 0x6a, 0x5b, 0x7c, 0x41, 0x8b, 0xb7, 0xe6, 0xe8, 0x3d, 0x01, 0xde, - 0x93, 0xd8, 0xa8, 0x0b, 0x6b, 0x31, 0x8d, 0xe2, 0x88, 0x89, 0x62, 0x66, 0xcb, 0xcd, 0xbe, 0xf5, - 0x8b, 0x69, 0x1d, 0x8e, 0x95, 0x96, 0xd3, 0xb3, 0x21, 0x45, 0x70, 0x18, 0xba, 0x0e, 0x05, 0xd9, - 0x33, 0x72, 0xf2, 0xe9, 0x76, 0xf2, 0x80, 0xde, 0x07, 0x83, 0x91, 0x9f, 0x8d, 0x49, 0xe8, 0x11, - 0x39, 0xc7, 0xf4, 0xd6, 0xe6, 0xab, 0x69, 0xbd, 0x22, 0x33, 0xdb, 0x53, 0x0c, 0x7b, 0x26, 0x92, - 0xbc, 0xdc, 0x88, 0x6a, 0x6e, 0xfc, 0x4a, 0x83, 0xf5, 0xbd, 0xd3, 0x88, 0x72, 0x51, 0xa5, 0x9d, - 0x90, 0xd3, 0xc9, 0xdb, 0xbe, 0x6e, 0xbe, 0xf8, 0x12, 0x43, 0xdb, 0x60, 0xc4, 0x34, 0x88, 0x68, - 0xc0, 0x93, 0x4f, 0xc2, 0x82, 0x3d, 0x7b, 0xce, 0x2c, 0xb8, 0x5f, 0xea, 0x50, 0x76, 0xce, 0xc3, - 0xfd, 0x28, 0xa2, 0xbe, 0x28, 0x7b, 0xf4, 0xdd, 0xa4, 0x5f, 0x92, 0xda, 0xac, 0xbd, 0xbd, 0xd0, - 0xb3, 0x3d, 0x92, 0x19, 0x92, 0xb9, 0x2b, 0x0d, 0xc9, 0xf7, 0xa0, 0xe2, 0x45, 0xa3, 0x11, 0x0e, - 0x7d, 0xd7, 0x8b, 0xc6, 0x21, 0x57, 0xc6, 0x96, 0x15, 0x71, 0x5f, 0xd0, 0x50, 0x4b, 0x4c, 0x49, - 0x39, 0x6a, 0x5c, 0x4a, 0xb0, 0x2f, 0x2a, 0x69, 0x89, 0x3b, 0xca, 0x4a, 0xc7, 0x16, 0x2a, 0xa8, - 0x0d, 0xeb, 0x29, 0x86, 0x1c, 0xc0, 0xa2, 0x46, 0x96, 0x00, 0x49, 0x2f, 0x7e, 0x2c, 0x75, 0xd0, - 0xf7, 0xa1, 0x9a, 0xa9, 0xee, 0x14, 0xf0, 0x0c, 0x0f, 0x03, 0x5f, 0x56, 0x8a, 0x61, 0xdf, 0x98, - 0xf3, 0xed, 0x84, 0xfd, 0x48, 0x70, 0xd1, 0x37, 0x60, 0x23, 0x15, 0x0f, 0xc2, 0x44, 0xa1, 0x28, - 0x15, 0x52, 0xb3, 0xac, 0x84, 0x7a, 0xe7, 0xb7, 0x1a, 0x94, 0xe4, 0xb7, 0xab, 0xfc, 0x08, 0x58, - 0x83, 0xe2, 0x49, 0xf7, 0x61, 0xf7, 0xe8, 0x71, 0xd7, 0x5c, 0x41, 0x45, 0xd0, 0xad, 0xae, 0x63, - 0x6a, 0xa8, 0x04, 0x85, 0x0f, 0x0e, 0x8e, 0xf6, 0x1c, 0x33, 0x27, 0x8e, 0xad, 0x9f, 0x38, 0x9d, - 0x9e, 0xa9, 0xa3, 0x6b, 0xb0, 0xd1, 0xee, 0x1c, 0x58, 0x87, 0x96, 0xd3, 0x69, 0xbb, 0x09, 0xd1, - 0x40, 0x06, 0xe4, 0x1d, 0xeb, 0xb0, 0x63, 0xe6, 0x05, 0x54, 0xbb, 0xb3, 0x6f, 0x1d, 0xee, 0x1d, - 0x98, 0x05, 0xb4, 0x05, 0x9b, 0x73, 0xd9, 0x94, 0x5c, 0x42, 0x65, 0x30, 0xda, 0x27, 0xf6, 0x9e, - 0x63, 0x1d, 0x75, 0xcd, 0x55, 0x81, 0xed, 0x9c, 0x1c, 0x1f, 0x74, 0x4c, 0x40, 0xeb, 0x00, 0x02, - 0xa6, 0xd7, 0xb1, 0xad, 0x4e, 0xcf, 0xf4, 0x1b, 0x79, 0xa3, 0x68, 0x16, 0xef, 0xfc, 0x18, 0x36, - 0x2f, 0x7d, 0xc3, 0xa0, 0x0d, 0x58, 0xdb, 0x6b, 0xb7, 0x5d, 0xbb, 0x73, 0x7c, 0x60, 0xed, 0xef, - 0x99, 0x2b, 0x08, 0xc1, 0xba, 0xdd, 0x39, 0x3c, 0x7a, 0xd4, 0x99, 0xd1, 0xb4, 0xed, 0xfc, 0x27, - 0xbf, 0xa9, 0xad, 0xdc, 0x69, 0xc1, 0xe6, 0xa5, 0xc9, 0x29, 0xec, 0x3c, 0xee, 0x74, 0xdb, 0x56, - 0xf7, 0x43, 0x73, 0x05, 0x55, 0xa0, 0xb4, 0x7f, 0x74, 0x78, 0x68, 0x39, 0x4e, 0xa7, 0x6d, 0x6a, - 0x82, 0xb7, 0xd7, 0x3a, 0xb2, 0xc5, 0x43, 0x2e, 0xc1, 0x68, 0x7d, 0xf5, 0xe5, 0xdf, 0x6b, 0x2b, - 0x2f, 0x2f, 0x6a, 0xda, 0x67, 0x17, 0x35, 0xed, 0x2f, 0x17, 0x35, 0xed, 0x6f, 0x17, 0x35, 0xed, - 0xd3, 0x7f, 0xd4, 0x56, 0x9e, 0x14, 0x55, 0x3e, 0x4f, 0x57, 0xe5, 0x2f, 0x25, 0xf7, 0xfe, 0x13, - 0x00, 0x00, 0xff, 0xff, 0xff, 0x37, 0xfc, 0x57, 0xbf, 0x11, 0x00, 0x00, + // 1836 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xcc, 0x58, 0xcb, 0x6f, 0xdb, 0xc8, + 0x19, 0x37, 0xf5, 0xb0, 0xa8, 0xcf, 0x92, 0x4d, 0x8f, 0xe3, 0x44, 0xf5, 0x62, 0xa5, 0xac, 0xb6, + 0x68, 0x83, 0xa0, 0x2b, 0xa3, 0x49, 0x9f, 0x46, 0x5b, 0xc0, 0xb2, 0xb4, 0x09, 0x13, 0x5b, 0x36, + 0x28, 0x3a, 0x41, 0xb3, 0x58, 0xb0, 0x63, 0x72, 0x22, 0x11, 0x91, 0x48, 0x76, 0x66, 0xe4, 0x58, + 0xfd, 0x0b, 0xf6, 0xd6, 0x3d, 0xf4, 0xd0, 0x63, 0x80, 0x9e, 0xda, 0xbf, 0xa0, 0xf7, 0x1e, 0x9a, + 0x4b, 0x81, 0xbd, 0xb5, 0xe8, 0x41, 0x68, 0xdd, 0x4b, 0x8b, 0xde, 0x7a, 0x0c, 0xb0, 0x40, 0x31, + 0xc3, 0xa1, 0x44, 0xd7, 0x8a, 0x21, 0x37, 0x3d, 0xf4, 0x92, 0x0c, 0xbf, 0xc7, 0x6f, 0xbe, 0xf7, + 0x37, 0x32, 0x20, 0x1a, 0x62, 0xb7, 0x1f, 0x9d, 0x6c, 0x7b, 0x98, 0xe3, 0x46, 0x44, 0x43, 0x1e, + 0xa2, 0x75, 0x37, 0x74, 0x5f, 0x48, 0x7a, 0x43, 0x71, 0xb7, 0x6e, 0x26, 0x62, 0x43, 0xc2, 0xf1, + 0x4c, 0x74, 0xeb, 0x43, 0xc6, 0x43, 0x8a, 0x7b, 0x64, 0x9b, 0x04, 0x3d, 0x3f, 0x48, 0xfe, 0x13, + 0x72, 0xa7, 0xae, 0x7b, 0x5f, 0x09, 0x55, 0x46, 0xdc, 0x1f, 0x6c, 0xf7, 0x07, 0xee, 0x36, 0xf7, + 0x87, 0x84, 0x71, 0x3c, 0x8c, 0x14, 0xe7, 0x46, 0x2f, 0xec, 0x85, 0xf2, 0xb8, 0x2d, 0x4e, 0x31, + 0xb5, 0xfe, 0x29, 0xe4, 0xba, 0x11, 0x0e, 0xd0, 0x57, 0x20, 0xfb, 0x82, 0x8c, 0x2b, 0xd9, 0xdb, + 0xda, 0x9d, 0x52, 0xb3, 0xf0, 0x66, 0x52, 0xcb, 0x3e, 0x26, 0x63, 0x4b, 0xd0, 0xd0, 0x6d, 0x28, + 0x90, 0xc0, 0x73, 0x04, 0x3b, 0x77, 0x91, 0xbd, 0x4c, 0x02, 0xef, 0x31, 0x19, 0xef, 0x94, 0x7e, + 0xf9, 0xaa, 0xb6, 0xf4, 0xdb, 0x57, 0x35, 0xed, 0xef, 0xaf, 0x6a, 0xda, 0xa3, 0x9c, 0xae, 0x19, + 0x99, 0x47, 0x39, 0x3d, 0x63, 0x64, 0xeb, 0x43, 0xc8, 0x3f, 0xc1, 0x83, 0x11, 0x41, 0xef, 0x41, + 0x91, 0xe2, 0x97, 0xce, 0xc9, 0x98, 0x13, 0x56, 0xd1, 0x04, 0x8c, 0xa5, 0x53, 0xfc, 0xb2, 0x29, + 0xbe, 0xd1, 0x2e, 0x14, 0xa7, 0xd6, 0x56, 0x32, 0xb7, 0xb5, 0x3b, 0x2b, 0xf7, 0xde, 0x6f, 0xcc, + 0x02, 0x23, 0x5c, 0x6a, 0xf4, 0x07, 0x6e, 0xc3, 0x4e, 0x84, 0x9a, 0xb9, 0xd7, 0x93, 0xda, 0x92, + 0x35, 0xd3, 0xda, 0xc9, 0x89, 0xab, 0xeb, 0x9f, 0x80, 0xfe, 0x98, 0x8c, 0xe3, 0x1b, 0x95, 0x47, + 0xda, 0x1c, 0x8f, 0xbe, 0x05, 0xf9, 0x53, 0x21, 0xa3, 0xee, 0xaa, 0x34, 0x2e, 0x25, 0xa1, 0x21, + 0x31, 0xd4, 0x35, 0xb1, 0x70, 0xfd, 0x8f, 0x1a, 0x40, 0x97, 0x87, 0x94, 0x98, 0x1e, 0x09, 0x38, + 0xea, 0x01, 0xb8, 0x83, 0x11, 0xe3, 0x84, 0x3a, 0xbe, 0xa7, 0xae, 0x79, 0x28, 0xe4, 0xff, 0x3c, + 0xa9, 0xdd, 0xef, 0xf9, 0xbc, 0x3f, 0x3a, 0x69, 0xb8, 0xe1, 0x70, 0x7b, 0x8a, 0xed, 0x9d, 0xcc, + 0xce, 0xdb, 0xd1, 0x8b, 0xde, 0xb6, 0x4c, 0xd5, 0x68, 0xe4, 0x7b, 0x8d, 0xe3, 0x63, 0xb3, 0x75, + 0x3e, 0xa9, 0x15, 0xf7, 0x62, 0x40, 0xb3, 0x65, 0x15, 0x15, 0xb6, 0xe9, 0xa1, 0x8f, 0xa0, 0x10, + 0x84, 0x1e, 0x11, 0xb7, 0x08, 0x7b, 0xf3, 0xcd, 0x1b, 0xe7, 0x93, 0xda, 0x72, 0x27, 0xf4, 0x88, + 0xd9, 0x7a, 0x33, 0x3d, 0x59, 0xcb, 0x42, 0xc8, 0xf4, 0xd0, 0x37, 0x41, 0x17, 0x85, 0x22, 0xe5, + 0xb3, 0x52, 0xfe, 0xe6, 0xf9, 0xa4, 0x56, 0x88, 0x2d, 0x17, 0x0a, 0xc9, 0xd1, 0x2a, 0xb0, 0xd8, + 0x9b, 0xfa, 0x6f, 0x34, 0x28, 0x75, 0xa3, 0x81, 0xcf, 0x6d, 0xea, 0xf7, 0x7a, 0x84, 0xa2, 0x36, + 0x14, 0x07, 0xe4, 0x39, 0x77, 0x3c, 0xc2, 0x5c, 0xe9, 0xda, 0xca, 0xbd, 0xfa, 0x9c, 0x20, 0x59, + 0x38, 0xe8, 0x91, 0x16, 0x61, 0x2e, 0xf5, 0x23, 0x1e, 0x52, 0x15, 0x2e, 0x5d, 0xa8, 0x0a, 0x2a, + 0x7a, 0x00, 0x40, 0xfd, 0x5e, 0x5f, 0xe1, 0x64, 0xae, 0x89, 0x53, 0x94, 0xba, 0x82, 0x1c, 0x67, + 0xf7, 0x51, 0x4e, 0xcf, 0x1a, 0xb9, 0xfa, 0xef, 0x34, 0x28, 0x1d, 0x10, 0xda, 0x23, 0xff, 0xa7, + 0xc6, 0xa2, 0xf7, 0xa7, 0x40, 0x98, 0xe3, 0xb8, 0xa3, 0x12, 0x36, 0xe6, 0x58, 0x55, 0xea, 0x1f, + 0x32, 0xb0, 0xb9, 0xd7, 0x17, 0x50, 0x16, 0x89, 0x06, 0xbe, 0x8b, 0xd9, 0xcc, 0x9d, 0x15, 0x57, + 0x32, 0x1c, 0x3e, 0x8e, 0x88, 0x74, 0x68, 0xf5, 0xde, 0x57, 0xe7, 0x19, 0x12, 0x2b, 0xc6, 0x28, + 0xf6, 0x38, 0x22, 0x16, 0xb8, 0xd3, 0x33, 0x6a, 0x41, 0x81, 0xc6, 0x02, 0xca, 0x97, 0x2b, 0x20, + 0x2e, 0x79, 0x93, 0xa8, 0xa2, 0x63, 0x30, 0x46, 0x91, 0x87, 0x39, 0xf1, 0x1c, 0x45, 0x62, 0x95, + 0xec, 0xed, 0xec, 0x35, 0xe1, 0xd6, 0x14, 0x46, 0xe2, 0x2a, 0xfa, 0x18, 0xd6, 0x02, 0x72, 0xc6, + 0x13, 0x4c, 0x51, 0xaa, 0x39, 0x59, 0xaa, 0xd5, 0xf3, 0x49, 0xad, 0xdc, 0x21, 0x67, 0x5c, 0x89, + 0xca, 0x82, 0x2d, 0x4e, 0x3f, 0xac, 0x72, 0x90, 0xe2, 0x79, 0x3b, 0xba, 0x18, 0x3c, 0x32, 0x9e, + 0x9f, 0x6b, 0xb0, 0x71, 0x10, 0x7a, 0xfe, 0x73, 0x9f, 0x78, 0x62, 0xa0, 0x25, 0xd1, 0xfc, 0x06, + 0x20, 0x36, 0x66, 0x9c, 0x0c, 0x1d, 0x37, 0x0c, 0x9e, 0xfb, 0x3d, 0x87, 0x45, 0x38, 0x90, 0x41, + 0xd5, 0x2d, 0x23, 0xe6, 0xec, 0x49, 0x86, 0x9c, 0x82, 0x6d, 0x40, 0xb2, 0xd5, 0x06, 0xfe, 0x29, + 0x09, 0x08, 0x63, 0xb1, 0x74, 0x1c, 0xbf, 0x5b, 0x73, 0x1c, 0x16, 0x4a, 0x96, 0x21, 0x54, 0xf6, + 0x95, 0x86, 0xa0, 0xa8, 0x14, 0xff, 0x33, 0x03, 0x9b, 0x66, 0xc0, 0x09, 0x0d, 0xf0, 0x60, 0x2f, + 0x1c, 0x0e, 0x67, 0xed, 0xd5, 0x82, 0x32, 0x13, 0xed, 0xe6, 0xf0, 0x98, 0xa0, 0xaa, 0xb6, 0x36, + 0xf7, 0x86, 0x59, 0x5b, 0x5a, 0x25, 0x96, 0x6e, 0xd2, 0x16, 0x94, 0x87, 0xa2, 0x0f, 0xa6, 0x28, + 0x99, 0xb7, 0xa2, 0xa4, 0xfb, 0xc5, 0x2a, 0x0d, 0xd3, 0xdd, 0xf3, 0x13, 0xb8, 0xa5, 0xca, 0x2d, + 0x49, 0xf0, 0x14, 0x2f, 0x2b, 0xf1, 0xee, 0xcc, 0xc1, 0x9b, 0x5b, 0xb9, 0xd6, 0xa6, 0x3b, 0xb7, + 0xa0, 0x9f, 0xc1, 0xe6, 0x50, 0x65, 0x46, 0xc6, 0x73, 0x8a, 0x9f, 0x93, 0xf8, 0x5f, 0x9b, 0x67, + 0xef, 0xe5, 0x4c, 0x5a, 0x1b, 0xc3, 0xcb, 0xc4, 0x1d, 0xfd, 0x33, 0xb5, 0x75, 0xea, 0x3f, 0xd7, + 0x60, 0xfd, 0xf0, 0x84, 0x11, 0x7a, 0x4a, 0xbc, 0xe9, 0x9e, 0x48, 0xcf, 0x4e, 0x6d, 0x81, 0xd9, + 0xf9, 0x3f, 0x58, 0x44, 0x7a, 0xb2, 0x07, 0xeb, 0x5f, 0x2e, 0xc3, 0x8a, 0x4d, 0x71, 0xc0, 0xb0, + 0xcb, 0xfd, 0x30, 0x40, 0x0f, 0x21, 0x27, 0x36, 0xba, 0x4a, 0xf6, 0xdd, 0x14, 0xae, 0x5a, 0xec, + 0x8d, 0x78, 0xa3, 0x37, 0x92, 0xc5, 0xde, 0xb0, 0xcf, 0x82, 0x03, 0xc2, 0x71, 0x53, 0x17, 0x97, + 0x7c, 0x31, 0xa9, 0x69, 0x96, 0x44, 0x40, 0x08, 0x72, 0x01, 0x1e, 0xc6, 0xeb, 0xab, 0x68, 0xc9, + 0x33, 0xfa, 0x01, 0x2c, 0x33, 0x8e, 0xf9, 0x88, 0xc9, 0xb0, 0xce, 0x9f, 0x18, 0x29, 0x6b, 0xba, + 0x52, 0xd6, 0x52, 0x3a, 0xe8, 0x11, 0xac, 0x0e, 0x30, 0xe3, 0x4e, 0x9f, 0x60, 0xca, 0x4f, 0x08, + 0xe6, 0x95, 0xfc, 0xe2, 0xde, 0x97, 0x85, 0xea, 0xc3, 0x44, 0x53, 0x60, 0x85, 0xd4, 0xef, 0x39, + 0xb3, 0x48, 0x2e, 0x5f, 0x03, 0x4b, 0xa8, 0xce, 0xf2, 0xf7, 0x10, 0xca, 0x43, 0x7c, 0x96, 0x82, + 0x2a, 0x2c, 0x0e, 0x55, 0x1a, 0xe2, 0xb3, 0x19, 0xd2, 0x27, 0xb0, 0x11, 0xaa, 0xf2, 0x98, 0xc1, + 0xb1, 0x8a, 0xfe, 0xd6, 0x61, 0x76, 0xa9, 0x98, 0x14, 0x2c, 0x0a, 0xff, 0x93, 0xc1, 0x50, 0x05, + 0x0a, 0x2f, 0xa9, 0xcf, 0xfd, 0xa0, 0x57, 0x29, 0xca, 0xd1, 0x92, 0x7c, 0xa2, 0xef, 0x42, 0xc1, + 0x0f, 0x38, 0x09, 0x38, 0xab, 0xac, 0xc8, 0xab, 0xde, 0x36, 0x46, 0x92, 0xc9, 0xab, 0xa4, 0x51, + 0x1d, 0xca, 0x02, 0x83, 0x38, 0x3c, 0x0c, 0x9d, 0x70, 0xe0, 0x55, 0x4a, 0x12, 0x78, 0x45, 0x12, + 0xed, 0x30, 0x3c, 0x1c, 0x78, 0x42, 0x86, 0x12, 0x4e, 0xc7, 0x4e, 0x18, 0x38, 0xd1, 0x88, 0xf5, + 0x2b, 0xe5, 0x58, 0x46, 0x12, 0x0f, 0x83, 0xa3, 0x11, 0xeb, 0xa3, 0x63, 0xb8, 0x41, 0xa2, 0xd0, + 0xed, 0x3b, 0x3f, 0x23, 0x34, 0x4c, 0x05, 0x72, 0x75, 0xf1, 0x40, 0x22, 0x09, 0xf0, 0x8c, 0xd0, + 0x70, 0x16, 0x4e, 0x1b, 0x36, 0x28, 0x79, 0x4e, 0x09, 0xeb, 0xa7, 0xe3, 0x59, 0x59, 0xbb, 0x06, + 0xea, 0x54, 0x7f, 0x86, 0xfa, 0x43, 0x78, 0xef, 0x62, 0xe9, 0x38, 0x2f, 0x31, 0x73, 0x92, 0x80, + 0x57, 0x0c, 0xe9, 0x5e, 0xe5, 0x42, 0x89, 0x3c, 0xc5, 0x2c, 0xc9, 0xd4, 0xc5, 0x77, 0x68, 0xfd, + 0xf7, 0x1a, 0x2c, 0x9b, 0x32, 0x9a, 0xe8, 0xdb, 0x90, 0x9b, 0xce, 0xfd, 0x2b, 0x52, 0x90, 0xea, + 0x33, 0x21, 0x8e, 0x9a, 0x90, 0xe5, 0x67, 0xc9, 0xfc, 0xbf, 0x4e, 0xc3, 0xc6, 0x1e, 0x0a, 0xe5, + 0x54, 0x5f, 0x66, 0xaf, 0xdf, 0x97, 0x6a, 0x93, 0x3c, 0x80, 0xd5, 0x2e, 0xf9, 0xe9, 0x88, 0x04, + 0x2e, 0xf1, 0x9e, 0x8a, 0xfc, 0x5f, 0xf5, 0xb8, 0xdd, 0x02, 0x9d, 0x29, 0xe1, 0xf8, 0xbd, 0x68, + 0x4d, 0xbf, 0xeb, 0x5f, 0x66, 0x21, 0xbf, 0x4f, 0x30, 0x23, 0xe8, 0xfb, 0x90, 0x67, 0x1c, 0x53, + 0xae, 0x42, 0xb2, 0x50, 0xc6, 0x62, 0x0d, 0xf4, 0x29, 0x00, 0x39, 0x8b, 0x7c, 0x8a, 0x85, 0xbd, + 0x8b, 0x4d, 0xc9, 0xea, 0xbf, 0x26, 0xb5, 0xad, 0xd4, 0x43, 0x78, 0xa7, 0x4e, 0x71, 0xe0, 0x05, + 0xa3, 0xc1, 0x00, 0x9f, 0x0c, 0x48, 0xdd, 0x4a, 0x01, 0xa6, 0x1f, 0x2e, 0xd9, 0xff, 0xfe, 0xe1, + 0x32, 0x82, 0x5b, 0x1e, 0x89, 0x28, 0x71, 0xe5, 0xdb, 0x45, 0x1a, 0x2e, 0xfe, 0x65, 0x3e, 0x53, + 0x6b, 0xe7, 0x1d, 0x2d, 0xde, 0x9c, 0xa1, 0x77, 0x05, 0x78, 0x57, 0x62, 0xa3, 0x0e, 0xac, 0x44, + 0x34, 0x8c, 0x42, 0x26, 0xba, 0x82, 0x2d, 0x36, 0x44, 0x57, 0xcf, 0x27, 0x35, 0x38, 0x52, 0x5a, + 0x76, 0xd7, 0x82, 0x04, 0xc1, 0x66, 0xe8, 0x06, 0xe4, 0x65, 0xf3, 0xc9, 0x11, 0x9a, 0xb5, 0xe2, + 0x0f, 0xf4, 0x51, 0x2a, 0xc5, 0x62, 0x20, 0x66, 0x9b, 0xeb, 0x6f, 0x26, 0xb5, 0xb2, 0xcc, 0x6c, + 0x52, 0x28, 0xb3, 0xac, 0xc7, 0xaf, 0x24, 0xd1, 0x16, 0xf5, 0x5f, 0x68, 0xb0, 0xba, 0x7b, 0x12, + 0x52, 0x2e, 0xca, 0xbd, 0x1d, 0x70, 0x3a, 0xbe, 0xaa, 0x92, 0xde, 0x7d, 0x1b, 0x8a, 0x62, 0x8c, + 0xa8, 0x1f, 0x52, 0x9f, 0xc7, 0xbf, 0x2d, 0xf3, 0xd6, 0xf4, 0x3b, 0xb5, 0x29, 0xff, 0x91, 0x85, + 0x92, 0x7d, 0x16, 0xec, 0x85, 0x21, 0xf5, 0x44, 0xff, 0xa0, 0xef, 0xc4, 0x8d, 0x17, 0xd7, 0x66, + 0xf5, 0xea, 0x8e, 0x49, 0x37, 0x5b, 0x6a, 0xda, 0x66, 0xae, 0x35, 0x6d, 0x3f, 0x84, 0xb2, 0x1b, + 0x0e, 0x87, 0x38, 0xf0, 0x1c, 0x37, 0x1c, 0x05, 0x5c, 0x19, 0x5b, 0x52, 0xc4, 0x3d, 0x41, 0x43, + 0x4d, 0x31, 0x6e, 0xe5, 0xcc, 0x72, 0x28, 0xc1, 0x9e, 0xa8, 0xa4, 0x05, 0xee, 0x28, 0x29, 0x1d, + 0x4b, 0xa8, 0xa0, 0x16, 0xac, 0x26, 0x18, 0x72, 0x92, 0x8b, 0x1a, 0x59, 0x00, 0x24, 0xb9, 0x58, + 0x76, 0x3f, 0x43, 0xdf, 0x83, 0x4a, 0xaa, 0xba, 0x13, 0xc0, 0x53, 0x3c, 0xf0, 0x3d, 0x59, 0x29, + 0xba, 0x75, 0x73, 0xc6, 0xb7, 0x62, 0xf6, 0x13, 0xc1, 0x45, 0x5f, 0x87, 0xb5, 0x44, 0xdc, 0x0f, + 0x62, 0x85, 0x82, 0x54, 0x48, 0xcc, 0x32, 0x63, 0x2a, 0x7a, 0x02, 0x28, 0x1c, 0x71, 0xc6, 0x71, + 0xe0, 0xf9, 0x41, 0x2f, 0x31, 0x36, 0x5e, 0x97, 0x1f, 0xcc, 0x33, 0xf6, 0xc2, 0x80, 0x52, 0x66, + 0xaf, 0xa7, 0x20, 0x62, 0xd3, 0xef, 0xfe, 0x5a, 0x83, 0xa2, 0xfc, 0x71, 0x2d, 0x7f, 0xa5, 0xac, + 0x40, 0xe1, 0xb8, 0xf3, 0xb8, 0x73, 0xf8, 0xb4, 0x63, 0x2c, 0xa1, 0x02, 0x64, 0xcd, 0x8e, 0x6d, + 0x68, 0xa8, 0x08, 0xf9, 0x8f, 0xf7, 0x0f, 0x77, 0x6d, 0x23, 0x23, 0x8e, 0xcd, 0x1f, 0xdb, 0xed, + 0xae, 0x91, 0x45, 0x1b, 0xb0, 0xd6, 0x6a, 0xef, 0x9b, 0x07, 0xa6, 0xdd, 0x6e, 0x39, 0x31, 0x51, + 0x47, 0x3a, 0xe4, 0x6c, 0xf3, 0xa0, 0x6d, 0xe4, 0x04, 0x54, 0xab, 0xbd, 0x67, 0x1e, 0xec, 0xee, + 0x1b, 0x79, 0xb4, 0x09, 0xeb, 0x33, 0xd9, 0x84, 0x5c, 0x44, 0x25, 0xd0, 0x5b, 0xc7, 0xd6, 0xae, + 0x6d, 0x1e, 0x76, 0x8c, 0x65, 0x81, 0x6d, 0x1f, 0x1f, 0xed, 0xb7, 0x0d, 0x40, 0xab, 0x00, 0x02, + 0xa6, 0xdb, 0xb6, 0xcc, 0x76, 0xd7, 0xf0, 0xea, 0x39, 0xbd, 0x60, 0x14, 0xee, 0xfe, 0x08, 0xd6, + 0x2f, 0xfd, 0xc8, 0x42, 0x6b, 0xb0, 0xb2, 0xdb, 0x6a, 0x39, 0x56, 0xfb, 0x68, 0xdf, 0xdc, 0xdb, + 0x35, 0x96, 0x10, 0x82, 0x55, 0xab, 0x7d, 0x70, 0xf8, 0xa4, 0x3d, 0xa5, 0x69, 0x5b, 0xb9, 0xcf, + 0x7e, 0x55, 0x5d, 0xba, 0xdb, 0x84, 0xf5, 0x4b, 0xa3, 0x5d, 0xd8, 0x79, 0xd4, 0xee, 0xb4, 0xcc, + 0xce, 0x03, 0x63, 0x09, 0x95, 0xa1, 0xb8, 0x77, 0x78, 0x70, 0x60, 0xda, 0x76, 0xbb, 0x65, 0x68, + 0x82, 0xb7, 0xdb, 0x3c, 0xb4, 0xc4, 0x47, 0x26, 0xc6, 0x68, 0x7e, 0xf0, 0xfa, 0xaf, 0xd5, 0xa5, + 0xd7, 0xe7, 0x55, 0xed, 0x8b, 0xf3, 0xaa, 0xf6, 0xa7, 0xf3, 0xaa, 0xf6, 0x97, 0xf3, 0xaa, 0xf6, + 0xf9, 0xdf, 0xaa, 0x4b, 0xcf, 0x0a, 0x2a, 0xf4, 0x27, 0xcb, 0xf2, 0x4f, 0x39, 0xf7, 0xff, 0x1d, + 0x00, 0x00, 0xff, 0xff, 0xce, 0x58, 0xcb, 0x38, 0x60, 0x12, 0x00, 0x00, } diff --git a/pkg/roachpb/data.proto b/pkg/roachpb/data.proto index f0b0430d4ce7..3f8e9a36d17b 100644 --- a/pkg/roachpb/data.proto +++ b/pkg/roachpb/data.proto @@ -382,6 +382,18 @@ message Intent { TransactionStatus status = 3; } +// A SequencedWrite is a point write to a key with a certain sequence number. +// +// TODO(nvanbenschoten/tschottdorf): This message type can be used as the +// PromisedWrites repeated field in EndTransaction in the parallel commits +// proposal (#24194). +message SequencedWrite { + // The key that the write was made at. + bytes key = 1 [(gogoproto.casttype) = "Key"]; + // The sequence number of the request that created the write. + int32 sequence = 2; +} + // Lease contains information about range leases including the // expiration and lease holder. message Lease { @@ -484,4 +496,9 @@ message TxnCoordMeta { // zero value. // TODO(nvanbenschoten): Can be removed in 2.2. bool deprecated_refresh_valid = 6; + // outstanding_writes stores all writes that are outstanding and have + // not yet been resolved. Any client wishing to send a request that + // overlaps with them must chain on to their success using a QueryIntent + // request. + repeated SequencedWrite outstanding_writes = 8 [(gogoproto.nullable) = false]; } diff --git a/pkg/roachpb/errors.pb.go b/pkg/roachpb/errors.pb.go index bc462e1883a9..1dc7a0fc3e83 100644 --- a/pkg/roachpb/errors.pb.go +++ b/pkg/roachpb/errors.pb.go @@ -35,6 +35,8 @@ const ( // A possible replay caused by duplicate begin txn or out-of-order // txn sequence number. RETRY_POSSIBLE_REPLAY TransactionRetryReason = 4 + // An asynchronous write was observed to have failed. + RETRY_ASYNC_WRITE_FAILURE TransactionRetryReason = 5 ) var TransactionRetryReason_name = map[int32]string{ @@ -43,13 +45,15 @@ var TransactionRetryReason_name = map[int32]string{ 2: "RETRY_DELETE_RANGE", 3: "RETRY_SERIALIZABLE", 4: "RETRY_POSSIBLE_REPLAY", + 5: "RETRY_ASYNC_WRITE_FAILURE", } var TransactionRetryReason_value = map[string]int32{ - "RETRY_REASON_UNKNOWN": 0, - "RETRY_WRITE_TOO_OLD": 1, - "RETRY_DELETE_RANGE": 2, - "RETRY_SERIALIZABLE": 3, - "RETRY_POSSIBLE_REPLAY": 4, + "RETRY_REASON_UNKNOWN": 0, + "RETRY_WRITE_TOO_OLD": 1, + "RETRY_DELETE_RANGE": 2, + "RETRY_SERIALIZABLE": 3, + "RETRY_POSSIBLE_REPLAY": 4, + "RETRY_ASYNC_WRITE_FAILURE": 5, } func (x TransactionRetryReason) Enum() *TransactionRetryReason { @@ -7659,158 +7663,159 @@ var ( func init() { proto.RegisterFile("roachpb/errors.proto", fileDescriptorErrors) } var fileDescriptorErrors = []byte{ - // 2437 bytes of a gzipped FileDescriptorProto + // 2457 bytes of a gzipped FileDescriptorProto 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x9c, 0x58, 0xcf, 0x53, 0x1b, 0xc9, - 0xf5, 0x47, 0x48, 0x20, 0x78, 0x42, 0x48, 0xb4, 0x31, 0x3b, 0xc6, 0xb6, 0x84, 0xf1, 0x7e, 0xbf, - 0x6b, 0x3b, 0x15, 0xd8, 0xc2, 0xf1, 0xa6, 0xd6, 0xd9, 0x1c, 0x04, 0x12, 0xac, 0x0c, 0x48, 0xa4, - 0x11, 0x6b, 0xb3, 0x9b, 0xaa, 0xa9, 0x61, 0xa6, 0x11, 0xb3, 0x96, 0x66, 0x94, 0x9e, 0x1e, 0x10, - 0x97, 0x9c, 0x73, 0x4c, 0x6e, 0x39, 0xa4, 0x2a, 0xae, 0xca, 0x29, 0x95, 0x4b, 0x0e, 0xa9, 0x1c, - 0x73, 0xf6, 0x31, 0xc7, 0x54, 0xaa, 0x42, 0x25, 0xe4, 0x90, 0x54, 0xe5, 0x3f, 0xf0, 0x29, 0xd5, - 0x3f, 0x46, 0x1a, 0xa1, 0x19, 0x59, 0xf1, 0x49, 0xa3, 0xd7, 0xef, 0x7d, 0xfa, 0xf5, 0x8f, 0xf7, - 0xde, 0xe7, 0x35, 0x2c, 0x52, 0xd7, 0x30, 0xcf, 0x3a, 0x27, 0xeb, 0x84, 0x52, 0x97, 0x7a, 0x6b, - 0x1d, 0xea, 0x32, 0x17, 0x2d, 0x98, 0xae, 0xf9, 0x5a, 0x8c, 0xac, 0xa9, 0xf1, 0x65, 0x14, 0x28, - 0x5a, 0x06, 0x33, 0xa4, 0xda, 0xf2, 0x52, 0x20, 0x6b, 0x13, 0x66, 0x84, 0xe4, 0x0f, 0x3d, 0xe6, - 0x52, 0xa3, 0x49, 0xd6, 0x89, 0xd3, 0xb4, 0x9d, 0xe0, 0x87, 0xeb, 0x9d, 0x9b, 0xe6, 0x53, 0xa5, - 0xa4, 0xf9, 0xcc, 0x6e, 0xad, 0x9f, 0xb5, 0xcc, 0x75, 0x66, 0xb7, 0x89, 0xc7, 0x8c, 0x76, 0x47, - 0x8d, 0x2c, 0x36, 0xdd, 0xa6, 0x2b, 0x3e, 0xd7, 0xf9, 0x97, 0x94, 0xae, 0xfe, 0x71, 0x12, 0x6e, - 0xd5, 0x5c, 0xb6, 0x47, 0x0c, 0x8f, 0x7c, 0xe9, 0xb6, 0x2c, 0x42, 0x2b, 0xdc, 0x65, 0x54, 0x86, - 0x34, 0x25, 0x9d, 0x96, 0x6d, 0x1a, 0x5a, 0x62, 0x25, 0xf1, 0x28, 0xb3, 0xf1, 0xf1, 0xda, 0x90, - 0xf7, 0x6b, 0x58, 0x6a, 0x94, 0x89, 0x67, 0x52, 0xbb, 0xc3, 0x5c, 0xba, 0x99, 0x7a, 0x7b, 0x55, - 0x9c, 0xc0, 0x81, 0x29, 0xda, 0x81, 0xb9, 0x16, 0x47, 0xd6, 0xcf, 0x04, 0xb4, 0x36, 0x39, 0x3e, - 0x14, 0xce, 0xb4, 0xfa, 0x3e, 0xa1, 0x67, 0x30, 0x43, 0x0d, 0xa7, 0x49, 0x74, 0xdb, 0xd2, 0x92, - 0x2b, 0x89, 0x47, 0xc9, 0xcd, 0x65, 0x3e, 0xd3, 0xf5, 0x55, 0x31, 0x8d, 0xb9, 0xbc, 0x5a, 0x7e, - 0xd7, 0xff, 0xc4, 0x69, 0xa1, 0x5b, 0xb5, 0xd0, 0x1a, 0x4c, 0x09, 0x14, 0x2d, 0x25, 0x26, 0xd6, - 0x22, 0x26, 0x16, 0x2b, 0xc7, 0x52, 0x0d, 0x3d, 0x04, 0x30, 0x7d, 0x8f, 0xb9, 0x6d, 0xbd, 0xed, - 0x35, 0xb5, 0xa9, 0x95, 0xc4, 0xa3, 0x59, 0xb5, 0xa4, 0x59, 0x29, 0xdf, 0xf7, 0x9a, 0xcf, 0x53, - 0xff, 0x7e, 0x53, 0x4c, 0xac, 0xde, 0x83, 0xc5, 0x9a, 0x6b, 0x91, 0x23, 0xc7, 0x38, 0x37, 0xec, - 0x96, 0x71, 0xd2, 0x22, 0x62, 0xe3, 0xd4, 0x68, 0x11, 0x3e, 0x3a, 0x72, 0x3c, 0xbf, 0xd3, 0x71, - 0x29, 0x23, 0x16, 0x26, 0x3f, 0xf1, 0x89, 0xc7, 0xc2, 0x0a, 0x3f, 0x02, 0x24, 0xbc, 0xad, 0xb9, - 0x6c, 0xdb, 0xf5, 0x1d, 0x4b, 0xee, 0x7a, 0x78, 0x99, 0x89, 0xb1, 0x97, 0xa9, 0x20, 0x7f, 0x3d, - 0x09, 0xb7, 0xc5, 0xd0, 0x2e, 0xb9, 0xdc, 0xb7, 0xbd, 0xb6, 0xc1, 0xcc, 0x33, 0x09, 0xfb, 0x14, - 0x16, 0xa8, 0x74, 0x41, 0xf7, 0x98, 0x41, 0x99, 0xfe, 0x9a, 0x5c, 0x0a, 0xfc, 0xb9, 0xcd, 0xf4, - 0xbb, 0xab, 0x62, 0x72, 0x97, 0x5c, 0xe2, 0x9c, 0xd2, 0x38, 0xe4, 0x0a, 0xbb, 0xe4, 0x12, 0xad, - 0x43, 0x20, 0xd2, 0x89, 0x63, 0x09, 0x93, 0xc9, 0x41, 0x93, 0xac, 0x1a, 0xaf, 0x38, 0x16, 0x37, - 0xd8, 0x87, 0x7c, 0x5b, 0x4d, 0x4b, 0x2c, 0x5d, 0xf8, 0x26, 0xce, 0x2a, 0xb3, 0xb1, 0x1a, 0x75, - 0xe0, 0x7c, 0x3c, 0x74, 0xdc, 0xb9, 0xbe, 0xad, 0x18, 0x42, 0xbb, 0x90, 0xf3, 0xfc, 0x66, 0x93, - 0x78, 0xac, 0x87, 0x96, 0x1a, 0x1b, 0x6d, 0xbe, 0x67, 0x2a, 0x46, 0xd4, 0x0e, 0xfd, 0x67, 0x12, - 0x56, 0x31, 0x31, 0xac, 0x97, 0x36, 0x3b, 0xb3, 0x9d, 0x23, 0xc7, 0x24, 0x94, 0x19, 0xb6, 0xc3, - 0x2e, 0xab, 0x0e, 0x23, 0xf4, 0xdc, 0x68, 0xc9, 0xed, 0x7a, 0x01, 0xf3, 0x94, 0x18, 0x96, 0xde, - 0x8b, 0x20, 0x15, 0x02, 0xf7, 0x43, 0x13, 0xf3, 0x30, 0x5b, 0x3b, 0x6b, 0x99, 0x6b, 0x8d, 0x40, - 0x49, 0x5d, 0x94, 0x2c, 0x37, 0xed, 0x09, 0x11, 0x06, 0x44, 0xba, 0xb6, 0xc7, 0x6c, 0xa7, 0x19, - 0xc2, 0x9b, 0x1c, 0x1f, 0x6f, 0x21, 0x30, 0xef, 0x63, 0x6e, 0x42, 0xb6, 0x6d, 0x74, 0x43, 0x70, - 0xc9, 0x31, 0xe0, 0xf0, 0x5c, 0xdb, 0xe8, 0xf6, 0x31, 0xbe, 0x81, 0x5b, 0xee, 0x89, 0x47, 0xe8, - 0x39, 0x09, 0xad, 0xd3, 0xd3, 0x52, 0x2b, 0xc9, 0x98, 0x00, 0xad, 0x2b, 0xed, 0x9b, 0xfe, 0x21, - 0xf7, 0xe6, 0x80, 0xd7, 0x8f, 0x81, 0x06, 0x35, 0x1c, 0xcf, 0x30, 0x99, 0xed, 0x3a, 0xa5, 0x13, - 0x11, 0x0a, 0xe1, 0x18, 0x30, 0x60, 0x31, 0xa4, 0x70, 0xe0, 0x7b, 0xea, 0xba, 0x6e, 0x01, 0x74, - 0x7c, 0xef, 0x8c, 0x10, 0x9d, 0x75, 0x1d, 0xb5, 0xf7, 0x85, 0x08, 0x97, 0x42, 0xc6, 0x41, 0x94, - 0x4a, 0xbb, 0x46, 0xd7, 0x51, 0x53, 0x9c, 0xc2, 0xed, 0x90, 0x16, 0x26, 0x8c, 0x5e, 0xca, 0x39, - 0x76, 0x60, 0x9a, 0x12, 0xc3, 0x73, 0x25, 0xfe, 0xfc, 0xc6, 0xe3, 0xd1, 0xf8, 0xc2, 0x12, 0x0b, - 0x03, 0x35, 0x95, 0x32, 0x57, 0xf3, 0x14, 0x60, 0x69, 0x40, 0xbb, 0xd3, 0x32, 0x2e, 0xc3, 0x4b, - 0xfd, 0x53, 0x62, 0x40, 0xe1, 0x90, 0x19, 0xcc, 0xf7, 0xa4, 0x27, 0x4b, 0x90, 0xe4, 0xc9, 0x26, - 0x11, 0x4a, 0x36, 0x5c, 0x80, 0x6a, 0x3d, 0x0f, 0x27, 0x85, 0x87, 0x9f, 0x8e, 0xf6, 0x30, 0x04, - 0xb9, 0x16, 0xe5, 0xe8, 0xea, 0x67, 0x30, 0x2d, 0xe5, 0x08, 0xc1, 0x3c, 0xae, 0x94, 0x0e, 0xeb, - 0x35, 0xfd, 0xa8, 0xb6, 0x5b, 0xab, 0xbf, 0xac, 0xe5, 0x27, 0x90, 0x06, 0x8b, 0x4a, 0xd6, 0x78, - 0x55, 0xd3, 0x6b, 0xf5, 0x86, 0xbe, 0x5d, 0x3f, 0xaa, 0x95, 0xf3, 0x09, 0xb5, 0x80, 0x63, 0xc8, - 0xbf, 0xa4, 0x36, 0x23, 0x3c, 0x52, 0x1c, 0x99, 0xc9, 0xd0, 0xe7, 0x90, 0xb6, 0xc5, 0x5f, 0x4f, - 0x4b, 0x88, 0x7b, 0x73, 0x27, 0xc2, 0x45, 0x69, 0x10, 0x14, 0x06, 0xa5, 0x2f, 0x41, 0x5f, 0xa4, - 0x66, 0x26, 0xf3, 0xc9, 0xd5, 0xdf, 0x25, 0x14, 0x76, 0xc3, 0x75, 0xeb, 0x2d, 0x95, 0x09, 0x4b, - 0x30, 0xfb, 0x41, 0xe1, 0xd7, 0xb7, 0x42, 0x35, 0xc8, 0x1b, 0x26, 0xf3, 0x8d, 0xd6, 0x87, 0x05, - 0x5e, 0x4e, 0x1a, 0xf7, 0xc4, 0x6a, 0x23, 0x96, 0x01, 0xd5, 0x3b, 0x3c, 0xa1, 0xdb, 0x94, 0x78, - 0x8d, 0xae, 0x13, 0x3e, 0xe5, 0x63, 0x58, 0xdc, 0x72, 0x1d, 0xcb, 0xe6, 0xe7, 0xb1, 0x6d, 0xd8, - 0x2d, 0x75, 0xdd, 0xd1, 0x0f, 0x60, 0x4e, 0x79, 0x72, 0x6e, 0xb4, 0x7c, 0xa2, 0xd6, 0x13, 0x55, - 0x8d, 0xbe, 0xe2, 0xe3, 0x38, 0x23, 0xb5, 0xc5, 0x1f, 0x05, 0xfd, 0x87, 0x04, 0x20, 0x59, 0xaa, - 0xc8, 0xb7, 0xc4, 0x0c, 0x02, 0x09, 0x15, 0x20, 0xdd, 0x26, 0x9e, 0x67, 0x34, 0xc9, 0xc0, 0x05, - 0x0a, 0x84, 0xe8, 0x0b, 0x98, 0x55, 0x49, 0x9a, 0x58, 0x6a, 0xf1, 0xb1, 0x45, 0x30, 0xd8, 0xc1, - 0x9e, 0x01, 0x7a, 0x0e, 0x33, 0x41, 0xf6, 0x51, 0x39, 0xe6, 0x7d, 0xc6, 0x3d, 0x7d, 0xe5, 0xf6, - 0xf7, 0x61, 0xf6, 0x90, 0x38, 0xe3, 0x39, 0x3b, 0x70, 0x29, 0x2e, 0x60, 0xb1, 0xd4, 0x3e, 0xb1, - 0x9b, 0xbe, 0xeb, 0x7b, 0x98, 0x78, 0x7e, 0x8b, 0x8d, 0xb7, 0xe0, 0xcf, 0x21, 0x73, 0x41, 0x8d, - 0x4e, 0x87, 0x58, 0x3a, 0xa1, 0x74, 0xc4, 0x92, 0x05, 0x1c, 0x06, 0xa5, 0x5c, 0xa1, 0xc1, 0x19, - 0xde, 0xe7, 0x45, 0xf4, 0x94, 0xed, 0x50, 0xd7, 0xef, 0x94, 0x49, 0x8b, 0xdc, 0xc8, 0x59, 0x04, - 0x96, 0x14, 0x55, 0xd9, 0x72, 0x29, 0xf5, 0x3b, 0xfc, 0xa8, 0xa5, 0x67, 0x0f, 0x60, 0x56, 0xb0, - 0x3d, 0xfd, 0x66, 0x34, 0xcf, 0x08, 0xf1, 0xbe, 0xd7, 0x44, 0xab, 0x30, 0xdb, 0xa1, 0xae, 0x49, - 0x3c, 0x4f, 0x9d, 0xc6, 0x4c, 0x2f, 0x6f, 0x05, 0xe2, 0xde, 0x4d, 0x42, 0x6a, 0x9a, 0x70, 0x50, - 0xfc, 0x10, 0x40, 0x31, 0xab, 0x80, 0x20, 0x4c, 0x6d, 0x16, 0x14, 0x41, 0x98, 0x55, 0xfa, 0x82, - 0x22, 0xf4, 0xff, 0xf0, 0xe3, 0x94, 0x9f, 0x56, 0x9f, 0x79, 0x1c, 0x32, 0x97, 0x0e, 0x33, 0x0f, - 0x4e, 0x2f, 0xa3, 0x98, 0x87, 0xd0, 0x96, 0xcc, 0x43, 0x7d, 0xe2, 0xb4, 0xd0, 0xed, 0x41, 0xee, - 0x73, 0xb6, 0x73, 0x66, 0x38, 0x56, 0x8b, 0x73, 0x1d, 0x46, 0x2f, 0x7b, 0x74, 0x08, 0x6d, 0x40, - 0xaa, 0x53, 0xa1, 0x74, 0xc4, 0x95, 0x17, 0x7a, 0x6a, 0x1f, 0x84, 0xee, 0xea, 0xdf, 0x12, 0xa0, - 0x7d, 0x79, 0x03, 0x2d, 0x88, 0xb4, 0xd8, 0x74, 0xf9, 0x0d, 0x4c, 0xb3, 0xae, 0xc3, 0xdd, 0x97, - 0x2c, 0xa5, 0xcc, 0x87, 0xfe, 0x7a, 0x55, 0x7c, 0xda, 0xb4, 0xd9, 0x99, 0x7f, 0xb2, 0x66, 0xba, - 0xed, 0xf5, 0xde, 0xe4, 0xd6, 0x49, 0xff, 0x7b, 0xbd, 0xf3, 0xba, 0xb9, 0x2e, 0x58, 0xb3, 0xef, - 0xdb, 0xd6, 0xda, 0xd1, 0x51, 0xb5, 0x7c, 0x7d, 0x55, 0x9c, 0x6a, 0x74, 0x9d, 0x6a, 0x19, 0x4f, - 0xb1, 0xae, 0x53, 0xb5, 0xd0, 0x36, 0x64, 0x58, 0x3f, 0xd5, 0xaa, 0x58, 0x18, 0xaf, 0x24, 0x85, - 0x0d, 0xd5, 0x76, 0xdd, 0x85, 0x5b, 0x8d, 0xae, 0x73, 0x40, 0xc9, 0x79, 0x89, 0x31, 0xd2, 0xee, - 0x0c, 0x10, 0xc3, 0x4f, 0xa0, 0xd8, 0xe8, 0x3a, 0xa5, 0x16, 0xa7, 0x11, 0x97, 0x15, 0xc7, 0x74, - 0x7d, 0xce, 0x4d, 0xd4, 0x25, 0x0c, 0x2b, 0xfe, 0x22, 0x01, 0x8b, 0x3c, 0xb9, 0x36, 0x09, 0xad, - 0x9f, 0x13, 0x7a, 0xda, 0x72, 0x2f, 0xe4, 0x0e, 0xdd, 0x81, 0x64, 0x04, 0xbf, 0xe3, 0x32, 0xf4, - 0x18, 0xb2, 0xa6, 0x4f, 0x29, 0x71, 0x98, 0xca, 0x44, 0x93, 0xe2, 0xa8, 0xa5, 0xa7, 0x73, 0x6a, - 0x48, 0xa4, 0x1d, 0xf4, 0x5d, 0xc8, 0xd9, 0x8e, 0x49, 0x49, 0xbb, 0xaf, 0x9c, 0x0c, 0x29, 0xcf, - 0xf7, 0x06, 0xc3, 0x59, 0x6a, 0x1f, 0x16, 0xf6, 0xed, 0x2e, 0xb1, 0x0e, 0x7d, 0x93, 0x5f, 0xe7, - 0xe0, 0x0a, 0xa4, 0x55, 0x94, 0xbd, 0xef, 0x16, 0xe0, 0x40, 0x51, 0xc1, 0xfd, 0x36, 0x01, 0x77, - 0x37, 0x39, 0x27, 0xec, 0xe7, 0x66, 0x72, 0xea, 0x52, 0xb2, 0xb3, 0xd5, 0x2b, 0x12, 0x8d, 0x0f, - 0x2a, 0x12, 0x7d, 0x1e, 0xc4, 0x21, 0xce, 0x28, 0xf1, 0x78, 0x83, 0xf2, 0xbf, 0x54, 0x87, 0xbe, - 0x95, 0xf2, 0xf5, 0x15, 0x20, 0x59, 0xea, 0xf6, 0x6d, 0xcf, 0xb3, 0x9d, 0xa6, 0xf4, 0xf0, 0x0b, - 0x98, 0xbb, 0xa0, 0xae, 0xd3, 0xd4, 0x65, 0xe1, 0x53, 0x4e, 0xc6, 0xd7, 0x49, 0x9c, 0x11, 0xea, - 0xf2, 0x8f, 0x42, 0xfe, 0xd7, 0x2d, 0xc8, 0x08, 0xb4, 0x32, 0x61, 0x86, 0xdd, 0x42, 0x07, 0x90, - 0x77, 0x5c, 0xa6, 0x0f, 0x34, 0x56, 0x12, 0xf7, 0xff, 0x23, 0x70, 0x23, 0x9a, 0x3b, 0x3c, 0xef, - 0x0c, 0x08, 0xd1, 0x3e, 0xe4, 0x64, 0xdb, 0xc1, 0x71, 0x4f, 0x79, 0x52, 0x50, 0x5b, 0xf1, 0x7f, - 0x71, 0x54, 0x7b, 0x20, 0x79, 0xe0, 0x2c, 0x0d, 0xcb, 0xd0, 0x57, 0x80, 0x24, 0xdc, 0x6b, 0x72, - 0xa9, 0x07, 0xb4, 0x5e, 0x05, 0xcd, 0xa3, 0x38, 0xc4, 0x9b, 0x4d, 0x0b, 0xce, 0xd3, 0x1b, 0x62, - 0xf4, 0x53, 0x58, 0x11, 0xbc, 0xfc, 0x42, 0xd0, 0x77, 0xdd, 0xef, 0xf3, 0x77, 0xb1, 0xbd, 0x9c, - 0xc0, 0xab, 0x16, 0xe1, 0x59, 0x64, 0x87, 0xf9, 0x3e, 0xe2, 0x8f, 0xef, 0xd3, 0x51, 0x3a, 0x9c, - 0x33, 0x87, 0x82, 0x59, 0x37, 0x24, 0xa3, 0x15, 0x6d, 0x62, 0x66, 0xe3, 0xc9, 0xe8, 0x6c, 0x10, + 0xf5, 0x47, 0x48, 0x20, 0x78, 0x42, 0x48, 0xb4, 0x31, 0x3b, 0xc6, 0x8b, 0xc4, 0xe2, 0xfd, 0x7e, + 0xd7, 0x76, 0x2a, 0xb0, 0x85, 0xe3, 0x4d, 0xad, 0xb3, 0x39, 0x08, 0x24, 0x58, 0x19, 0x90, 0x48, + 0x23, 0xd6, 0x66, 0x37, 0x55, 0x53, 0xc3, 0x4c, 0x23, 0x66, 0x2d, 0xcd, 0x28, 0x3d, 0x3d, 0x20, + 0x2e, 0x39, 0xe7, 0x98, 0xdc, 0x72, 0x8b, 0xab, 0x72, 0x4a, 0xe5, 0x92, 0x43, 0x2a, 0x97, 0x54, + 0xe5, 0xec, 0x63, 0x8e, 0xa9, 0x54, 0x85, 0x4a, 0xc8, 0x21, 0xa9, 0xca, 0x7f, 0xe0, 0x53, 0xaa, + 0x7f, 0x8c, 0x34, 0x42, 0x33, 0xb2, 0xe2, 0x93, 0x46, 0xaf, 0xdf, 0xfb, 0xf4, 0xeb, 0x1f, 0xef, + 0xbd, 0xcf, 0x6b, 0x58, 0xa4, 0xae, 0x61, 0x9e, 0x77, 0x4e, 0x37, 0x08, 0xa5, 0x2e, 0xf5, 0xd6, + 0x3b, 0xd4, 0x65, 0x2e, 0x5a, 0x30, 0x5d, 0xf3, 0x95, 0x18, 0x59, 0x57, 0xe3, 0xcb, 0x28, 0x50, + 0xb4, 0x0c, 0x66, 0x48, 0xb5, 0xe5, 0xa5, 0x40, 0xd6, 0x26, 0xcc, 0x08, 0xc9, 0x1f, 0x78, 0xcc, + 0xa5, 0x46, 0x93, 0x6c, 0x10, 0xa7, 0x69, 0x3b, 0xc1, 0x0f, 0xd7, 0xbb, 0x30, 0xcd, 0x27, 0x4a, + 0x49, 0xf3, 0x99, 0xdd, 0xda, 0x38, 0x6f, 0x99, 0x1b, 0xcc, 0x6e, 0x13, 0x8f, 0x19, 0xed, 0x8e, + 0x1a, 0x59, 0x6c, 0xba, 0x4d, 0x57, 0x7c, 0x6e, 0xf0, 0x2f, 0x29, 0x5d, 0xfb, 0xc3, 0x24, 0xdc, + 0xa9, 0xb9, 0x6c, 0x9f, 0x18, 0x1e, 0xf9, 0xd2, 0x6d, 0x59, 0x84, 0x56, 0xb8, 0xcb, 0xa8, 0x0c, + 0x69, 0x4a, 0x3a, 0x2d, 0xdb, 0x34, 0xb4, 0xc4, 0x6a, 0xe2, 0x61, 0x66, 0xf3, 0xe3, 0xf5, 0x21, + 0xef, 0xd7, 0xb1, 0xd4, 0x28, 0x13, 0xcf, 0xa4, 0x76, 0x87, 0xb9, 0x74, 0x2b, 0xf5, 0xe6, 0xba, + 0x38, 0x81, 0x03, 0x53, 0xb4, 0x0b, 0x73, 0x2d, 0x8e, 0xac, 0x9f, 0x0b, 0x68, 0x6d, 0x72, 0x7c, + 0x28, 0x9c, 0x69, 0xf5, 0x7d, 0x42, 0x4f, 0x61, 0x86, 0x1a, 0x4e, 0x93, 0xe8, 0xb6, 0xa5, 0x25, + 0x57, 0x13, 0x0f, 0x93, 0x5b, 0xcb, 0x7c, 0xa6, 0x9b, 0xeb, 0x62, 0x1a, 0x73, 0x79, 0xb5, 0xfc, + 0xb6, 0xff, 0x89, 0xd3, 0x42, 0xb7, 0x6a, 0xa1, 0x75, 0x98, 0x12, 0x28, 0x5a, 0x4a, 0x4c, 0xac, + 0x45, 0x4c, 0x2c, 0x56, 0x8e, 0xa5, 0x1a, 0x7a, 0x00, 0x60, 0xfa, 0x1e, 0x73, 0xdb, 0x7a, 0xdb, + 0x6b, 0x6a, 0x53, 0xab, 0x89, 0x87, 0xb3, 0x6a, 0x49, 0xb3, 0x52, 0x7e, 0xe0, 0x35, 0x9f, 0xa5, + 0xfe, 0xfd, 0xba, 0x98, 0x58, 0xfb, 0x10, 0x16, 0x6b, 0xae, 0x45, 0x8e, 0x1d, 0xe3, 0xc2, 0xb0, + 0x5b, 0xc6, 0x69, 0x8b, 0x88, 0x8d, 0x53, 0xa3, 0x45, 0xf8, 0xe0, 0xd8, 0xf1, 0xfc, 0x4e, 0xc7, + 0xa5, 0x8c, 0x58, 0x98, 0xfc, 0xc4, 0x27, 0x1e, 0x0b, 0x2b, 0xfc, 0x08, 0x90, 0xf0, 0xb6, 0xe6, + 0xb2, 0x1d, 0xd7, 0x77, 0x2c, 0xb9, 0xeb, 0xe1, 0x65, 0x26, 0xc6, 0x5e, 0xa6, 0x82, 0xfc, 0xd5, + 0x24, 0xdc, 0x15, 0x43, 0x7b, 0xe4, 0xea, 0xc0, 0xf6, 0xda, 0x06, 0x33, 0xcf, 0x25, 0xec, 0x13, + 0x58, 0xa0, 0xd2, 0x05, 0xdd, 0x63, 0x06, 0x65, 0xfa, 0x2b, 0x72, 0x25, 0xf0, 0xe7, 0xb6, 0xd2, + 0x6f, 0xaf, 0x8b, 0xc9, 0x3d, 0x72, 0x85, 0x73, 0x4a, 0xe3, 0x88, 0x2b, 0xec, 0x91, 0x2b, 0xb4, + 0x01, 0x81, 0x48, 0x27, 0x8e, 0x25, 0x4c, 0x26, 0x07, 0x4d, 0xb2, 0x6a, 0xbc, 0xe2, 0x58, 0xdc, + 0xe0, 0x00, 0xf2, 0x6d, 0x35, 0x2d, 0xb1, 0x74, 0xe1, 0x9b, 0x38, 0xab, 0xcc, 0xe6, 0x5a, 0xd4, + 0x81, 0xf3, 0xf1, 0xd0, 0x71, 0xe7, 0xfa, 0xb6, 0x62, 0x08, 0xed, 0x41, 0xce, 0xf3, 0x9b, 0x4d, + 0xe2, 0xb1, 0x1e, 0x5a, 0x6a, 0x6c, 0xb4, 0xf9, 0x9e, 0xa9, 0x18, 0x51, 0x3b, 0xf4, 0x9f, 0x49, + 0x58, 0xc3, 0xc4, 0xb0, 0x5e, 0xd8, 0xec, 0xdc, 0x76, 0x8e, 0x1d, 0x93, 0x50, 0x66, 0xd8, 0x0e, + 0xbb, 0xaa, 0x3a, 0x8c, 0xd0, 0x0b, 0xa3, 0x25, 0xb7, 0xeb, 0x39, 0xcc, 0x53, 0x62, 0x58, 0x7a, + 0x2f, 0x82, 0x54, 0x08, 0xac, 0x84, 0x26, 0xe6, 0x61, 0xb6, 0x7e, 0xde, 0x32, 0xd7, 0x1b, 0x81, + 0x92, 0xba, 0x28, 0x59, 0x6e, 0xda, 0x13, 0x22, 0x0c, 0x88, 0x74, 0x6d, 0x8f, 0xd9, 0x4e, 0x33, + 0x84, 0x37, 0x39, 0x3e, 0xde, 0x42, 0x60, 0xde, 0xc7, 0xdc, 0x82, 0x6c, 0xdb, 0xe8, 0x86, 0xe0, + 0x92, 0x63, 0xc0, 0xe1, 0xb9, 0xb6, 0xd1, 0xed, 0x63, 0x7c, 0x03, 0x77, 0xdc, 0x53, 0x8f, 0xd0, + 0x0b, 0x12, 0x5a, 0xa7, 0xa7, 0xa5, 0x56, 0x93, 0x31, 0x01, 0x5a, 0x57, 0xda, 0xb7, 0xfd, 0x43, + 0xee, 0xed, 0x01, 0xaf, 0x1f, 0x03, 0x0d, 0x6a, 0x38, 0x9e, 0x61, 0x32, 0xdb, 0x75, 0x4a, 0xa7, + 0x22, 0x14, 0xc2, 0x31, 0x60, 0xc0, 0x62, 0x48, 0xe1, 0xd0, 0xf7, 0xd4, 0x75, 0xdd, 0x06, 0xe8, + 0xf8, 0xde, 0x39, 0x21, 0x3a, 0xeb, 0x3a, 0x6a, 0xef, 0x0b, 0x11, 0x2e, 0x85, 0x8c, 0x83, 0x28, + 0x95, 0x76, 0x8d, 0xae, 0xa3, 0xa6, 0x38, 0x83, 0xbb, 0x21, 0x2d, 0x4c, 0x18, 0xbd, 0x92, 0x73, + 0xec, 0xc2, 0x34, 0x25, 0x86, 0xe7, 0x4a, 0xfc, 0xf9, 0xcd, 0x47, 0xa3, 0xf1, 0x85, 0x25, 0x16, + 0x06, 0x6a, 0x2a, 0x65, 0xae, 0xe6, 0x29, 0xc0, 0xd2, 0x80, 0x76, 0xa7, 0x65, 0x5c, 0x85, 0x97, + 0xfa, 0xa7, 0xc4, 0x80, 0xc2, 0x11, 0x33, 0x98, 0xef, 0x49, 0x4f, 0x96, 0x20, 0xc9, 0x93, 0x4d, + 0x22, 0x94, 0x6c, 0xb8, 0x00, 0xd5, 0x7a, 0x1e, 0x4e, 0x0a, 0x0f, 0x3f, 0x1d, 0xed, 0x61, 0x08, + 0x72, 0x3d, 0xca, 0xd1, 0xb5, 0xcf, 0x60, 0x5a, 0xca, 0x11, 0x82, 0x79, 0x5c, 0x29, 0x1d, 0xd5, + 0x6b, 0xfa, 0x71, 0x6d, 0xaf, 0x56, 0x7f, 0x51, 0xcb, 0x4f, 0x20, 0x0d, 0x16, 0x95, 0xac, 0xf1, + 0xb2, 0xa6, 0xd7, 0xea, 0x0d, 0x7d, 0xa7, 0x7e, 0x5c, 0x2b, 0xe7, 0x13, 0x6a, 0x01, 0x27, 0x90, + 0x7f, 0x41, 0x6d, 0x46, 0x78, 0xa4, 0x38, 0x32, 0x93, 0xa1, 0xcf, 0x21, 0x6d, 0x8b, 0xbf, 0x9e, + 0x96, 0x10, 0xf7, 0xe6, 0x5e, 0x84, 0x8b, 0xd2, 0x20, 0x28, 0x0c, 0x4a, 0x5f, 0x82, 0x3e, 0x4f, + 0xcd, 0x4c, 0xe6, 0x93, 0x6b, 0xbf, 0x4d, 0x28, 0xec, 0x86, 0xeb, 0xd6, 0x5b, 0x2a, 0x13, 0x96, + 0x60, 0xf6, 0xbd, 0xc2, 0xaf, 0x6f, 0x85, 0x6a, 0x90, 0x37, 0x4c, 0xe6, 0x1b, 0xad, 0xf7, 0x0b, + 0xbc, 0x9c, 0x34, 0xee, 0x89, 0xd5, 0x46, 0x2c, 0x03, 0xaa, 0x77, 0x78, 0x42, 0xb7, 0x29, 0xf1, + 0x1a, 0x5d, 0x27, 0x7c, 0xca, 0x27, 0xb0, 0xb8, 0xed, 0x3a, 0x96, 0xcd, 0xcf, 0x63, 0xc7, 0xb0, + 0x5b, 0xea, 0xba, 0xa3, 0x1f, 0xc0, 0x9c, 0xf2, 0xe4, 0xc2, 0x68, 0xf9, 0x44, 0xad, 0x27, 0xaa, + 0x1a, 0x7d, 0xc5, 0xc7, 0x71, 0x46, 0x6a, 0x8b, 0x3f, 0x0a, 0xfa, 0xf7, 0x09, 0x40, 0xb2, 0x54, + 0x91, 0x6f, 0x89, 0x19, 0x04, 0x12, 0x2a, 0x40, 0xba, 0x4d, 0x3c, 0xcf, 0x68, 0x92, 0x81, 0x0b, + 0x14, 0x08, 0xd1, 0x17, 0x30, 0xab, 0x92, 0x34, 0xb1, 0xd4, 0xe2, 0x63, 0x8b, 0x60, 0xb0, 0x83, + 0x3d, 0x03, 0xf4, 0x0c, 0x66, 0x82, 0xec, 0xa3, 0x72, 0xcc, 0xbb, 0x8c, 0x7b, 0xfa, 0xca, 0xed, + 0xef, 0xc3, 0xec, 0x11, 0x71, 0xc6, 0x73, 0x76, 0xe0, 0x52, 0x5c, 0xc2, 0x62, 0xa9, 0x7d, 0x6a, + 0x37, 0x7d, 0xd7, 0xf7, 0x30, 0xf1, 0xfc, 0x16, 0x1b, 0x6f, 0xc1, 0x9f, 0x43, 0xe6, 0x92, 0x1a, + 0x9d, 0x0e, 0xb1, 0x74, 0x42, 0xe9, 0x88, 0x25, 0x0b, 0x38, 0x0c, 0x4a, 0xb9, 0x42, 0x83, 0x33, + 0x5c, 0xe1, 0x45, 0xf4, 0x8c, 0xed, 0x52, 0xd7, 0xef, 0x94, 0x49, 0x8b, 0xdc, 0xca, 0x59, 0x04, + 0x96, 0x14, 0x55, 0xd9, 0x76, 0x29, 0xf5, 0x3b, 0xfc, 0xa8, 0xa5, 0x67, 0x1f, 0xc1, 0xac, 0x60, + 0x7b, 0xfa, 0xed, 0x68, 0x9e, 0x11, 0xe2, 0x03, 0xaf, 0x89, 0xd6, 0x60, 0xb6, 0x43, 0x5d, 0x93, + 0x78, 0x9e, 0x3a, 0x8d, 0x99, 0x5e, 0xde, 0x0a, 0xc4, 0xbd, 0x9b, 0x84, 0xd4, 0x34, 0xe1, 0xa0, + 0xf8, 0x21, 0x80, 0x62, 0x56, 0x01, 0x41, 0x98, 0xda, 0x2a, 0x28, 0x82, 0x30, 0xab, 0xf4, 0x05, + 0x45, 0xe8, 0xff, 0xe1, 0xc7, 0x29, 0x3f, 0xad, 0x3e, 0xf3, 0x38, 0x62, 0x2e, 0x1d, 0x66, 0x1e, + 0x9c, 0x5e, 0x46, 0x31, 0x0f, 0xa1, 0x2d, 0x99, 0x87, 0xfa, 0xc4, 0x69, 0xa1, 0xdb, 0x83, 0x3c, + 0xe0, 0x6c, 0xe7, 0xdc, 0x70, 0xac, 0x16, 0xe7, 0x3a, 0x8c, 0x5e, 0xf5, 0xe8, 0x10, 0xda, 0x84, + 0x54, 0xa7, 0x42, 0xe9, 0x88, 0x2b, 0x2f, 0xf4, 0xd4, 0x3e, 0x08, 0xdd, 0xb5, 0xbf, 0x25, 0x40, + 0xfb, 0xf2, 0x16, 0x5a, 0x10, 0x69, 0xb1, 0xe9, 0xf2, 0x1b, 0x98, 0x66, 0x5d, 0x87, 0xbb, 0x2f, + 0x59, 0x4a, 0x99, 0x0f, 0xfd, 0xf5, 0xba, 0xf8, 0xa4, 0x69, 0xb3, 0x73, 0xff, 0x74, 0xdd, 0x74, + 0xdb, 0x1b, 0xbd, 0xc9, 0xad, 0xd3, 0xfe, 0xf7, 0x46, 0xe7, 0x55, 0x73, 0x43, 0xb0, 0x66, 0xdf, + 0xb7, 0xad, 0xf5, 0xe3, 0xe3, 0x6a, 0xf9, 0xe6, 0xba, 0x38, 0xd5, 0xe8, 0x3a, 0xd5, 0x32, 0x9e, + 0x62, 0x5d, 0xa7, 0x6a, 0xa1, 0x1d, 0xc8, 0xb0, 0x7e, 0xaa, 0x55, 0xb1, 0x30, 0x5e, 0x49, 0x0a, + 0x1b, 0xaa, 0xed, 0xba, 0x0f, 0x77, 0x1a, 0x5d, 0xe7, 0x90, 0x92, 0x8b, 0x12, 0x63, 0xa4, 0xdd, + 0x19, 0x20, 0x86, 0x9f, 0x40, 0xb1, 0xd1, 0x75, 0x4a, 0x2d, 0x4e, 0x23, 0xae, 0x2a, 0x8e, 0xe9, + 0xfa, 0x9c, 0x9b, 0xa8, 0x4b, 0x18, 0x56, 0xfc, 0x45, 0x02, 0x16, 0x79, 0x72, 0x6d, 0x12, 0x5a, + 0xbf, 0x20, 0xf4, 0xac, 0xe5, 0x5e, 0xca, 0x1d, 0xba, 0x07, 0xc9, 0x08, 0x7e, 0xc7, 0x65, 0xe8, + 0x11, 0x64, 0x4d, 0x9f, 0x52, 0xe2, 0x30, 0x95, 0x89, 0x26, 0xc5, 0x51, 0x4b, 0x4f, 0xe7, 0xd4, + 0x90, 0x48, 0x3b, 0xe8, 0xbb, 0x90, 0xb3, 0x1d, 0x93, 0x92, 0x76, 0x5f, 0x39, 0x19, 0x52, 0x9e, + 0xef, 0x0d, 0x86, 0xb3, 0xd4, 0x01, 0x2c, 0x1c, 0xd8, 0x5d, 0x62, 0x1d, 0xf9, 0x26, 0xbf, 0xce, + 0xc1, 0x15, 0x48, 0xab, 0x28, 0x7b, 0xd7, 0x2d, 0xc0, 0x81, 0xa2, 0x82, 0xfb, 0x4d, 0x02, 0xee, + 0x6f, 0x71, 0x4e, 0xd8, 0xcf, 0xcd, 0xe4, 0xcc, 0xa5, 0x64, 0x77, 0xbb, 0x57, 0x24, 0x1a, 0xef, + 0x55, 0x24, 0xfa, 0x3c, 0x88, 0x43, 0x9c, 0x53, 0xe2, 0xf1, 0x06, 0xe5, 0x7f, 0xa9, 0x0e, 0x7d, + 0x2b, 0xe5, 0xeb, 0x4b, 0x40, 0xb2, 0xd4, 0x1d, 0xd8, 0x9e, 0x67, 0x3b, 0x4d, 0xe9, 0xe1, 0x17, + 0x30, 0x77, 0x49, 0x5d, 0xa7, 0xa9, 0xcb, 0xc2, 0xa7, 0x9c, 0x8c, 0xaf, 0x93, 0x38, 0x23, 0xd4, + 0xe5, 0x1f, 0x85, 0xfc, 0xaf, 0x3b, 0x90, 0x11, 0x68, 0x65, 0xc2, 0x0c, 0xbb, 0x85, 0x0e, 0x21, + 0xef, 0xb8, 0x4c, 0x1f, 0x68, 0xac, 0x24, 0xee, 0xff, 0x47, 0xe0, 0x46, 0x34, 0x77, 0x78, 0xde, + 0x19, 0x10, 0xa2, 0x03, 0xc8, 0xc9, 0xb6, 0x83, 0xe3, 0x9e, 0xf1, 0xa4, 0xa0, 0xb6, 0xe2, 0xff, + 0xe2, 0xa8, 0xf6, 0x40, 0xf2, 0xc0, 0x59, 0x1a, 0x96, 0xa1, 0xaf, 0x00, 0x49, 0xb8, 0x57, 0xe4, + 0x4a, 0x0f, 0x68, 0xbd, 0x0a, 0x9a, 0x87, 0x71, 0x88, 0xb7, 0x9b, 0x16, 0x9c, 0xa7, 0xb7, 0xc4, + 0xe8, 0xa7, 0xb0, 0x2a, 0x78, 0xf9, 0xa5, 0xa0, 0xef, 0xba, 0xdf, 0xe7, 0xef, 0x62, 0x7b, 0x39, + 0x81, 0x57, 0x2d, 0xc2, 0xd3, 0xc8, 0x0e, 0xf3, 0x5d, 0xc4, 0x1f, 0xaf, 0xd0, 0x51, 0x3a, 0x9c, + 0x33, 0x87, 0x82, 0x59, 0x37, 0x24, 0xa3, 0x15, 0x6d, 0x62, 0x66, 0xf3, 0xf1, 0xe8, 0x6c, 0x10, 0xa6, 0xbf, 0x18, 0xb1, 0xa1, 0x01, 0x84, 0x21, 0x1f, 0x06, 0xe7, 0x44, 0x56, 0x9b, 0x16, 0xc8, - 0x9f, 0x8c, 0x46, 0xee, 0xf1, 0x66, 0x9c, 0x63, 0x83, 0x52, 0x74, 0x04, 0x0b, 0x61, 0x4c, 0xca, - 0x93, 0xa9, 0x96, 0x8e, 0x3d, 0x87, 0x48, 0xa6, 0x8c, 0xc3, 0x6e, 0x09, 0x31, 0x7a, 0x05, 0xe1, - 0x05, 0xf0, 0x96, 0x92, 0xf9, 0x9e, 0x36, 0x23, 0x70, 0x1f, 0x8f, 0xcd, 0x52, 0x71, 0xd8, 0x37, - 0x29, 0x47, 0xdb, 0x3c, 0x5c, 0x6c, 0x46, 0x82, 0x70, 0x99, 0x15, 0x98, 0x0f, 0x23, 0x30, 0x6f, - 0x92, 0x51, 0x1e, 0x38, 0x3d, 0x09, 0xda, 0x81, 0xac, 0xc4, 0x61, 0xae, 0xab, 0xf3, 0xc8, 0x86, + 0x9f, 0x8c, 0x46, 0xee, 0xf1, 0x66, 0x9c, 0x63, 0x83, 0x52, 0x74, 0x0c, 0x0b, 0x61, 0x4c, 0xca, + 0x93, 0xa9, 0x96, 0x8e, 0x3d, 0x87, 0x48, 0xa6, 0x8c, 0xc3, 0x6e, 0x09, 0x31, 0x7a, 0x09, 0xe1, + 0x05, 0xf0, 0x96, 0x92, 0xf9, 0x9e, 0x36, 0x23, 0x70, 0x1f, 0x8d, 0xcd, 0x52, 0x71, 0xd8, 0x37, + 0x29, 0x47, 0x3b, 0x3c, 0x5c, 0x6c, 0x46, 0x82, 0x70, 0x99, 0x15, 0x98, 0x0f, 0x22, 0x30, 0x6f, + 0x93, 0x51, 0x1e, 0x38, 0x3d, 0x09, 0xda, 0x85, 0xac, 0xc4, 0x61, 0xae, 0xab, 0xf3, 0xc8, 0x86, 0xd1, 0x40, 0xa1, 0x22, 0xab, 0x80, 0xa4, 0x84, 0x47, 0x86, 0xdb, 0xd1, 0xa9, 0xa2, 0x7b, 0xa2, 0x1f, 0xc9, 0xc4, 0x46, 0xc6, 0x30, 0x2f, 0xc4, 0x59, 0x37, 0x2c, 0xe3, 0x87, 0x6c, 0x06, 0x04, - 0x51, 0x3f, 0x15, 0x0c, 0x51, 0x9b, 0x8b, 0x3d, 0xe4, 0x28, 0x2e, 0x89, 0x73, 0xe6, 0xa0, 0x14, - 0xed, 0xc1, 0xbc, 0x4c, 0x05, 0x54, 0x31, 0x43, 0x2d, 0x1b, 0xeb, 0xe1, 0x30, 0x83, 0xc4, 0xd9, + 0x51, 0x3f, 0x13, 0x0c, 0x51, 0x9b, 0x8b, 0x3d, 0xe4, 0x28, 0x2e, 0x89, 0x73, 0xe6, 0xa0, 0x14, + 0xed, 0xc3, 0xbc, 0x4c, 0x05, 0x54, 0x31, 0x43, 0x2d, 0x1b, 0xeb, 0xe1, 0x30, 0x83, 0xc4, 0xd9, 0x56, 0x58, 0xc6, 0x3d, 0x74, 0x5c, 0x8b, 0xe8, 0x7e, 0xff, 0x5d, 0x43, 0x9b, 0x8f, 0xf5, 0x30, 0xea, 0x05, 0x04, 0xe7, 0x9c, 0x41, 0x29, 0xfa, 0x14, 0x52, 0x1e, 0x71, 0x2c, 0x2d, 0x27, 0x70, - 0xee, 0x45, 0xe0, 0xf4, 0x38, 0x22, 0x16, 0x9a, 0x32, 0x83, 0x9c, 0x32, 0xbd, 0xc9, 0x59, 0x98, - 0x6e, 0x49, 0x1a, 0xa6, 0xe5, 0x47, 0x64, 0x90, 0x08, 0xc6, 0xc6, 0x33, 0xc8, 0xa0, 0x98, 0xdf, - 0xdc, 0x80, 0x40, 0x99, 0x3d, 0xfa, 0xa6, 0x2d, 0xc4, 0xde, 0xdc, 0x68, 0xaa, 0x87, 0x17, 0xe8, - 0x4d, 0xb9, 0x48, 0xa1, 0x0a, 0x39, 0xb8, 0x73, 0x28, 0x3e, 0x85, 0x0e, 0x51, 0x3b, 0x9c, 0xa5, - 0x61, 0xd9, 0xcd, 0x10, 0xa3, 0xa2, 0xa1, 0xd4, 0x96, 0xc6, 0x09, 0xb1, 0x50, 0xf3, 0x39, 0x10, - 0x62, 0x52, 0xce, 0x0f, 0xd8, 0x08, 0x88, 0xb5, 0x4e, 0x05, 0xb3, 0xd6, 0x96, 0x63, 0x0f, 0x38, - 0x8a, 0x83, 0xe3, 0x9c, 0x31, 0x28, 0xe5, 0x8b, 0x97, 0xe4, 0xb1, 0x5f, 0x3f, 0xee, 0xc6, 0x2e, - 0x7e, 0x98, 0x7c, 0xe2, 0xac, 0x17, 0x96, 0xa1, 0x6f, 0xe1, 0xae, 0x22, 0x93, 0x32, 0x65, 0xf1, - 0x4b, 0xc4, 0x43, 0x4f, 0x17, 0x44, 0x5a, 0xbb, 0x27, 0xa0, 0xbf, 0x13, 0x01, 0x1d, 0x47, 0x1a, - 0xb1, 0x76, 0x16, 0x47, 0x27, 0x8f, 0xe1, 0x36, 0x47, 0x56, 0xb9, 0x5c, 0x37, 0xbc, 0x4b, 0xc7, - 0x14, 0x9d, 0x43, 0x21, 0xb6, 0xa2, 0x46, 0x70, 0x37, 0x8c, 0x58, 0x37, 0xc8, 0xe4, 0x25, 0x0e, - 0x51, 0xa1, 0x94, 0xef, 0xb4, 0x2d, 0xf9, 0x99, 0xee, 0x2a, 0x82, 0xa6, 0x15, 0x63, 0x77, 0x3a, - 0x8a, 0xca, 0xe1, 0x9c, 0x3d, 0x28, 0xe5, 0x25, 0xc8, 0xef, 0xbf, 0x2b, 0xea, 0xaa, 0x55, 0xd3, - 0x56, 0x62, 0x4b, 0x50, 0xcc, 0x2b, 0x24, 0x46, 0xfe, 0xd0, 0x00, 0xaa, 0x42, 0xb6, 0xcd, 0xd9, - 0x9b, 0xee, 0x49, 0xfa, 0xa6, 0x3d, 0x88, 0x7d, 0xae, 0x1d, 0x62, 0x79, 0x78, 0xae, 0x1d, 0x12, - 0xa1, 0x63, 0xc8, 0xf7, 0x9a, 0x6e, 0xfd, 0x44, 0x90, 0x36, 0x6d, 0x55, 0xa0, 0xad, 0x45, 0xa0, - 0x8d, 0xe0, 0x78, 0x38, 0xc7, 0x06, 0xe5, 0xc8, 0x87, 0xfb, 0xe2, 0xc4, 0x24, 0x43, 0xd6, 0x49, - 0x9f, 0x22, 0xab, 0xfb, 0xf1, 0x50, 0xcc, 0xb3, 0x11, 0x7d, 0x72, 0xa3, 0x88, 0x35, 0x5e, 0x66, - 0xb1, 0x0a, 0x3c, 0xcd, 0xca, 0xa2, 0xc4, 0x19, 0x0d, 0x27, 0x78, 0xda, 0xc7, 0xb1, 0x57, 0x7c, - 0x98, 0x08, 0xe2, 0xac, 0x1d, 0x96, 0x3d, 0x9f, 0x79, 0xfb, 0xa6, 0x98, 0x50, 0xed, 0xee, 0x9d, - 0xfc, 0xf2, 0x8b, 0xd4, 0xcc, 0xfd, 0x7c, 0x61, 0x75, 0x5d, 0x10, 0xbd, 0x03, 0xd7, 0x13, 0xf9, - 0x1d, 0x2d, 0xc3, 0x94, 0xed, 0x58, 0xa4, 0xab, 0x3a, 0x3d, 0x49, 0x3c, 0xa5, 0x48, 0x51, 0xc3, - 0xdf, 0x27, 0x61, 0x6a, 0xbc, 0xbe, 0xf8, 0xc7, 0x83, 0xdc, 0x85, 0x12, 0xf1, 0x10, 0x2c, 0x48, - 0xd9, 0x7c, 0xe4, 0x1a, 0x06, 0x32, 0x8a, 0x50, 0x0e, 0x1e, 0xfc, 0xd8, 0xd0, 0x08, 0xda, 0x82, - 0xac, 0xef, 0x90, 0x6e, 0xc7, 0xf5, 0x88, 0x25, 0x8a, 0x64, 0x6a, 0x9c, 0x0e, 0x09, 0xcf, 0xf5, - 0x8c, 0x78, 0x71, 0x5c, 0x87, 0x8c, 0x4b, 0xed, 0xa6, 0xed, 0xe8, 0xbc, 0x80, 0x08, 0x5a, 0x35, - 0xb5, 0x39, 0xcf, 0xe7, 0x7c, 0x77, 0x55, 0x9c, 0xe6, 0xa5, 0xa6, 0x5a, 0xc6, 0x20, 0x55, 0xf8, - 0x3f, 0xf4, 0x19, 0x4c, 0x5b, 0x82, 0x12, 0x2b, 0xa2, 0x54, 0x88, 0xeb, 0x2b, 0x24, 0x71, 0xc6, - 0x4a, 0x1b, 0x7d, 0x2f, 0xd8, 0xd7, 0xf4, 0x28, 0xb3, 0xe0, 0x18, 0xd4, 0x8e, 0xa3, 0x67, 0x90, - 0x74, 0xdc, 0x0b, 0x45, 0x73, 0xc6, 0xea, 0x11, 0xb8, 0xfe, 0xf3, 0x99, 0x5f, 0xbe, 0x29, 0x4e, - 0xf4, 0x1f, 0x36, 0x9e, 0xfc, 0x2a, 0x71, 0xe3, 0xa9, 0xb0, 0xf7, 0xb0, 0x28, 0xdf, 0xe0, 0x1a, - 0xf8, 0x58, 0x1f, 0x7a, 0x9d, 0xfb, 0x08, 0x6e, 0xc9, 0x91, 0x97, 0xb8, 0xda, 0xa8, 0xe8, 0x8d, - 0x7a, 0x5d, 0xaf, 0xef, 0x95, 0xf3, 0x09, 0xb4, 0x04, 0x48, 0x0e, 0x94, 0x2b, 0x7b, 0x95, 0x46, - 0x45, 0xc7, 0xa5, 0xda, 0x4e, 0x25, 0x3f, 0xd9, 0x97, 0x1f, 0x56, 0x70, 0xb5, 0xb4, 0x57, 0xfd, - 0xba, 0xb4, 0xb9, 0x57, 0xc9, 0x27, 0xd1, 0x1d, 0xb8, 0x2d, 0xe5, 0x07, 0xf5, 0xc3, 0xc3, 0xea, - 0xe6, 0x5e, 0x45, 0xc7, 0x95, 0x83, 0xbd, 0xd2, 0x71, 0x3e, 0xb5, 0x9c, 0xfa, 0xd9, 0x6f, 0x0a, - 0x13, 0x4f, 0x9e, 0x03, 0x1a, 0x3e, 0x79, 0x34, 0x03, 0xa9, 0x5a, 0xbd, 0x56, 0xc9, 0x4f, 0xa0, - 0x0c, 0xa4, 0x37, 0x4b, 0x5b, 0xbb, 0xf5, 0xed, 0xed, 0x7c, 0x02, 0x65, 0x61, 0xb6, 0xba, 0xbf, - 0x5f, 0x29, 0x57, 0x4b, 0x8d, 0x4a, 0x7e, 0x72, 0xf3, 0xc1, 0xdb, 0x7f, 0x14, 0x26, 0xde, 0x5e, - 0x17, 0x12, 0x7f, 0xbe, 0x2e, 0x24, 0xfe, 0x72, 0x5d, 0x48, 0xfc, 0xfd, 0xba, 0x90, 0xf8, 0xf9, - 0x3f, 0x0b, 0x13, 0x5f, 0xa7, 0xd5, 0xbe, 0xfe, 0x37, 0x00, 0x00, 0xff, 0xff, 0xcf, 0x1b, 0x8c, - 0x2a, 0x19, 0x1b, 0x00, 0x00, + 0x3e, 0x8c, 0xc0, 0xe9, 0x71, 0x44, 0x2c, 0x34, 0x65, 0x06, 0x39, 0x63, 0x7a, 0x93, 0xb3, 0x30, + 0xdd, 0x92, 0x34, 0x4c, 0xcb, 0x8f, 0xc8, 0x20, 0x11, 0x8c, 0x8d, 0x67, 0x90, 0x41, 0x31, 0xbf, + 0xb9, 0x01, 0x81, 0x32, 0x7b, 0xf4, 0x4d, 0x5b, 0x88, 0xbd, 0xb9, 0xd1, 0x54, 0x0f, 0x2f, 0xd0, + 0xdb, 0x72, 0x91, 0x42, 0x15, 0x72, 0x70, 0xe7, 0x50, 0x7c, 0x0a, 0x1d, 0xa2, 0x76, 0x38, 0x4b, + 0xc3, 0xb2, 0xdb, 0x21, 0x46, 0x45, 0x43, 0xa9, 0x2d, 0x8d, 0x13, 0x62, 0xa1, 0xe6, 0x73, 0x20, + 0xc4, 0xa4, 0x9c, 0x1f, 0xb0, 0x11, 0x10, 0x6b, 0x9d, 0x0a, 0x66, 0xad, 0x2d, 0xc7, 0x1e, 0x70, + 0x14, 0x07, 0xc7, 0x39, 0x63, 0x50, 0xca, 0x17, 0x2f, 0xc9, 0x63, 0xbf, 0x7e, 0xdc, 0x8f, 0x5d, + 0xfc, 0x30, 0xf9, 0xc4, 0x59, 0x2f, 0x2c, 0x43, 0xdf, 0xc2, 0x7d, 0x45, 0x26, 0x65, 0xca, 0xe2, + 0x97, 0x88, 0x87, 0x9e, 0x2e, 0x88, 0xb4, 0xf6, 0xa1, 0x80, 0xfe, 0x4e, 0x04, 0x74, 0x1c, 0x69, + 0xc4, 0xda, 0x79, 0x1c, 0x9d, 0x3c, 0x81, 0xbb, 0x1c, 0x59, 0xe5, 0x72, 0xdd, 0xf0, 0xae, 0x1c, + 0x53, 0x74, 0x0e, 0x85, 0xd8, 0x8a, 0x1a, 0xc1, 0xdd, 0x30, 0x62, 0xdd, 0x20, 0x93, 0x97, 0x38, + 0x44, 0x85, 0x52, 0xbe, 0xd3, 0xb6, 0xe4, 0x67, 0xba, 0xab, 0x08, 0x9a, 0x56, 0x8c, 0xdd, 0xe9, + 0x28, 0x2a, 0x87, 0x73, 0xf6, 0xa0, 0x94, 0x97, 0x20, 0xbf, 0xff, 0xae, 0xa8, 0xab, 0x56, 0x4d, + 0x5b, 0x8d, 0x2d, 0x41, 0x31, 0xaf, 0x90, 0x18, 0xf9, 0x43, 0x03, 0xa8, 0x0a, 0xd9, 0x36, 0x67, + 0x6f, 0xba, 0x27, 0xe9, 0x9b, 0xf6, 0x51, 0xec, 0x73, 0xed, 0x10, 0xcb, 0xc3, 0x73, 0xed, 0x90, + 0x08, 0x9d, 0x40, 0xbe, 0xd7, 0x74, 0xeb, 0xa7, 0x82, 0xb4, 0x69, 0x6b, 0x02, 0x6d, 0x3d, 0x02, + 0x6d, 0x04, 0xc7, 0xc3, 0x39, 0x36, 0x28, 0x47, 0x3e, 0xac, 0x88, 0x13, 0x93, 0x0c, 0x59, 0x27, + 0x7d, 0x8a, 0xac, 0xee, 0xc7, 0x03, 0x31, 0xcf, 0x66, 0xf4, 0xc9, 0x8d, 0x22, 0xd6, 0x78, 0x99, + 0xc5, 0x2a, 0xf0, 0x34, 0x2b, 0x8b, 0x12, 0x67, 0x34, 0x9c, 0xe0, 0x69, 0x1f, 0xc7, 0x5e, 0xf1, + 0x61, 0x22, 0x88, 0xb3, 0x76, 0x58, 0xf6, 0x6c, 0xe6, 0xcd, 0xeb, 0x62, 0x42, 0xb5, 0xbb, 0xf7, + 0xf2, 0xcb, 0xcf, 0x53, 0x33, 0x2b, 0xf9, 0xc2, 0xda, 0x86, 0x20, 0x7a, 0x87, 0xae, 0x27, 0xf2, + 0x3b, 0x5a, 0x86, 0x29, 0xdb, 0xb1, 0x48, 0x57, 0x75, 0x7a, 0x92, 0x78, 0x4a, 0x91, 0xa2, 0x86, + 0xbf, 0x4b, 0xc2, 0xd4, 0x78, 0x7d, 0xf1, 0x8f, 0x07, 0xb9, 0x0b, 0x25, 0xe2, 0x21, 0x58, 0x90, + 0xb2, 0xf9, 0xc8, 0x35, 0x0c, 0x64, 0x14, 0xa1, 0x1c, 0x3c, 0xf8, 0xb1, 0xa1, 0x11, 0xb4, 0x0d, + 0x59, 0xdf, 0x21, 0xdd, 0x8e, 0xeb, 0x11, 0x4b, 0x14, 0xc9, 0xd4, 0x38, 0x1d, 0x12, 0x9e, 0xeb, + 0x19, 0xf1, 0xe2, 0xb8, 0x01, 0x19, 0x97, 0xda, 0x4d, 0xdb, 0xd1, 0x79, 0x01, 0x11, 0xb4, 0x6a, + 0x6a, 0x6b, 0x9e, 0xcf, 0xf9, 0xf6, 0xba, 0x38, 0xcd, 0x4b, 0x4d, 0xb5, 0x8c, 0x41, 0xaa, 0xf0, + 0x7f, 0xe8, 0x33, 0x98, 0xb6, 0x04, 0x25, 0x56, 0x44, 0xa9, 0x10, 0xd7, 0x57, 0x48, 0xe2, 0x8c, + 0x95, 0x36, 0xfa, 0x5e, 0xb0, 0xaf, 0xe9, 0x51, 0x66, 0xc1, 0x31, 0xa8, 0x1d, 0x47, 0x4f, 0x21, + 0xe9, 0xb8, 0x97, 0x8a, 0xe6, 0x8c, 0xd5, 0x23, 0x70, 0xfd, 0x67, 0x33, 0xbf, 0x7c, 0x5d, 0x9c, + 0xe8, 0x3f, 0x6c, 0x3c, 0xfe, 0x63, 0xe2, 0xd6, 0x53, 0x61, 0xef, 0x61, 0x51, 0xbe, 0xc1, 0x35, + 0xf0, 0x89, 0x3e, 0xf4, 0x3a, 0xf7, 0x01, 0xdc, 0x91, 0x23, 0x2f, 0x70, 0xb5, 0x51, 0xd1, 0x1b, + 0xf5, 0xba, 0x5e, 0xdf, 0x2f, 0xe7, 0x13, 0x68, 0x09, 0x90, 0x1c, 0x28, 0x57, 0xf6, 0x2b, 0x8d, + 0x8a, 0x8e, 0x4b, 0xb5, 0xdd, 0x4a, 0x7e, 0xb2, 0x2f, 0x3f, 0xaa, 0xe0, 0x6a, 0x69, 0xbf, 0xfa, + 0x75, 0x69, 0x6b, 0xbf, 0x92, 0x4f, 0xa2, 0x7b, 0x70, 0x57, 0xca, 0x0f, 0xeb, 0x47, 0x47, 0xd5, + 0xad, 0xfd, 0x8a, 0x8e, 0x2b, 0x87, 0xfb, 0xa5, 0x93, 0x7c, 0x0a, 0xad, 0xc0, 0x3d, 0x39, 0x54, + 0x3a, 0x3a, 0xa9, 0x6d, 0xab, 0x99, 0x76, 0x4a, 0xd5, 0xfd, 0x63, 0x5c, 0xc9, 0x4f, 0x2d, 0xa7, + 0x7e, 0xf6, 0xeb, 0xc2, 0xc4, 0xe3, 0x67, 0x80, 0x86, 0x2f, 0x06, 0x9a, 0x81, 0x54, 0xad, 0x5e, + 0xab, 0xe4, 0x27, 0x50, 0x06, 0xd2, 0x5b, 0xa5, 0xed, 0xbd, 0xfa, 0xce, 0x4e, 0x3e, 0x81, 0xb2, + 0x30, 0x5b, 0x3d, 0x38, 0xa8, 0x94, 0xab, 0xa5, 0x46, 0x25, 0x3f, 0xb9, 0xf5, 0xd1, 0x9b, 0x7f, + 0x14, 0x26, 0xde, 0xdc, 0x14, 0x12, 0x7f, 0xbe, 0x29, 0x24, 0xfe, 0x72, 0x53, 0x48, 0xfc, 0xfd, + 0xa6, 0x90, 0xf8, 0xf9, 0x3f, 0x0b, 0x13, 0x5f, 0xa7, 0xd5, 0xb6, 0xff, 0x37, 0x00, 0x00, 0xff, + 0xff, 0xf8, 0xc7, 0xcb, 0x05, 0x38, 0x1b, 0x00, 0x00, } diff --git a/pkg/roachpb/errors.proto b/pkg/roachpb/errors.proto index 0a9caa2bc63f..06c8175b45f0 100644 --- a/pkg/roachpb/errors.proto +++ b/pkg/roachpb/errors.proto @@ -145,6 +145,8 @@ enum TransactionRetryReason { // A possible replay caused by duplicate begin txn or out-of-order // txn sequence number. RETRY_POSSIBLE_REPLAY = 4; + // An asynchronous write was observed to have failed. + RETRY_ASYNC_WRITE_FAILURE = 5; } // A TransactionRetryError indicates that the transaction must be diff --git a/pkg/sql/logictest/testdata/logic_test/show_trace b/pkg/sql/logictest/testdata/logic_test/show_trace index 92d5eca79a23..6a3839770345 100644 --- a/pkg/sql/logictest/testdata/logic_test/show_trace +++ b/pkg/sql/logictest/testdata/logic_test/show_trace @@ -53,7 +53,7 @@ dist sender querying next range at /Table/SystemConfigSpan/Start dist sender r1: sending batch 2 CPut, 1 BeginTxn to (n1,s1):1 sql txn rows affected: 0 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 7 QueryIntent to (n1,s1):1 # More KV operations. @@ -88,7 +88,7 @@ dist sender querying next range at /Table/3/1/53/2/1 dist sender r1: sending batch 1 Get to (n1,s1):1 sql txn rows affected: 0 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 7 QueryIntent to (n1,s1):1 # We avoid using the full trace output, because that would make the # ensuing trace especially chatty, as it traces the index backfill at @@ -123,7 +123,7 @@ dist sender querying next range at /Table/3/1/54/2/1 dist sender r1: sending batch 1 Put to (n1,s1):1 sql txn rows affected: 0 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 9 QueryIntent to (n1,s1):1 statement ok SET tracing = on,kv,results; INSERT INTO t.kv(k, v) VALUES (1,2); SET tracing = off @@ -258,7 +258,7 @@ sql txn CPut /Table/55/1/...PK.../0 -> /TUPLE/1:1:Int/1/1:2:Int/2 sql txn fetched: /kv/primary/2/v -> /3 sql txn CPut /Table/55/1/...PK.../0 -> /TUPLE/1:1:Int/2/1:2:Int/3 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 2 CPut, 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 2 CPut, 1 EndTxn, 7 QueryIntent to (n1,s1):1 sql txn fast path completed sql txn rows affected: 2 @@ -277,7 +277,7 @@ dist sender querying next range at /Table/2/1/53/"kv2"/3/1 dist sender r1: sending batch 1 Get to (n1,s1):1 dist sender querying next range at /Table/3/1/55/2/1 dist sender r1: sending batch 1 Get to (n1,s1):1 -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 1 QueryIntent to (n1,s1):1 sql txn Scan /Table/55/{1-2} dist sender querying next range at /Table/55/1 dist sender r1: sending batch 1 Scan to (n1,s1):1 @@ -342,7 +342,7 @@ dist sender querying next range at /Table/SystemConfigSpan/Start dist sender r1: sending batch 1 Put, 1 BeginTxn to (n1,s1):1 sql txn rows affected: 0 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 6 QueryIntent to (n1,s1):1 statement ok SET tracing = on,kv,results; DELETE FROM t.kv; SET tracing = off @@ -403,7 +403,7 @@ dist sender querying next range at /Table/3/1/54/2/1 dist sender r1: sending batch 1 Put to (n1,s1):1 sql txn rows affected: 0 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 9 QueryIntent to (n1,s1):1 statement ok SET tracing = on,kv,results; DROP TABLE t.kv; SET tracing = off @@ -441,7 +441,7 @@ dist sender querying next range at /Table/SystemConfigSpan/Start dist sender r1: sending batch 1 Put, 1 BeginTxn to (n1,s1):1 sql txn rows affected: 0 dist sender querying next range at /Table/SystemConfigSpan/Start -dist sender r1: sending batch 1 EndTxn to (n1,s1):1 +dist sender r1: sending batch 1 EndTxn, 6 QueryIntent to (n1,s1):1 # Check that session tracing does not inhibit the fast path for inserts & # friends (the path resulting in 1PC transactions). diff --git a/pkg/storage/replica_command.go b/pkg/storage/replica_command.go index cb967f2fa00a..6187a8b650cd 100644 --- a/pkg/storage/replica_command.go +++ b/pkg/storage/replica_command.go @@ -471,15 +471,22 @@ func (r *Replica) AdminMerge( // a consistent view of the data from the right-hand range. If the merge // commits, we'll write this data to the left-hand range in the merge // trigger. - br, pErr := client.SendWrapped(ctx, r.store.DB().NonTransactionalSender(), - &roachpb.GetSnapshotForMergeRequest{ - RequestHeader: roachpb.RequestHeader{Key: rightDesc.StartKey.AsRawKey()}, - LeftRange: *origLeftDesc, - }) - if pErr != nil { - return pErr.GoError() + // + // We make sure to send this request through the txn so that we can stall + // its write pipeline and prevent any reads to the right-hand range when + // the EndTransactionRequest is issued. + // TODO(nvanbenschoten): this is subtle and it's not clear that its the + // best way to achieve this goal. Reconsider. + b = txn.NewBatch() + b.AddRawRequest(&roachpb.GetSnapshotForMergeRequest{ + RequestHeader: roachpb.RequestHeader{Key: rightDesc.StartKey.AsRawKey()}, + LeftRange: *origLeftDesc, + }) + if err := txn.Run(ctx, b); err != nil { + return err } - rhsSnapshotRes := br.(*roachpb.GetSnapshotForMergeResponse) + br := b.RawResponse() + rhsSnapshotRes := br.Responses[0].GetInner().(*roachpb.GetSnapshotForMergeResponse) // Successful subsume, so we're guaranteed that the right-hand range will // not serve another request unless this transaction aborts. End the