br restore may cause tiflash crash

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tiup br:nightly restore full -u [172.16.4.204:2579](http://172.16.4.204:2579/) -s "s3://nfs/tiflash/tpcc-1k-partition-range-100-noflash?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%[2fminio.pingcap.net](http://2fminio.pingcap.net/)%3a9000&force-path-style=true"

2. What did you expect to see? (Required)

3. What did you see instead (Required)

TiFlash crash

[2022/03/16 19:55:57.054 +08:00] [ERROR] [Exception.cpp:78] ["DB::EngineStoreApplyRes DB::HandleIngestSST(DB::EngineStoreServerWrap *, DB::SSTViewVec, DB::RaftCmdHeader):Code: 49, e.displayText() = DB::Exception: Detected overflow when decoding integer of length 5 with column type Int32, e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x13) [0x57ec393]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)+0x26) [0x57e9996]\n2. bin/tiflash/tiflash(DB::ColumnVector<int>::decodeTiDBRowV2Datum(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool)+0xe1) [0x6339ab1]\n3. bin/tiflash/tiflash(DB::ColumnNullable::decodeTiDBRowV2Datum(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool)+0x17) [0xb1463f7]\n4. bin/tiflash/tiflash(bool DB::appendRowV2ToBlockImpl<false>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, DB::Block&, unsigned long, std::__1::vector<TiDB::ColumnInfo, std::__1::allocator<TiDB::ColumnInfo> > const&, long, bool)+0x987) [0xb4362d7]\n5. bin/tiflash/tiflash(DB::appendRowToBlock(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, DB::Block&, unsigned long, std::__1::vector<TiDB::ColumnInfo, std::__1::allocator<TiDB::ColumnInfo> > const&, long, bool)+0x3b) [0xb43320b]\n6. bin/tiflash/tiflash(bool DB::RegionBlockReader::readImpl<(DB::TMTPKType)0>(DB::Block&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > > const&, bool)+0x61d) [0xb40776d]\n7. bin/tiflash/tiflash(DB::GenRegionBlockDataWithSchema(std::__1::shared_ptr<DB::Region> const&, std::__1::shared_ptr<DB::DecodingStorageSchemaSnapshot const> const&, unsigned long, bool, DB::TMTContext&)+0x3c3) [0xb3e19c3]\n8. bin/tiflash/tiflash(DB::DM::SSTFilesToBlockInputStream::readCommitedBlock()+0x76) [0xb3a11e6]\n9. bin/tiflash/tiflash(DB::DM::SSTFilesToBlockInputStream::read()+0x3b7) [0xb3a0a37]\n10. bin/tiflash/tiflash(DB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&)+0x25) [0xb2b4bb5]\n11. bin/tiflash/tiflash(DB::DM::PKSquashingBlockInputStream<true>::read()+0x39) [0xb3a3de9]\n12. bin/tiflash/tiflash(DB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&)+0x25) [0xb2b4bb5]\n13. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock()+0x25) [0xb2dc5f5]\n14. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool)+0xf9) [0xb2daba9]\n15. bin/tiflash/tiflash(DB::DM::DMVersionFilterBlockInputStream<1>::read()+0x15) [0xb2c4705]\n16. bin/tiflash/tiflash(DB::DM::BoundedSSTFilesToBlockInputStream::read()+0x13) [0xb3a15d3]\n17. bin/tiflash/tiflash(DB::DM::SSTFilesToDTFilesOutputStream::write()+0x5b) [0xb3a5fcb]\n18. bin/tiflash/tiflash(DB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&)+0x2e6) [0xb3970e6]\n19. bin/tiflash/tiflash(DB::KVStore::handleIngestSSTByDTFile(std::__1::shared_ptr<DB::Region> const&, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&)+0x114) [0xb398a04]\n20. bin/tiflash/tiflash(DB::KVStore::handleIngestSST(unsigned long, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&)+0x3b0) [0xb398130]\n21. bin/tiflash/tiflash(HandleIngestSST+0x42) [0xb3e7a62]\n22. /data1/llhtest5/deploy/tiflash-9901/bin/tiflash/libtiflash_proxy.so(+0x1a2fcc8) [0x7fd64e336cc8]\n23. /data1/llhtest5/deploy/tiflash-9901/bin/tiflash/libtiflash_proxy.so(+0x1a35e00) [0x7fd64e33ce00]\n24. /data1/llhtest5/deploy/tiflash-9901/bin/tiflash/libtiflash_proxy.so(+0x13c9618) [0x7fd64dcd0618]\n25. /data1/llhtest5/deploy/tiflash-9901/bin/tiflash/libtiflash_proxy.so(+0x1403f71) [0x7fd64dd0af71]\n26. /data1/llhtest5/deploy/tiflash-9901/bin/tiflash/libtiflash_proxy.so(+0x143896f) [0x7fd64dd3f96f]\n27. /data1/llhtest5/deploy/tiflash-9901/bin/tiflash/libtiflash_proxy.so(+0x1b9c88b) [0x7fd64e4a388b]\n28. /lib64/libpthread.so.0(+0x7ea5) [0x7fd64bfe4ea5]\n29. /lib64/libc.so.6(clone+0x6d) [0x7fd64baf79fd]\n"] [thread_id=21]

4. What is your TiFlash version? (Required)

master

[lidezhu]

mysql> set @@session.tidb_isolation_read_engines = "tikv";
Query OK, 0 rows affected (0.00 sec)

mysql> select count() from test.t;
+----------+
| count() |
+----------+
| 16385 |
+----------+
1 row in set (0.02 sec)

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> desc t;
+-------+---------+------+------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+------+---------+-------+
| a | int(11) | YES | | NULL | |
+-------+---------+------+------+---------+-------+
1 row in set (0.00 sec)

mysql> select a from test.t;
ERROR 1105 (HY000): invalid data type: Failed to decode row v2 data as i64
mysql> select a from test.t limit 1;
ERROR 1105 (HY000): invalid data type: Failed to decode datum flag
mysql> explain select a from test.t limit 1;
+---------------------------+---------+-----------+---------------+-------------------+
| id | estRows | task | access object | operator info |
+---------------------------+---------+-----------+---------------+-------------------+
| Limit_7 | 1.00 | root | | offset:0, count:1 |
| └─TableReader_11 | 1.00 | root | | data:Limit_10 |
| └─Limit_10 | 1.00 | cop[tikv] | | offset:0, count:1 |
| └─TableFullScan_9 | 1.00 | cop[tikv] | table:t | keep order:false |
+---------------------------+---------+-----------+---------------+-------------------+
4 rows in set (0.00 sec)

[lidezhu]

It seems that the data in test.t is corrupted and tikv cannot read from it. So tiflash cannot parse the data and it crashes. And this is not a problem with tiflash.