running ch partition workload and meets error

[AkiraXie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. run ch list partition workload
2. check consistency between tiflash and tikv

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023/05/21 23:25:24.121 +08:00] [ERROR] [ch.go:256] ["check warehouse 2 at condition 3.3.2.5 failed exec SELECT count(*) FROM orders LEFT
JOIN new_order ON (no_w_id=o_w_id AND o_d_id=no_d_id AND o_id=no_o_id) where o_w_id = ? and ((o_carrier_id IS NULL and no_o_id IS NULL) O
R (o_carrier_id IS NOT NULL and no_o_id IS NOT NULL )) failed Error 1105: other error for mpp stream: From MPP<query:<query_ts:1684682723
943347087, local_query_id:77237, server_id:2001687, start_ts:441629467975614567>,task_id:4>: Code: 0, e.displayText() = DB::Exception: Code
: 0, e.displayText() = DB::Exception: code: 49\nmsg: "Check !filter || !filter->before_where || read_mode == ReadMode::Bitmap failed: Push
down filters needs bitmap: (while creating InputStreams from storage db_31987.t_34126, table_id: 34126)"\n, e.what() = DB::Exception,
"] [stack="github.com/pingcap/endless/testcase/tiflash/pkg/workload.(*ChWorkload).checkConsistence\n\t/home/jenkins/agent/workspace/endless
-master-build/testcase/tiflash/pkg/workload/ch.go:256[ngithub.com/pingcap/endless/testcase/tiflash/pkg/workload.(*ChWorkload).RunTP.func3\n](http://ngithub.com/pingcap/endless/testcase/tiflash/pkg/workload.(*ChWorkload).RunTP.func3/n)
\t/home/jenkins/agent/workspace/endless-master-build/testcase/tiflash/pkg/workload/ch.go:208[ngolang.org/x/sync/errgroup.(Group).Go.func1](http://ngolang.org/x/sync/errgroup.(*Group).Go.func1)\
n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"]
[2023/05/21 23:27:51.519 +08:00] [ERROR] [ch.go:256] ["check warehouse 7 at condition 3.3.2.10 failed exec SELECT count() \n\tFROM ( SELE
CT c.c_id, c.c_d_id, c.c_w_id, c.c_balance c1, \n\t\t\t\t (SELECT sum(ol_amount) FROM orders STRAIGHT_JOIN order_line \n\t\t\t\t\t WHERE
OL_W_ID=O_W_ID \n\t\t\t\t\t AND OL_D_ID = O_D_ID \n\t\t\t\t\t AND OL_O_ID = O_ID \n\t\t\t\t\t AND OL_DELIVERY_D IS NOT NULL \n\t\t\t
\t\t AND O_W_ID=? \n\t\t\t\t\t AND O_D_ID=c.C_D_ID \n\t\t\t\t\t AND O_C_ID=c.C_ID) sm, (SELECT sum(h_amount) from history \n\t\t\t
\t\t\t\t\t\t\t\t\tWHERE H_C_W_ID=? \n\t\t\t\t\t\t\t\t\t\t\t\t AND H_C_D_ID=c.C_D_ID \n\t\t\t\t\t\t\t\t\t\t\t\t AND H_C_ID=c.C_ID) smh \n
t\t\t FROM customer c \n\t\t\tWHERE c.c_w_id = ? ) t\n WHERE c1<>sm-smh failed Error 1105: other error for mpp stream: From MPP<query:<q
uery_ts:1684682861658032567, local_query_id:77264, server_id:2001687, start_ts:441629504072843366>,task_id:10>: Code: 0, e.displayText() =
DB::Exception: Code: 0, e.displayText() = DB::Exception: code: 49\nmsg: "Check !filter || !filter->before_where || read_mode == ReadMode::
Bitmap failed: Push down filters needs bitmap: (while creating InputStreams from storage db_31987.t_34636, table_id: 34636)"\n, e.what
() = DB::Exception,"] [stack="github.com/pingcap/endless/testcase/tiflash/pkg/workload.(*ChWorkload).checkConsistence\n\t/home/jenkins/agen
t/workspace/endless-master-build/testcase/tiflash/pkg/workload/ch.go:256\ngithub.com/pingcap/endless/testcase/tiflash/pkg/workload.(*ChWork
load).RunTP.func3\n\t/home/jenkins/agent/workspace/endless-master-build/testcase/tiflash/pkg/workload/ch.go:208\ngolang.org/x/sync/errgroup
.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"]

4. What is your TiFlash version? (Required)

sh-5.1# ./tiflash --version
TiFlash
Release Version: v7.1.0
Edition: Community
Git Commit Hash: 70200bc
Git Branch: heads/refs/tags/v7.1.0
UTC Build Time: 2023-05-12 23:23:30
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO

Raft Proxy
Git Commit Hash: 5cdf47726c58e04683c2da3e2892c276fccb3e8a
Git Commit Branch: HEAD
UTC Build Time: 2023-05-12 23:32:47
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: Unknown (env var does not exist when building)

[AkiraXie]

/severity major

[Lloyd-Pottiger]

[2023/05/22 21:24:22.019 +08:00] [Info] [FlashService.cpp:215] ["Begin process cop request after wait 0 ms, start ts: 441650213540593714, region info: 206338781, region epoch: conf_ver: 725\nversion: 35442\n"] [source=FlashService] [thread_id=6632]
[2023/05/22 21:24:22.019 +08:00] [Debug] [CoprocessorHandler.cpp:91] ["Handling DAG request: time_zone_offset: 28800\noutput_offsets: 0\noutput_offsets: 1\noutput_offsets: 2\noutput_offsets: 3\noutput_offsets: 4\nencode_type: TypeCHBlock\ntime_zone_name: \"Asia/Shanghai\"\ncollect_execution_summaries: false\nroot_executor {\n  tp: TypeSelection\n  selection {\n    conditions {\n      tp: ScalarFunc\n      children {\n        tp: ScalarFunc\n        children {\n          tp: ColumnRef\n          val: \"\\200\\000\\000\\000\\000\\000\\000\\003\"\n          sig: Unspecified\n          field_type {\n            tp: 3\n            flag: 0\n            flen: 11\n            decimal: 0\n            collate: -63\n            charset: \"binary\"\n          }\n          has_distinct: false\n        }\n        sig: IntIsNull\n        field_type {\n          tp: 8\n          flag: 524417\n          flen: 1\n          decimal: 0\n          collate: -63\n          charset: \"binary\"\n        }\n        has_distinct: false\n      }\n      sig: UnaryNotInt\n      field_type {\n        tp: 8\n        flag: 524416\n        flen: 1\n        decimal: 0\n        collate: -63\n        charset: \"binary\"\n      }\n      has_distinct: false\n    }\n    child {\n      tp: TypeTableScan\n      tbl_scan {\n        table_id: 38484\n        columns {\n          column_id: 1\n          tp: 3\n          collation: -63\n          columnLen: 11\n          decimal: 0\n          flag: 4099\n          pk_handle: false\n        }\n        columns {\n          column_id: 2\n          tp: 3\n          collation: -63\n          columnLen: 11\n          decimal: 0\n          flag: 4099\n          pk_handle: false\n        }\n        columns {\n          column_id: 3\n          tp: 3\n          collation: -63\n          columnLen: 11\n          decimal: 0\n          flag: 4107\n          pk_handle: false\n        }\n        columns {\n          column_id: 4\n          tp: 3\n          collation: -63\n          columnLen: 11\n          decimal: 0\n          flag: 0\n          pk_handle: false\n        }\n        columns {\n          column_id: 6\n          tp: 3\n          collation: -63\n          columnLen: 11\n          decimal: 0\n          flag: 0\n          pk_handle: false\n        }\n        desc: false\n        next_read_engine: Local\n        is_fast_scan: false\n        pushed_down_filter_conditions {\n          tp: ScalarFunc\n          children {\n            tp: ColumnRef\n            val: \"\\200\\000\\000\\000\\000\\000\\000\\004\"\n            sig: Unspecified\n            field_type {\n              tp: 3\n              flag: 0\n              flen: 11\n              decimal: 0\n              collate: -63\n              charset: \"binary\"\n            }\n            has_distinct: false\n          }\n          children {\n            tp: Int64\n            val: \"\\200\\000\\000\\000\\000\\000\\000\\010\"\n            sig: Unspecified\n            field_type {\n              tp: 8\n              flag: 129\n              flen: 1\n              decimal: 0\n              collate: -63\n              charset: \"binary\"\n            }\n            has_distinct: false\n          }\n          sig: GTInt\n          field_type {\n            tp: 8\n            flag: 524416\n            flen: 1\n            decimal: 0\n            collate: -63\n            charset: \"binary\"\n          }\n          has_distinct: false\n        }\n      }\n      executor_id: \"TableFullScan_47\"\n    }\n  }\n  executor_id: \"Selection_48\"\n}\nforce_encode_type: true\n"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.019 +08:00] [Debug] [executeQuery.cpp:386] ["(from 10.233.84.45:54794, query_id: 779f7438-0d7c-41c1-ba89-1cc8aaecf4fb) time_zone_offset: 28800 output_offsets: 0 output_offsets: 1 output_offsets: 2 output_offsets: 3 output_offsets: 4 encode_type: TypeCHBlock time_zone_name: \"Asia/Shanghai\" collect_execution_summaries: false root_executor {   tp: TypeSelection   selection {     conditions {       tp: ScalarFunc       children {         tp: ScalarFunc         children {           tp: ColumnRef           val: \"\\200\\000\\000\\000\\000\\000\\000\\003\"           sig: Unspecified           field_type {             tp: 3             flag: 0             flen: 11             decimal: 0             collate: -63             charset: \"binary\"           }           has_distinct: false         }         sig: IntIsNull         field_type {           tp: 8           flag: 524417           flen: 1           decimal: 0           collate: -63           charset: \"binary\"         }         has_distinct: false       }       sig: UnaryNotInt       field_type {         tp: 8         flag: 524416         flen: 1         decimal: 0         collate: -63         charset: \"binary\"       }       has_distinct: false     }     child {       tp: TypeTableScan       tbl_scan {         table_id: 38484         columns {           column_id: 1           tp: 3           collation: -63           columnLen: 11           decimal: 0           flag: 4099           pk_handle: false         }         columns {           column_id: 2           tp: 3           collation: -63           columnLen: 11           decimal: 0           flag: 4099           pk_handle: false         }         columns {           column_id: 3           tp: 3           collation: -63           columnLen: 11           decimal: 0           flag: 4107           pk_handle: false         }         columns {           column_id: 4           tp: 3           collation: -63           columnLen: 11           decimal: 0           flag: 0           pk_handle: false         }         columns {           column_id: 6           tp: 3           collation: -63           columnLen: 11           decimal: 0           flag: 0           pk_handle: false         }         desc: false         next_read_engine: Local         is_fast_scan: false         pushed_down_filter_conditions {           tp: ScalarFunc           children {             tp: ColumnRef             val: \"\\200\\000\\000\\000\\000\\000\\000\\004\"             sig: Unspecified             field_type {               tp: 3               flag: 0               flen: 11               decimal: 0               collate: -63               charset: \"binary\"             }             has_distinct: false           }           children {             tp: Int64             val: \"\\200\\000\\000\\000\\000\\000\\000\\010\"             sig: Unspecified             field_type {               tp: 8               flag: 129               flen: 1               decimal: 0               collate: -63               charset: \"binary\"             }             has_distinct: false           }           sig: GTInt           field_type {             tp: 8             flag: 524416             flen: 1             decimal: 0             collate: -63             charset: \"binary\"           }           has_distinct: false         }       }       executor_id: \"TableFullScan_47\"     }   }   executor_id: \"Selection_48\" } force_encode_type: true "] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.019 +08:00] [Debug] [PhysicalPlan.cpp:253] ["build unoptimized physical plan: \n<Projection, TableFullScan_47> | is_tidb_operator: false, schema: <TableFullScan_47_table_scan_0, Int32>, <TableFullScan_47_table_scan_1, Int32>, <TableFullScan_47_table_scan_2, Int32>, <TableFullScan_47_table_scan_3, Nullable(Int32)>, <TableFullScan_47_table_scan_4, Nullable(Int32)>\n <TableScan, TableFullScan_47> | is_tidb_operator: true, schema: <table_scan_0, Int32>, <table_scan_1, Int32>, <table_scan_2, Int32>, <table_scan_3, Nullable(Int32)>, <table_scan_4, Nullable(Int32)>\n"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.019 +08:00] [Debug] [PhysicalPlan.cpp:259] ["build optimized physical plan: \n<Projection, TableFullScan_47> | is_tidb_operator: false, schema: <TableFullScan_47_table_scan_0, Int32>, <TableFullScan_47_table_scan_1, Int32>, <TableFullScan_47_table_scan_2, Int32>, <TableFullScan_47_table_scan_3, Nullable(Int32)>, <TableFullScan_47_table_scan_4, Nullable(Int32)>\n <TableScan, TableFullScan_47> | is_tidb_operator: true, schema: <table_scan_0, Int32>, <table_scan_1, Int32>, <table_scan_2, Int32>, <table_scan_3, Nullable(Int32)>, <table_scan_4, Nullable(Int32)>\n"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [LearnerRead.cpp:285] ["Batch read index, original size 1, send & get 1 message, cost 3ms"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [Region.cpp:538] ["[region 206338781, applied: term 102 index 500509] need to wait learner index 500510"] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [Region.cpp:548] ["[region 206338781] wait learner index 500510 done"] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [LearnerRead.cpp:407] ["Finish wait index | resolve locks | check memory cache for 1 regions, cost 0ms, 0 unavailable regions"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [LearnerRead.cpp:431] ["[Learner Read] batch read index | wait index total_cost=4 read_cost=3 wait_cost=0 n_regions=1 n_stale_read=0 n_unavailable=0"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [DAGStorageInterpreter.cpp:980] ["Table 38484 schema OK, no syncing required. Schema version [storage, global, query]: [22969, 53618, 53618]"] [source=CoprocessorHandler] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [StorageDeltaMerge.cpp:652] ["Read with tso: 441650213540593714"] [source="db_38065.t_38484 CoprocessorHandler table_id=38484"] [thread_id=6632]
[2023/05/22 21:24:22.023 +08:00] [Debug] [StorageDeltaMerge.cpp:735] ["Rough set filter: {\"op\":\"and\",\"children\":[{\"op\":\"not\",\"children\":[{\"op\":\"isnull\",\"col\":\"o_c_id\"}]},{\"op\":\"greater\",\"col\":\"o_carrier_id\",\"value\":\"8\"}]}"] [source="db_38065.t_38484 CoprocessorHandler table_id=38484"] [thread_id=6632]
[2023/05/22 21:24:22.024 +08:00] [Debug] [StorageDeltaMerge.cpp:840] ["Push down filter: input:\no_id Int32\no_d_id Int32\no_w_id Int32\no_c_id Nullable(Int32)\no_carrier_id Nullable(Int32)\n\nactions:\nADD 8_Int64 Int64 Const(Int64)\nFUNCTION greater(o_carrier_id, 8_Int64)_collator_63  Nullable(UInt8) = greater(o_carrier_id, 8_Int64)\nREMOVE 8_Int64\n\noutput:\no_id Int32\no_d_id Int32\no_w_id Int32\no_c_id Nullable(Int32)\no_carrier_id Nullable(Int32)\ngreater(o_carrier_id, 8_Int64)_collator_63  Nullable(UInt8)\n"] [source="db_38065.t_38484 CoprocessorHandler table_id=38484"] [thread_id=6632]
[2023/05/22 21:24:22.024 +08:00] [Debug] [DeltaMergeStore.cpp:1770] ["[sorted_ranges: 1] [tasks before split: 1] [tasks final: 1] [ranges final: 1]"] [source="keyspace_id=4294967295 table_id=38484 CoprocessorHandler table_id=38484"] [thread_id=6632]
[2023/05/22 21:24:22.024 +08:00] [Info] [DeltaMergeStore.cpp:1063] ["Read create segment snapshot done, keep_order=true dt_enable_read_thread=true enable_read_thread=false, is_fast_scan=false, is_push_down_filter_empty=false"] [source="keyspace_id=4294967295 table_id=38484 CoprocessorHandler table_id=38484"] [thread_id=6632]
[2023/05/22 21:24:22.024 +08:00] [Debug] [DeltaMergeStore.cpp:1009] ["dt_enable_bitmap_filter: true, is_fast_scan: false, keep_order: true"] [thread_id=6632]
[2023/05/22 21:24:22.024 +08:00] [Error] [DAGDriver.cpp:209] ["DB Exception: Check !filter || !filter->before_where || read_mode == ReadMode::Bitmap failed: Push down filters needs bitmap, push down filters is empty: false, read mode: Normal: (while creating InputStreams from storage `db_38065`.`t_38484`, table_id: 38484)\n\n       0x1bfe3ee\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+29352942]\n                \tdbms/src/Common/Exception.h:46\n       0x73575f6\tDB::DM::DeltaMergeStore::getReadMode(DB::Context const&, bool, bool, std::__1::shared_ptr<DB::DM::PushDownFilter> const&) [tiflash+120944118]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1024\n       0x7357cd5\tDB::DM::DeltaMergeStore::read(DB::Context const&, DB::Settings const&, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, std::__1::vector<DB::DM::RowKeyRange, std::__1::allocator<DB::DM::RowKeyRange> > const&, unsigned long, unsigned long, std::__1::shared_ptr<DB::DM::PushDownFilter> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool, unsigned long, std::__1::unordered_set<unsigned long, std::__1::hash<unsigned long>, std::__1::equal_to<unsigned long>, std::__1::allocator<unsigned long> > const&, unsigned long, std::__1::shared_ptr<DB::DM::ScanContext>) [tiflash+120945877]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1072\n       0x7e627b9\tDB::StorageDeltaMerge::read(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, DB::SelectQueryInfo const&, DB::Context const&, DB::QueryProcessingStage::Enum&, unsigned long, unsigned int) [tiflash+132523961]\n                \tdbms/src/Storages/StorageDeltaMerge.cpp:919\n       0x81b37f2\tDB::DAGStorageInterpreter::buildLocalStreams(DB::DAGPipeline&, unsigned long) [tiflash+136001522]\n                \tdbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp:802\n       0x81ae31b\tDB::DAGStorageInterpreter::executeImpl(DB::DAGPipeline&) [tiflash+135979803]\n                \tdbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp:287\n       0x8344e4a\tDB::PhysicalTableScan::buildBlockInputStreamImpl(DB::DAGPipeline&, DB::Context&, unsigned long) [tiflash+137645642]\n                \tdbms/src/Flash/Planner/Plans/PhysicalTableScan.cpp:68\n       0x82e9006\tDB::PhysicalPlanNode::buildBlockInputStream(DB::DAGPipeline&, DB::Context&, unsigned long) [tiflash+137269254]\n                \tdbms/src/Flash/Planner/PhysicalPlanNode.cpp:83\n       0x8341bf6\tDB::PhysicalProjection::buildBlockInputStreamImpl(DB::DAGPipeline&, DB::Context&, unsigned long) [tiflash+137632758]\n                \tdbms/src/Flash/Planner/Plans/PhysicalProjection.cpp:143\n       0x82e9006\tDB::PhysicalPlanNode::buildBlockInputStream(DB::DAGPipeline&, DB::Context&, unsigned long) [tiflash+137269254]\n                \tdbms/src/Flash/Planner/PhysicalPlanNode.cpp:83\n       0x82f0a9e\tDB::Planner::execute() [tiflash+137300638]\n                \tdbms/src/Flash/Planner/Planner.cpp:36\n       0x8141239\tDB::(anonymous namespace)::doExecuteAsBlockIO(DB::IQuerySource&, DB::Context&, bool) [tiflash+135533113]\n                \tdbms/src/Flash/executeQuery.cpp:95\n       0x81403ef\tDB::queryExecute(DB::Context&, bool) [tiflash+135529455]\n                \tdbms/src/Flash/executeQuery.cpp:177\n       0x813b63e\tDB::DAGDriver<false>::execute() [tiflash+135509566]\n                \tdbms/src/Flash/Coprocessor/DAGDriver.cpp:94\n       0x80e5478\tDB::CoprocessorHandler::execute() [tiflash+135156856]\n                \tdbms/src/Flash/CoprocessorHandler.cpp:122\n       0x810a743\tstd::__1::__function::__func<DB::FlashService::Coprocessor(grpc::ServerContext*, coprocessor::Request const*, coprocessor::Response*)::$_6, std::__1::allocator<DB::FlashService::Coprocessor(grpc::ServerContext*, coprocessor::Request const*, coprocessor::Response*)::$_6>, grpc::Status ()>::operator()() [tiflash+135309123]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:345\n       0x8109358\tstd::__1::__packaged_task_func<std::__1::function<grpc::Status ()>, std::__1::allocator<std::__1::function<grpc::Status ()> >, grpc::Status ()>::operator()() [tiflash+135304024]\n                \t/usr/local/bin/../include/c++/v1/future:1687\n       0x8109515\tstd::__1::packaged_task<grpc::Status ()>::operator()() [tiflash+135304469]\n                \t/usr/local/bin/../include/c++/v1/future:1960\n       0x8675e54\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, legacy::ThreadPool::ThreadPool(unsigned long, std::__1::function<void ()>)::$_0> >(void*) [tiflash+140992084]\n                \t/usr/local/bin/../include/c++/v1/thread:291\n  0x7f0d65718802\tstart_thread [libc.so.6+653314]\n  0x7f0d656b8314\t__GI___clone [libc.so.6+258836]"] [source=DAGDriver] [thread_id=6632]
[2023/05/22 21:24:22.024 +08:00] [Debug] [CoprocessorHandler.cpp:124] ["Handle DAG request done"] [source=CoprocessorHandler] [thread_id=6632]

The root cause is keep_order in tablescan executor is missing. Then we will view it as true, but it has pushed_down_filters, which are conflict. (We have explicitly disable push down filter condition when keep_order is true in TiDB)