use tokio threadpool and thread local metrics for readpool #4486

fredchenbj · 2019-04-05T06:24:08Z

What have you changed? (mandatory)

Before uses futures-cpupool to implement ReadPool, but tokio-threadpool is faster and more stable under high race condition or workload, so replace it to improve the performance for storage read and coprocessor request. Meanwhile, use thread local variable to replace context struct for metrics.

What are the type of the changes? (mandatory)

Improvement (change which is an improvement to an existing feature).

How has this PR been tested? (mandatory)

Unit tests, integration tests, and partial manual tests.

Does this PR affect documentation (docs) update? (mandatory)

No.

Does this PR affect tidb-ansible update? (mandatory)

No.

Refer to a related PR or issue link (optional)

No.

Benchmark result if necessary (optional)

From the pic above, under high wordload and stable qps, the p99 latency of read reduced about 14%, and the p999 latency reduced about 20%.

Add a few positive/negative examples (optional)

Signed-off-by: Breezewish <breezewish@pingcap.com>

Signed-off-by: fredchenbj <cfworking@163.com>

sre-bot · 2019-04-05T06:24:12Z

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

siddontang · 2019-04-05T07:27:10Z

src/coprocessor/endpoint.rs

                // Keep running stream producer
-                cpu_future.forget();
+                // cpu_future.forget();


PTAL @hicqu

You can remove the line directly. It's OK because spawn has polled it internally. BTW I prefer to write

self.read_pool.spawn(...)?; Ok(rx.then(|r| r.unwrap()))

to make it more clear that spawn returns a Result<()>.

siddontang · 2019-04-05T07:30:48Z

src/server/readpool/mod.rs

-        assert_eq!(rx.recv().unwrap(), Ok(7));
-        assert_eq!(rx.recv().unwrap(), Ok(4));
+        // the recv order maybe: "Ok(2)Ok(4)Ok(7)Ok(3)" or “Ok(2)Ok(3)Ok(4)Ok(7)” or “Ok(2)Ok(4)Ok(3)Ok(7)”
+        print!("{:?}", rx.recv().unwrap());


I think we should use assert_eq, not print

Before the recv order was certainly Ok(2)Ok(3)Ok(7)Ok(4), but now it's order changes every runs. So I am not sure whether it is a problem.

Then let's not check the recv order any more. Let's only check whether or not full is returned, since futurepool already has complete tests. This is a work-stealing pool and the scheduling order is not as predictable as the previous one.

siddontang · 2019-04-05T07:33:24Z

src/coprocessor/mod.rs

@@ -35,21 +35,21 @@ mod endpoint;
 mod error;
 pub mod local_metrics;
 mod metrics;
-mod readpool_context;
+mod read_pool_impl;


em, I prefer readpool_impl

siddontang · 2019-04-05T07:34:53Z

src/storage/read_pool_impl.rs

+    }
+
+    #[inline]
+    fn thread_local_flush(pd_sender: &FutureScheduler<PdTask>) {


I think using thread_local everywhere is long and redundant. If we want to represent thread_local, mostly, we can use tls instead.

Yeah, tls looks to be a good name!

siddontang · 2019-04-05T07:36:41Z

src/util/future_pool/mod.rs

+    }
+}
+
+/// Tried to trigger a tick in current thread.


Maybe a typo. It should be Tries

siddontang · 2019-04-05T07:38:08Z

Thanks @fredchenbj

It is a very cool feature.

siddontang · 2019-04-05T07:38:33Z

I think we should do more benchmarks

/cc @breeswish please help @fredchenbj do some

siddontang · 2019-04-05T07:42:52Z

after this, I even think we can remove another threadpool, we can use future::lazy to wrap the task and so we can unify the thread pool.

But we should also do the benchmark, IMO, tokio thread pool has a better performance than our thread pool @breeswish

Another thing is to support dynamically changing thread number in the pool, but we must be careful about this, because now we will collect thread metrics and use thread ID as a label value. Dynamic thread means we may send too many label values to Prometheus. So maybe for the thread pool, we can use thread name instead of thread ID. /cc @overvenus

breezewish

Thanks a lot! Mostly fine. How about the metrics? Have you checked that they are working as intended?

breezewish · 2019-04-05T12:53:35Z

src/coprocessor/endpoint.rs

-            .future_execute(priority, move |ctxd| {
-                tracker.attach_ctxd(ctxd);
+            .spawn_handle(priority, move || {
+                tracker.init_current_stage();


We can now mark state as initialized when tracker is built, so that this line doesn't need any more.

breezewish · 2019-04-05T12:56:34Z

components/test_coprocessor/src/fixture.rs

-        ReadPoolContext::new(pd_worker.scheduler())
-    });
+    let pool =
+        coprocessor::ReadPoolImpl::build_read_pool(read_pool_cfg, pd_worker.scheduler(), "cop-fix");


Is the name really important? I guess most of time default name should be enough because the rest of the usage are in tests.

For this one maybe Builder::build_for_test() is enough

breezewish · 2019-04-05T12:59:29Z

components/test_raftstore/src/server.rs

-                storage::ReadPoolContext::new(pd_worker.scheduler())
-            });
+        let pd_worker = FutureWorker::new("test-pd-worker");
+        let storage_read_pool = storage::ReadPoolImpl::build_read_pool(


you may try to replace it using Builder::build_for_test as well. It may work.

breezewish · 2019-04-05T13:01:14Z

src/coprocessor/endpoint.rs

-        let read_pool = ReadPool::new(
-            "readpool",
+
+        let read_pool = ReadPoolImpl::build_read_pool(


For this one maybe we can use Builder::from_config(..).build() (because we don't need on_tick or before_stop in this test). Similar for others.

breezewish · 2019-04-05T13:01:59Z

src/storage/read_pool_impl.rs

+    static LOCAL_KV_COMMAND_SCAN_DETAILS: RefCell<LocalIntCounterVec> =
+        RefCell::new(KV_COMMAND_SCAN_DETAILS.local());
+
+    static LOCAL_PD_SENDER: RefCell<Option<FutureScheduler<PdTask>>> =


Let's remove it since it is not used.

breezewish · 2019-04-05T13:08:23Z

src/coprocessor/read_pool_impl.rs

+        LOCAL_COPR_EXECUTOR_COUNT.with(|m| m.borrow_mut().flush());
+    }
+
+    pub fn collect(region_id: u64, type_str: &str, metrics: ExecutorMetrics) {


let's rename it to make it more clear. maybe.. thread_local_collect_executor_metrics?

breezewish · 2019-04-05T13:11:50Z

src/server/readpool/mod.rs

-    struct Context;
-
-    impl futurepool::Context for Context {}
-
    #[test]
    fn test_future_execute() {


let's rename it because no more "future_execute".

breezewish · 2019-04-05T13:15:43Z

src/server/readpool/mod.rs

-        assert_eq!(rx.recv().unwrap(), Ok(7));
-        assert_eq!(rx.recv().unwrap(), Ok(4));
+        // the recv order maybe: "Ok(2)Ok(4)Ok(7)Ok(3)" or “Ok(2)Ok(3)Ok(4)Ok(7)” or “Ok(2)Ok(4)Ok(3)Ok(7)”
+        print!("{:?}", rx.recv().unwrap());


Then let's not check the recv order any more. Let's only check whether or not full is returned, since futurepool already has complete tests. This is a work-stealing pool and the scheduling order is not as predictable as the previous one.

breezewish · 2019-04-05T13:16:39Z

src/storage/read_pool_impl.rs

+    }
+
+    #[inline]
+    fn thread_local_flush(pd_sender: &FutureScheduler<PdTask>) {


Yeah, tls looks to be a good name!

breezewish · 2019-04-05T13:17:02Z

src/util/future_pool/mod.rs

+    }
+}
+
+/// Tried to trigger a tick in current thread.


Maybe a typo. It should be Tries

Signed-off-by: fredchenbj <cfworking@163.com>

siddontang · 2019-04-08T05:33:55Z

src/coprocessor/readpool_impl.rs

+use crate::coprocessor::dag::executor::ExecutorMetrics;
+
+thread_local! {
+    pub static LOCAL_COPR_REQ_HISTOGRAM_VEC: RefCell<LocalHistogramVec> =


oh, we have so many metrics, it is better to use a structure to wrap all so we can only use one thread local var instead? /cc @breeswish

I'm fine with both, maybe not much difference.

em, maybe we can do a benchmark, one local struct vs multi local vars

ok, I will do a benchmark about this.

Signed-off-by: fredchenbj <cfworking@163.com>

breezewish

Good job! I'm fine with this PR, as long as the metrics are working as intended.

siddontang · 2019-04-09T13:15:48Z

@breeswish

please paste your benchmark results too

…-local-metrics-for-readpool

breezewish · 2019-04-11T04:51:04Z

/run-integration-tests

Signed-off-by: fredchenbj <cfworking@163.com>

siddontang · 2019-04-11T06:52:52Z

src/coprocessor/readpool_impl.rs

+use prometheus::local::*;
+
+use crate::coprocessor::dag::executor::ExecutorMetrics;
+pub struct TlsCop {


add a blank line here

siddontang · 2019-04-11T06:57:25Z

src/storage/readpool_impl.rs

+pub struct ReadPoolImpl;
+
+impl ReadPoolImpl {
+    #[inline]


seem we don't need inline here

siddontang · 2019-04-11T06:58:56Z

Thanks @fredchenbj

Great work!!!

PTAL @breeswish @hicqu

breezewish · 2019-04-11T07:27:42Z

/run-integration-tests

hicqu · 2019-04-11T12:30:15Z

src/bin/tikv-server.rs

-        ReadPool::new("store-read", &cfg.readpool.storage.build_config(), || {
-            storage::ReadPoolContext::new(pd_sender.clone())
-        });
+    let storage_read_pool = storage::ReadPoolImpl::build_read_pool(


Personally I prefer storage::ReadPool. Impl looks like a private thing.

ReadPool had been used, maybe use ReadPoolProducer. Is it ok?

Or ReadPoolContext? It just can build a ReadPool and handle some metrics. It's not a ReadPool indeed.

It just "derive"s the common ReadPool to create a specialized ReadPool that attached some name, some lifetime hook functions (like on_tick). That's why it was called ReadPoolImpl. Producer or Builder might not be a very good name because it will be confusing for functions like Producer:: tls_collect_executor_metrics.

Agree with Producer and Builder are not good enough. How about remove the struct?

src/server/readpool/mod.rs

hicqu · 2019-04-11T13:07:06Z

src/coprocessor/readpool_impl.rs

+    pub local_copr_rocksdb_perf_counter: RefCell<LocalIntCounterVec>,
+    local_copr_executor_count: RefCell<LocalIntCounterVec>,
+    local_copr_get_or_scan_count: RefCell<LocalIntCounterVec>,
+    local_cop_flow_stats: RefCell<HashMap<u64, crate::storage::FlowStatistics>>,


There are too many RefCells. How about put the struct in a RefCell or Mutex? I think it's more clear.

Nice catch. You should arrange them as..

pub struct Xxx { field: LocalIntCounter, field_2: LocalIntCounter, ... } thread_local! { pub static TLS_COP_METRICS: RefCell<TlsCop> = ...; }

In this way, we only need to check borrow once when updating multiple fields.

hicqu · 2019-04-11T13:16:09Z

Rest LGTM. Thank you very much!

Signed-off-by: fredchenbj <cfworking@163.com>

fredchenbj · 2019-04-12T03:09:14Z

Friendly ping @siddontang @breeswish @hicqu

Signed-off-by: fredchenbj <cfworking@163.com>

hicqu · 2019-04-12T05:21:27Z

LGTM.

Signed-off-by: fredchenbj <cfworking@163.com>

siddontang · 2019-04-12T06:54:59Z

PTAL @breeswish

Signed-off-by: fredchenbj <cfworking@163.com>

fredchenbj · 2019-04-14T02:29:21Z

ping @siddontang @breeswish @hicqu , please take a look.

siddontang

LGTM

PTAL @hicqu @breeswish

* *:use tokio-threadpool and thread local metrics in Storage Signed-off-by: Breezewish <breezewish@pingcap.com>

breezewish and others added 2 commits April 5, 2019 14:03

*:use tokio-threadpool and thread local metrics in Storage

4c7ed43

Signed-off-by: Breezewish <breezewish@pingcap.com>

*:modify coprocessor metrics and tests with tokio-thread

1459d46

Signed-off-by: fredchenbj <cfworking@163.com>

siddontang reviewed Apr 5, 2019

View reviewed changes

breezewish reviewed Apr 5, 2019

View reviewed changes

siddontang added component/performance Component: Performance contribution This PR is from a community contributor. type/enhancement The issue or PR belongs to an enhancement. labels Apr 6, 2019

fredbjer added 2 commits April 8, 2019 13:12

fixes to make code clear

5cd442c

Signed-off-by: fredchenbj <cfworking@163.com>

little fix

5951bde

Signed-off-by: fredchenbj <cfworking@163.com>

siddontang reviewed Apr 8, 2019

View reviewed changes

fix test thread panic

babcc8d

Signed-off-by: fredchenbj <cfworking@163.com>

breezewish previously approved these changes Apr 8, 2019

View reviewed changes

breezewish mentioned this pull request Apr 9, 2019

Replace Remaining CpuPool with FuturePool #2970

Open

Merge branch 'master' into fredchenbj/use-tokio-threadpool-and-thread…

ecfc2bc

…-local-metrics-for-readpool

fredchenbj dismissed breezewish’s stale review via ecfc2bc April 11, 2019 04:05

fredbjer added 2 commits April 11, 2019 13:15

use struct of thread local

32627be

Signed-off-by: fredchenbj <cfworking@163.com>

use struct for thread local storage metrics

6524c9b

Signed-off-by: fredchenbj <cfworking@163.com>

siddontang reviewed Apr 11, 2019

View reviewed changes

src/storage/readpool_impl.rs Outdated

pub struct ReadPoolImpl;

impl ReadPoolImpl {

#[inline]

Copy link

Contributor

siddontang Apr 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seem we don't need inline here

hicqu reviewed Apr 11, 2019

View reviewed changes

src/server/readpool/mod.rs Show resolved Hide resolved

hicqu reviewed Apr 11, 2019

View reviewed changes

fredbjer added 2 commits April 12, 2019 10:25

merge branch master to resolve confict

7c1272a

Signed-off-by: fredchenbj <cfworking@163.com>

use struct to reduce refcell

6709847

Signed-off-by: fredchenbj <cfworking@163.com>

fredchenbj dismissed breezewish’s stale review via 6709847 April 12, 2019 02:34

fredbjer added 3 commits April 12, 2019 12:26

remove ReadPoolImpl struct

3e54ffd

Signed-off-by: fredchenbj <cfworking@163.com>

little fix

df0395c

Signed-off-by: fredchenbj <cfworking@163.com>

merge master branch to resolve conflict

78ca37f

Signed-off-by: fredchenbj <cfworking@163.com>

hicqu previously approved these changes Apr 12, 2019

View reviewed changes

fix bug of a test case

84367b8

Signed-off-by: fredchenbj <cfworking@163.com>

fredchenbj dismissed hicqu’s stale review via 84367b8 April 12, 2019 06:51

fredbjer added 3 commits April 14, 2019 09:50

4th merge master to resolve conflicts

43da565

Signed-off-by: fredchenbj <cfworking@163.com>

modify new files copyright info

58051f5

Signed-off-by: fredchenbj <cfworking@163.com>

little fixes

4338a7c

Signed-off-by: fredchenbj <cfworking@163.com>

fredchenbj force-pushed the fredchenbj/use-tokio-threadpool-and-thread-local-metrics-for-readpool branch from 90c8280 to 4338a7c Compare April 14, 2019 02:16

siddontang approved these changes Apr 14, 2019

View reviewed changes

breezewish approved these changes Apr 14, 2019

View reviewed changes

siddontang merged commit 545e479 into tikv:master Apr 14, 2019

This was referenced Apr 25, 2019

Use thread local variables for all metrics #4565

Open

Fix read pool names #4588

Merged

sticnarf pushed a commit to sticnarf/tikv that referenced this pull request Oct 27, 2019

use tokio threadpool and thread local metrics for readpool (tikv#4486)

5eac6bd

* *:use tokio-threadpool and thread local metrics in Storage Signed-off-by: Breezewish <breezewish@pingcap.com>

fredchenbj mentioned this pull request Dec 18, 2019

Add @fredchenbj to engine-sig reviewer tikv/community#42

Merged

zhangjinpeng87 mentioned this pull request Mar 26, 2020

MAINTAINERS: add Fu Chen as TiKV maintainer #7259

Merged

use tokio threadpool and thread local metrics for readpool #4486

use tokio threadpool and thread local metrics for readpool #4486

Conversation

fredchenbj commented Apr 5, 2019

What have you changed? (mandatory)

What are the type of the changes? (mandatory)

How has this PR been tested? (mandatory)

Does this PR affect documentation (docs) update? (mandatory)

Does this PR affect tidb-ansible update? (mandatory)

Refer to a related PR or issue link (optional)

Benchmark result if necessary (optional)

Add a few positive/negative examples (optional)

sre-bot commented Apr 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siddontang commented Apr 5, 2019

siddontang commented Apr 5, 2019

siddontang commented Apr 5, 2019

breezewish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

breezewish left a comment

Choose a reason for hiding this comment

siddontang commented Apr 9, 2019

breezewish commented Apr 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siddontang commented Apr 11, 2019

breezewish commented Apr 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hicqu commented Apr 11, 2019

fredchenbj commented Apr 12, 2019

hicqu commented Apr 12, 2019

siddontang commented Apr 12, 2019

fredchenbj commented Apr 14, 2019

siddontang left a comment

Choose a reason for hiding this comment