Add more metrics about heartbeat

[nolouch]

Feature Request

Describe your feature request related problem

We only have some metrics to see the ops about update cache or update kv, but cannot know the whole handle duration and the duration of each update operator. in the large cluster, about 200w regions, we meet the issue like #2783
we only have a method to profile and compare the flame graph.

Describe the feature you'd like

• Add a histogram about handle region heartbeat
• Add a histogram about handle store heartbeat
• Add a histogram about the update region cache.

Describe alternatives you've considered

we should consider the performance, the metrics are in the hot path, especially the region heartbeat.

Teachability, Documentation, Adoption, Migration Strategy

• Go

[lizhemingi]

I would like to give it a try if you don't mind.

[nolouch]

@duduainankai thanks, you can have a try.

[lizhemingi]

@nolouch Would you mind giving some tips where to start ? I am quite new to pd.

[nolouch]

you can read the function RegionHeartbeat in server/grpc_server.go

[nolouch]

Hi, @duduainankai. Do you have a try?

[lizhemingi]

@nolouch

Yep, I was trying. But I have some questions.

• In my understanding, I am supposed to add metric to record the time cost like in HandleRegionHeartbeat. (maybe also add a panel in grafana?) Correct me if I am wrong.

• Is there any easy ways for me to test locally ? (using tiup ?)

PS. Sorry for the slow response.

[lizhemingi]

In function RegionHeartBeat when rc.HandleRegionHeartbeat returns an error, sendErr will increase an "err" in regionHeartbeatCounter.

But then the for loop in RegionHeartBeat will also increase an "ok".

I think it's a bug ?