Some worker metrics are not correct

[dbw9580]

Alluxio Version:
2.8

Describe the bug

1. Worker.ActiveClients reports more clients than there actually are.
2. Cache hit/miss ratios are wrong, with reports showing nonsense values like 64400% (Cache Miss Percentage and Cache Hit Remote Percentage are wrong #16945)

To Reproduce
Perform I/O through Alluxio worker, and observe worker metrics.

Expected behavior
Worker.ActiveClients should reflect active clients that is currently reading or writing data on the worker.

Urgency
Normal

Are you planning to fix it
yes

Additional context
Add any other context about the problem here.

[dbw9580]

the metric is incremented twice when creating a block reader:

alluxio/core/server/worker/src/main/java/alluxio/worker/block/MonoBlockStore.java

Line 142 in bf8841d

DefaultBlockWorker.Metrics.WORKER_ACTIVE_CLIENTS.inc();

alluxio/core/server/worker/src/main/java/alluxio/worker/block/DefaultBlockWorker.java

Line 417 in 6bbc180

Metrics.WORKER_ACTIVE_CLIENTS.inc();

[jiacheliu3]

Two other metrics that are potentially relevant:

    "Cluster.ActiveRpcReadCount" : {
      "count" : -1
    },

    "Cluster.ActiveRpcWriteCount" : {
      "count" : -559
    },

[secfree]

@dbw9580 may I know if there is a ETA for the PR to resolve this issue? Our clusters have very big values of "Worker.ActiveClients", and we have backported #14088.

[dbw9580]

@secfree I'm not actively working on this, so feel free to come up with a PR to fix any metric you find incorrect.

For Worker.ActiveClients specifically, I had a little investigation here, but I suspect there're more subtle places where this metric is not correctly updated.

[secfree]

@dbw9580 thanks for your information.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.