Implement metrics for thread pool suppliers

[tjquinno]

Resolves #947 for 2.x

Overview

Helidon implements three categories of thread pool supplier, depending mostly on the type of executor service they create, implemented in helidon-common-configurable:

• scheduled (ScheduleThreadPoolSupplier)
• server (ServerThreadPoolSupplier)
• general (ThreadPoolSupplier with no overriding category from the other suppliers)

Beginning with this PR, for each ExecutorService created from these thread pool suppliers Helidon will register Gauge metrics in the vendor registry. Gauges wrap values which other objects expose. Only when the gauge values are retrieved (either by code retrieving the metric from the registry and querying its value or by an HTTP client accessing the /metrics endpoint) are the underlying values read, so the performance drain is minimal.

Metrics exposed

The classic ExecutorService implementations created by Helidon's thread pool suppliers are ThreadPoolExecutor subclasses. ThreadPoolExecutor exposes several methods which reveal dynamic information about the executor, and Helidon will create these metrics for each one :

Method	Metric name
getActiveCount	active-count
getCompletedTaskCount	completed-task-count
getPoolSize	pool-size
getLargestPoolSize	largest-pool-size
getTaskCount	task-count
getQueue.remainingCapacity	queue.remaining-capacity
getQueue.size	queue.size

The Loom-based ThreadPerTaskExecutor exposes much less:

Method	Metric name
threadCount	thread-count

and so does the new metrics support for these executors.

Metric identity

Helidon identifies the new executor service metrics by:

• name (executor-service.xxx where xxx is the metric name from above) and,
• tags:
  • supplierCategory - either scheduled, server, some other user-supplied name if user-created, or helidon-thread-pool-i where i is an increasing integer (if the supplier name is otherwise unspecified)
  • supplierIndex - index of the supplier within a given category
  • poolIndex - index of the thread pool within the supplier

Affected Helidon components

helidon-common-configurable

The several thread pool suppliers in helidon-common-configurable now locate observers, using the service loading mechanism, which implement a new SPI interface. When the suppliers are constructed, and when they create new thread pools (as ExecutorServices), they notify the observers.

helidon-metrics

The full-featured metrics component adds an implementation of the new observer interface which registers metrics for each new thread pool created from the thread pool suppliers.

Because the ThreadPerTaskExecutor (Loom) is not present in all JDK versions we support, the mechanism for getting the metrics values is different. The existing code in helidon-common-configurable uses reflection to obtain the thread-per-task executors, and the metrics support also uses reflection also to create a gauge to expose the thread count for those executors.

helidon-microprofile-lra

In the course of working on this I found a bean that declared an observer on all app-scoped beans being destroyed--including itself which meant CDI created the bean again just to notify it of the shutdown. I changed the method to @PreDestroy.

docs

The metrics guides (SE and MP) already contained a very brief mention of "built-in metrics" in Helidon. They now discuss them a bit more (the predefined base metrics for the JVM, the KPI metrics, and now the thread pool ones).

Example metrics output

"executor-service.active-count;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 0,
    "executor-service.completed-task-count;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 0,
    "executor-service.largest-pool-size;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 5,
    "executor-service.pool-size;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 5,
    "executor-service.queue.remaining-capacity;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 10000,
    "executor-service.queue.size;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 0,
    "executor-service.task-count;poolIndex=0;supplierCategory=helidon-thread-pool-2;supplierIndex=0": 0,