Instrumentation PoC (WIP) #4216
Conversation
|
dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 9281bd6 ldap_mnesia_25 / ldap_mnesia / 9281bd6 pgsql_mnesia_25 / pgsql_mnesia / 9281bd6 ldap_mnesia_26 / ldap_mnesia / 9281bd6 pgsql_mnesia_26 / pgsql_mnesia / 9281bd6 internal_mnesia_26 / internal_mnesia / 9281bd6 dynamic_domains_mysql_redis_26 / mysql_redis / 9281bd6 mysql_redis_26 / mysql_redis / 9281bd6 dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 9281bd6 pgsql_cets_26 / pgsql_cets / 9281bd6 mssql_mnesia_26 / odbc_mssql_mnesia / 9281bd6 dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 9281bd6 elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 9281bd6 small_tests_25 / small_tests / 9281bd6 small_tests_26 / small_tests / 9281bd6 small_tests_26_arm64 / small_tests / 9281bd6 |
chrzaszcz
force-pushed
the
metrics-poc
branch
2 times, most recently
from
b3af1ca
to
afcc464
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## feature/instrument #4216 +/- ##
======================================================
+ Coverage 84.35% 84.40% +0.04%
======================================================
Files 552 556 +4
Lines 33507 33534 +27
======================================================
+ Hits 28266 28303 +37
+ Misses 5241 5231 -10 ☔ View full report in Codecov by Sentry. |
chrzaszcz
force-pushed
the
metrics-poc
branch
from
afcc464
to
558e32a
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
metrics-poc
branch
from
558e32a
to
d1f6cbf
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
metrics-poc
branch
2 times, most recently
from
7adcbca
to
4b53b92
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
metrics-poc
branch
from
4b53b92
to
2337033
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
metrics-poc
branch
from
2337033
to
5e74027
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
metrics-poc
branch
from
48b7209
to
417571f
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
metrics-poc
branch
3 times, most recently
from
1b717f2
to
22950b8
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Key requirements:
- Each event is identified by its name (atom) and a map of labels.
Possible labels (keys) are strictly limited to eliminate mistakes.
- Config passed to 'set_up' should contain generic specs of the
instrumentation, e.g. #{metrics => {time => histogram}} indicates
the need for a histogram metric based on the 'time' measurement, but
there is no reliance on particular metric backends. The config
should be extensible as well, and allow specifying the need for
instruments other than metrics. Particular handlers return a boolean
indicating if they are interested in handling this event or not.
- A duplicated call to 'set_up' should fail.
- The call to 'execute' should call all the attached handlers.
- The call to 'span' should execute the measured function F.
A measure callback MeasureF is then used to determine the resulting
measurements. Execution time is passed to MeasureF alongside the
result from F.
- A call to 'execute' or 'span' should fail if there was no 'setup'
for the provided event name and labels.
The goal is to have a PoC for a single module.
Functionality will be extended as needed.
- Only spiral and histogram metrics are supported.
- All metrics are mapped to 3-element lists:
[EventName, HostTypeLabel, MetricName]
This means that only host-type metrics are supported.
- There is no support for 'all_metrics_are_global'
- The dependency on mongoose_metrics will be removed in the future.
- 'spiral' metrics are implemented as counters, because Prometheus allows rate calculation for counters. - 'histogram' metrics are implemented as histograms with exponential buckets: 2^0, 2^1, ... 2^30. The last boundary is more than 10^9, which is e.g. ~18 minutes expressed in microseconds. Buckets could be made configurable later to save space and computation resources for small metrics, and to allow negative values.
Any mongoose_instrument_* modules are allowed for now, which technically allows mongoose_instrument_registry, but it shoudn't be a real issue. We can rename if needed. Intrumentation is configured globally (i.e. not per host type).
It returns a list of instrument specs, just like hooks/1 returns hooks. Instrumentation is set up before hooks, and torn down after them.
- Enable exometer and prometheus for now. It would be needed to pass tests. - In the future we would most likely only enable prometheus by default. - Prometheus HTTP endpoint is exposed by default.
Unify event names: mod_mam_pm_* and mod_mam_muc_* No functional changes other than metric renaming
All instrumentation is done with mongoose_instrument now.
The checks in mongoose_instrument would cause an exception.
chrzaszcz
force-pushed
the
metrics-poc
branch
from
2b1dea2
to
5f6ebe4
Compare
|
elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 5f6ebe4 small_tests_25 / small_tests / 5f6ebe4 small_tests_26 / small_tests / 5f6ebe4 small_tests_26_arm64 / small_tests / 5f6ebe4 ldap_mnesia_25 / ldap_mnesia / 5f6ebe4 dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 5f6ebe4 ldap_mnesia_26 / ldap_mnesia / 5f6ebe4 dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 5f6ebe4 dynamic_domains_mysql_redis_26 / mysql_redis / 5f6ebe4 internal_mnesia_26 / internal_mnesia / 5f6ebe4 pgsql_cets_26 / pgsql_cets / 5f6ebe4 dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 5f6ebe4 mysql_redis_26 / mysql_redis / 5f6ebe4 pgsql_mnesia_26 / pgsql_mnesia / 5f6ebe4 pgsql_mnesia_25 / pgsql_mnesia / 5f6ebe4 mssql_mnesia_26 / odbc_mssql_mnesia / 5f6ebe4 |
This PR initiates the instrumentation rework, which will be done int the
feature/instrumentbranch.Main goals of this PR:
mongoose_instrumentfor setting up and executing instrumentation in the code. It shouldn't be limited to metrics.gen_mod, just like we do with hooks.mongoose_instrument. I chosemod_mamto ensure that the new approach can handle complex cases.mongoose_metricsto have all tests passing.Note: Individual commits contain much more information.