[Metricbeat] Add Airflow module in xpack (#26220)

* [Metricbeat] Add Airflow module

Airflow module is a lightweight module based on Statsd module.
It filters, renames and adds metrics according to mappings defined
in the manifest.yml

CHANGELOG.next.asciidoc

@@ -134,6 +134,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Remove xpack enabled flag on ES, Logstash, Beats and Kibana {pull}24427[24427]
 - Adjust host fields to adopt new names from 1.9.0 ECS. {pull}24312[24312]
 - Add replicas.ready field to state_statefulset in Kubernetes module{pull}26088[26088]
+- Added `statsd.mappings` configuration for Statsd module {pull}26220[26220]
+- Added Airflow lightweight module {pull}26220[26220]
 - Add state_job metricset to Kubernetes module{pull}26479[26479]
 
 *Packetbeat*

metricbeat/docs/fields.asciidoc

@@ -14,6 +14,7 @@ grouped in the following categories:
 
 * <<exported-fields-activemq>>
 * <<exported-fields-aerospike>>
+* <<exported-fields-airflow>>
 * <<exported-fields-apache>>
 * <<exported-fields-appsearch>>
 * <<exported-fields-aws>>
@@ -843,6 +844,234 @@ type: boolean
 
 --
 
+[[exported-fields-airflow]]
+== Airflow fields
+
+Airflow module
+
+
+
+
+*`airflow.*.1m_rate`*::
++
+--
+Airflow 1m rate timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.5m_rate`*::
++
+--
+Airflow 5m rate timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.15m_rate`*::
++
+--
+Airflow 15 rate timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.count`*::
++
+--
+Airflow counters
+
+
+type: object
+
+--
+
+*`airflow.*.max`*::
++
+--
+Airflow max timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.mean_rate`*::
++
+--
+Airflow mean rate timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.mean`*::
++
+--
+Airflow mean timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.median`*::
++
+--
+Airflow median timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.min`*::
++
+--
+Airflow min timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.p75`*::
++
+--
+Airflow 75 percentile timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.p95`*::
++
+--
+Airflow 95 percentile timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.p99_9`*::
++
+--
+Airflow 99.9 percentile timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.p99`*::
++
+--
+Airflow 99 percentile timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.stddev`*::
++
+--
+Airflow standard deviation timers metric
+
+
+type: object
+
+--
+
+*`airflow.*.value`*::
++
+--
+Airflow gauges
+
+
+type: object
+
+--
+
+*`airflow.dag_file`*::
++
+--
+Airflow dag file metadata
+
+
+type: keyword
+
+--
+
+*`airflow.dag_id`*::
++
+--
+Airflow dag id metadata
+
+
+type: keyword
+
+--
+
+*`airflow.job_name`*::
++
+--
+Airflow job name metadata
+
+
+type: keyword
+
+--
+
+*`airflow.operator_name`*::
++
+--
+Airflow operator name metadata
+
+
+type: keyword
+
+--
+
+*`airflow.pool_name`*::
++
+--
+Airflow pool name metadata
+
+
+type: keyword
+
+--
+
+*`airflow.status`*::
++
+--
+Airflow status metadata
+
+
+type: keyword
+
+--
+
+*`airflow.task_id`*::
++
+--
+Airflow task id metadata
+
+
+type: keyword
+
+--
+
 [[exported-fields-apache]]
 == Apache fields
 

metricbeat/docs/modules/airflow.asciidoc

@@ -0,0 +1,60 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-module-airflow]]
+[role="xpack"]
+== Airflow module
+
+beta[]
+
+This module collects metrics from
+https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/metrics.html[Airflow metrics] running a
+statsd server where airflow will send metrics to. The default metricset is `statsd`.
+
+[float]
+=== Compatibility
+
+The Airflow module is tested with Airflow 2.1.0. It should work with version
+2.0.0 and later.
+
+[float]
+=== Usage
+The Airflow module requires <<metricbeat-module-statsd,Statsd>> to receive Statsd metrics. Refer to the link for instructions about how to use Statsd.
+
+Add the following lines to your Airflow configuration file e.g. `airflow.cfg` ensuring `statsd_prefix` is left empty and replace `%METRICBEAT_HOST%` with the address metricbeat is running:
+
+```
+[metrics]
+statsd_on = True
+statsd_host = %METRICBEAT_HOST%
+statsd_port = 8126
+statsd_prefix =
+```
+
+
+[float]
+=== Example configuration
+
+The Airflow module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+- module: airflow
+  host: "localhost"
+  port: "8126"
+  #ttl: "30s"
+  metricsets: [ 'statsd' ]
+----
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-airflow-statsd,statsd>>
+
+include::airflow/statsd.asciidoc[]
+

metricbeat/docs/modules/airflow/statsd.asciidoc

@@ -0,0 +1,25 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-airflow-statsd]]
+[role="xpack"]
+=== Airflow statsd metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/airflow/statsd/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-airflow,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/airflow/statsd/_meta/data.json[]
+----

metricbeat/docs/modules/statsd.asciidoc

@@ -33,7 +33,28 @@ The `statsd` module has these additional config options:
 Irrespective of the given ttl, metrics will be reported at least once.
 A ttl of zero means metrics will never expire.
 
-[float]
+*`statsd.mapping`*:: It defines how metrics will mapped from the original metric label to the event json.
+Here's an example configuration:
+[source,yaml]
+----
+statsd.mappings:
+  - metric: 'ti_failures' <1>
+    value:
+      field: task_failures <2>
+  - metric: '<job_name>_start' <1>
+    labels:
+      - attr: job_name <3>
+        field: job_name <4>
+    value:
+      field: started <2>
+----
+
+<1> `metric`, required: the label key of the metric in statsd, either as a exact match string
+or as a template with named label placeholder in the format `<label_placeholder>`
+<2> `value.field`, required: field name where to save the metric value in the event json
+<3> `label[].attr`, required when using named label placeholder: reference to the named label placeholder defined in `metric`
+<4> `label[].field`, required when using named label placeholder field name where to save the named label placeholder value from the template in the event json
+
 === Metricsets
 
 Currently, there is only `server` metricset in `statsd` module.

metricbeat/docs/modules_list.asciidoc

@@ -11,6 +11,8 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-metricset-activemq-topic,topic>>   
 |<<metricbeat-module-aerospike,Aerospike>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-aerospike-namespace,namespace>>   
+|<<metricbeat-module-airflow,Airflow>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
+.1+| .1+|  |<<metricbeat-metricset-airflow-statsd,statsd>> beta[]  
 |<<metricbeat-module-apache,Apache>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-apache-status,status>>   
 |<<metricbeat-module-appsearch,App Search>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
@@ -306,6 +308,7 @@ This file is generated! See scripts/mage/docs_collector.go
 
 include::modules/activemq.asciidoc[]
 include::modules/aerospike.asciidoc[]
+include::modules/airflow.asciidoc[]
 include::modules/apache.asciidoc[]
 include::modules/appsearch.asciidoc[]
 include::modules/aws.asciidoc[]