Add cloudfoundry module to metricbeat (#16671)

* Split up parts into different metricsets.

* Add value and container metricsets.

* Add documentation.

* Revert metricbeat.yml.

* Run make update.

* Fix documentation add changelog.

* Fix bad change in azure module.

* Run make update.

* Improve reference config.

* Document Kibana Space ID setting (#16622)

* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

* Remove the special pins type for `ca_sha256` (#16615)

We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.

* Remove feature.Describer and enhance Details (#16658)

* Make memqueue.Broker internal (now memqueue.broker) (#16667)

* [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584)

* [Metricbeat] Add vpc metricset for aws module (#16111)

* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

* Update cisco.asciidoc (#16434)

* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

* Fix wording in dev guide (#16497)

* Added NetBeat (#12612)

* Close files earlier when generating fields files (#16681)

Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.

* [Metricbeat] Add new module for Redis Enterprise (#16482)

Add metricsets: node, proxy

* Add doppler_address, uaa_address, rlp_address to config.reference.yml.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>

CHANGELOG.next.asciidoc

@@ -194,6 +194,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add filtering option for prometheus collector. {pull}16420[16420]
 - Add metricsets based on Ceph Manager Daemon to the `ceph` module. {issue}7723[7723] {pull}16254[16254]
 - Release `statsd` module as GA. {pull}16447[16447] {issue}14280[14280]
+- Add `cloudfoundry` module to send events from Cloud Foundry. {pull}16671[16671]
 - Add `redisenterprise` module. {pull}16482[16482] {issue}15269[15269]
 
 *Packetbeat*

metricbeat/docs/fields.asciidoc

@@ -22,6 +22,7 @@ grouped in the following categories:
 * <<exported-fields-beat>>
 * <<exported-fields-ceph>>
 * <<exported-fields-cloud>>
+* <<exported-fields-cloudfoundry>>
 * <<exported-fields-cockroachdb>>
 * <<exported-fields-common>>
 * <<exported-fields-consul>>
@@ -4598,6 +4599,177 @@ alias to: cloud.region
 
 --
 
+[[exported-fields-cloudfoundry]]
+== cloudfoundry fields
+
+Cloud Foundry module
+
+
+
+[float]
+=== cloudfoundry
+
+
+
+
+[float]
+=== app
+
+The application the metric is associated with.
+
+
+
+*`cloudfoundry.app.id`*::
++
+--
+The ID of the application.
+
+
+type: keyword
+
+--
+
+[float]
+=== container
+
+`container` contains container metrics from Cloud Foundry.
+
+
+
+*`cloudfoundry.container.instance_index`*::
++
+--
+Index of the instance the metric belongs to.
+
+
+type: long
+
+--
+
+*`cloudfoundry.container.cpu.pct`*::
++
+--
+CPU usage percentage.
+
+
+type: float
+
+--
+
+*`cloudfoundry.container.memory.bytes`*::
++
+--
+Bytes of used memory.
+
+
+type: long
+
+--
+
+*`cloudfoundry.container.memory.quota.bytes`*::
++
+--
+Bytes of available memory.
+
+
+type: long
+
+--
+
+*`cloudfoundry.container.disk.bytes`*::
++
+--
+Bytes of used storage.
+
+
+type: long
+
+--
+
+*`cloudfoundry.container.disk.quota.bytes`*::
++
+--
+Bytes of available storage.
+
+
+type: long
+
+--
+
+[float]
+=== counter
+
+`counter` contains counter metrics from Cloud Foundry.
+
+
+
+*`cloudfoundry.counter.name`*::
++
+--
+The name of the counter.
+
+
+type: keyword
+
+--
+
+*`cloudfoundry.counter.delta`*::
++
+--
+The difference between the last time the counter event occurred.
+
+
+type: long
+
+--
+
+*`cloudfoundry.counter.total`*::
++
+--
+The total value for the counter.
+
+
+type: long
+
+--
+
+[float]
+=== value
+
+`value` contains counter metrics from Cloud Foundry.
+
+
+
+*`cloudfoundry.value.name`*::
++
+--
+The name of the value.
+
+
+type: keyword
+
+--
+
+*`cloudfoundry.value.unit`*::
++
+--
+The unit of the value.
+
+
+type: keyword
+
+--
+
+*`cloudfoundry.value.value`*::
++
+--
+The value of the value.
+
+
+type: float
+
+--
+
 [[exported-fields-cockroachdb]]
 == CockroachDB fields
 

metricbeat/docs/modules/cloudfoundry.asciidoc

@@ -0,0 +1,151 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-module-cloudfoundry]]
+[role="xpack"]
+== cloudfoundry module
+
+experimental[]
+
+This is the cloudfoundry module.
+
+The Cloud Foundry module connects to Cloud Foundry loggregator to gather container, counter, and value metrics into a common data platform where it can be used for analysis, visualization, and alerting.
+
+
+The cloudfoundry module metrics are numerical values that describe some aspect of a system at a particular point in time. They are collected when pushed from the loggregator and are identified with a timestamp, a name, a value, and one or more defining labels.
+
+The cloudfoundry module mericsets are `container`, `counter` and `value`.
+
+[float]
+=== Module-specific configuration notes
+
+All metrics come from the Cloud Foundry loggregator API. The loggregator API authenticates through the Cloud Foundry UAA API.
+This requires that a new client be added to UAA with the correct permissions. This can be done using the `uaac` client.
+
+[source,bash]
+----
+$ export CLOUDFOUNDRY_CLIENT_ID=metricbeat
+$ export CLOUDFOUNDRY_CLIENT_SECRET=yoursecret
+$ uaac client add $CLOUDFOUNDRY_CLIENT_ID --name $CLOUDFOUNDRY_CLIENT_ID --secret $CLOUDFOUNDRY_CLIENT_SECRET --authorized_grant_types client_credentials,refresh_token --authorities doppler.firehose,cloud_controller.admin_read_only
+----
+
+Then configuration of the module needs to contain the created `client_id` and `client_secret`.
+
+[source,yaml]
+----
+- module: cloudfoundry
+  api_address: https://api.dev.cfdev.sh
+  client_id: "${CLOUDFOUNDRY_CLIENT_ID}"
+  client_secret: "${CLOUDFOUNDRY_CLIENT_SECRET}"
+  ssl:
+      verification_mode: none
+----
+
+
+[float]
+== Metricsets
+
+[float]
+=== `container`
+The container metricset of Cloud Foundry module allows you to collect container metrics that the
+loggregator sends to metricbeat.
+
+[float]
+=== `counter`
+The counter metricset of Cloud Foundry module allows you to collect counter metrics that the
+loggregator sends to metricbeat.
+
+[float]
+=== `value`
+The value metricset of Cloud Foundry module allows you to collect value metrics that the
+loggregator sends to metricbeat.
+
+
+[float]
+== Configuration options
+
+The `cloudfoundry` input supports the following configuration options.
+
+[float]
+=== `api_address`
+
+The URL of the Cloud Foundry API. Optional. Default: "http://api.bosh-lite.com".
+
+[float]
+=== `doppler_address`
+
+The URL of the Cloud Foundry Doppler Websocket. Optional. Default: "(value from ${api_address}/v2/info)".
+
+[float]
+=== `uaa_address`
+
+The URL of the Cloud Foundry UAA API. Optional. Default: "(value from ${api_address}/v2/info)".
+
+[float]
+=== `rlp_address`
+
+The URL of the Cloud Foundry RLP Gateway. Optional. Default: "(value from ${api_address}/v2/info)".
+
+[float]
+=== `client_id`
+
+Client ID to authenticate with Cloud Foundry. Default: "".
+
+[float]
+=== `client_secret`
+
+Client Secret to authenticate with Cloud Foundry. Default: "".
+
+[float]
+=== `shard_id`
+
+Shard ID for connection to the RLP Gateway. Use the same ID across multiple {beatname_lc} to shard the load of events
+from the RLP Gateway. Default: "(generated UUID)".
+
+[float]
+=== `ssl`
+
+This specifies SSL/TLS common config. Default: not used.
+
+
+[float]
+=== Example configuration
+
+The cloudfoundry module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+- module: cloudfoundry
+  metricsets:
+    - container
+    - counter
+    - value
+  enabled: true
+  api_address: '${CLOUDFOUNDRY_API_ADDRESS:""}'
+  doppler_address: '${CLOUDFOUNDRY_DOPPLER_ADDRESS:""}'
+  uaa_address: '${CLOUDFOUNDRY_UAA_ADDRESS:""}'
+  rlp_address: '${CLOUDFOUNDRY_RLP_ADDRESS:""}'
+  client_id: '${CLOUDFOUNDRY_CLIENT_ID:""}'
+  client_secret: '${CLOUDFOUNDRY_CLIENT_SECRET:""}'
+----
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-cloudfoundry-container,container>>
+
+* <<metricbeat-metricset-cloudfoundry-counter,counter>>
+
+* <<metricbeat-metricset-cloudfoundry-value,value>>
+
+include::cloudfoundry/container.asciidoc[]
+
+include::cloudfoundry/counter.asciidoc[]
+
+include::cloudfoundry/value.asciidoc[]
+

metricbeat/docs/modules/cloudfoundry/container.asciidoc

@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-cloudfoundry-container]]
+=== cloudfoundry container metricset
+
+experimental[]
+
+include::../../../../x-pack/metricbeat/module/cloudfoundry/container/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-cloudfoundry,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/cloudfoundry/container/_meta/data.json[]
+----

metricbeat/docs/modules/cloudfoundry/counter.asciidoc

@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-cloudfoundry-counter]]
+=== cloudfoundry counter metricset
+
+experimental[]
+
+include::../../../../x-pack/metricbeat/module/cloudfoundry/counter/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-cloudfoundry,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/cloudfoundry/counter/_meta/data.json[]
+----