Backport to 6.x: Multiple doc changes (#7696 #7883 #7888 #7889 #7903 #…

…7937 #7939 #7948 #7949 #7950)  (#8188)

* Reset modulename attribute at the end of a file (#7888)

* Add safeguard related statements for max_backoff setting (#7889)

* Add docs about append_fields (#7903)

* Add docs about append_fields
* Remove experimental tag from setup.template.json.enabled description

* Metricbeat: Add compatibility notes where missing (#7939)

* Metricbeat: Add compatibility notes where missing, based on integration tests

* Added review suggestions

* Add missing config options to Logstash section of reference.yml (#7883)

Closes #3062

* Fix processor autodiscovery docs for Filebeat (#7937)

{ needs escaping as otherwise asciidoc will show an empty code block.

* Add document for beat export dashboard (#7696)

* Add document for beat export dashboard

Follow up from #7239

* move to command reference

* address review comments

* review comments applied

* apply review feedback

* Replace golang with Go (#7948)

I've been told that using "golang" instead of "Go" is considered a faux pas in some circles. :-)

I've replaced instances in the external docs, but I did not update the changelog, readmes, or code files.

* Minor fixes to attributes in module docs (#7949)

* Add recommendation to avoid harvesting symlinks when dealing with file rotation (#7950)

auditbeat/auditbeat.reference.yml

@@ -524,6 +524,21 @@ output.elasticsearch:
   # never, once, and freely. Default is never.
   #ssl.renegotiation: never
 
+  # The number of times to retry publishing an event after a publishing failure.
+  # After the specified number of retries, the events are typically dropped.
+  # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
+  # and retry until all events are published.  Set max_retries to a value less
+  # than 0 to retry until all events are published. The default is 3.
+  #max_retries: 3
+
+  # The maximum number of events to bulk in a single Logstash request. The
+  # default is 2048.
+  #bulk_max_size: 2048
+
+  # The number of seconds to wait for responses from the Logstash server before
+  # timing out. The default is 30s.
+  #timeout: 30s
+
 #------------------------------- Kafka output ----------------------------------
 #output.kafka:
   # Boolean flag to enable or disable the output module.
@@ -935,6 +950,13 @@ output.elasticsearch:
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
 
+# A list of fields to be added to the template and Kibana index pattern. Also
+# specify setup.template.overwrite: true to overwrite the existing template.
+# This setting is experimental.
+#setup.template.append_fields:
+#- name: field_name
+#  type: field_type
+
 # Enable json template loading. If this is enabled, the fields.yml is ignored.
 #setup.template.json.enabled: false
 

docs/devguide/contributing.asciidoc

@@ -45,8 +45,8 @@ Beats].
 === Setting Up Your Dev Environment
 
 The Beats are Go programs, so install the latest version of
-http://golang.org/[golang] if you don't have it already. The current Go version
-used for development is Golang {go-version}.
+http://golang.org/[Go] if you don't have it already. The current Go version
+used for development is Go {go-version}.
 
 The location where you clone is important. Please clone under the source
 directory of your `GOPATH`. If you don't have `GOPATH` already set, you can

docs/devguide/create-metricset.asciidoc

@@ -69,13 +69,13 @@ https://github.com/elastic/beats/blob/master/metricbeat/scripts/module/metricset
 include::../../metricbeat/scripts/module/metricset/metricset.go.tmpl[]
 ----
 
-The `package` clause and `import` declaration are part of the base structure of each Golang file. You should only
+The `package` clause and `import` declaration are part of the base structure of each Go file. You should only
 modify this part of the file if your implementation requires more imports.
 
 [float]
 ===== Initialisation
 
-The init method registers the metricset with the central registry. In Golang the `init()` function is called
+The init method registers the metricset with the central registry. In Go the `init()` function is called
 before the execution of all other code. This means the module will be automatically registered with the global registry.
 
 The `New` method, which is passed to `AddMetricSet`, will be called after the setup of the module and before starting to fetch data. You normally don't need to change this part of the file.

docs/devguide/creating-beat-from-metricbeat.asciidoc

@@ -7,7 +7,7 @@ own metricsets.
 [float]
 ==== Requirements
 
-To create your own Beat, you must have Golang {go-version} or later installed, and the `$GOPATH`
+To create your own Beat, you must have Go {go-version} or later installed, and the `$GOPATH`
 must be set up correctly. In addition, the following tools are required:
 
 * https://www.python.org/downloads/[python]

docs/devguide/metricset-details.asciidoc

@@ -145,8 +145,8 @@ It's important to also add tests for your metricset. There are three different t
 * system tests
 
 We recommend that you use all three when you create a metricset. Unit tests are
-written in Golang and have no dependencies. Integration tests are also written
-in Golang but require the service from which the module collects metrics to also be running.
+written in Go and have no dependencies. Integration tests are also written
+in Go but require the service from which the module collects metrics to also be running.
 System tests for Metricbeat also require the service to be running in most cases and are
 written in Python based on our small Python test framework.
 We use `virtualenv` to deal with Python dependencies.

docs/devguide/modules-dev-guide.asciidoc

@@ -260,7 +260,7 @@ input configuration are documented in the
 the Filebeat documentation.
 
 The template files use the templating language defined by the
-https://golang.org/pkg/text/template/[Golang standard library].
+https://golang.org/pkg/text/template/[Go standard library].
 
 Here is another example that also configures multiline stitching:
 

docs/devguide/newbeat.asciidoc

@@ -77,7 +77,7 @@ daemonzing, and Windows service handling, and data processing modules.
 
 image:./images/beat_overview.png[Beat overview architecture]
 
-The event that you create is a JSON-like object (Golang type `map[string]interface{}`) that
+The event that you create is a JSON-like object (Go type `map[string]interface{}`) that
 contains the collected data to send to the publisher. At a minimum, the event object
 must contain a `@timestamp` field and a `type` field. Beyond
 that, events can contain any additional fields, and they can be created as often

docs/devguide/newdashboards.asciidoc

@@ -208,7 +208,7 @@ make update
 === Exporting New and Modified Beat Dashboards
 
 To export all the dashboards for any Elastic Beat or any community Beat, including any new or modified dashboards and all dependencies such as
-visualizations, searches, you can use the Golang script `export_dashboards.go` from
+visualizations, searches, you can use the Go script `export_dashboards.go` from
 https://github.com/elastic/beats/tree/master/dev-tools/cmd/dashboards[dev-tools] for exporting Kibana 6.0 dashboards or later, and the Python script `export_5x_dashboards.py`
 for exporting Kibana 5.x dashboards. See the dev-tools
 https://github.com/elastic/beats/tree/master/dev-tools/README.md[readme] for more info.

docs/devguide/testing.asciidoc

@@ -5,20 +5,20 @@ Beats has a various sets of tests. This guide should help to understand how the
 
 In general there are two major test suites:
 
-* Tests written in Golang
+* Tests written in Go
 * Tests written in Python
 
-The tests written in Golang use the https://golang.org/pkg/testing/[Golang Testing package]. The tests written in Python depend on http://nose.readthedocs.io/en/latest/[nosetests] and require a compiled and executable binary from the Golang code. The python test run a beat with a specific config and params and either check if the output is as expected or if the correct things show up in the logs.
+The tests written in Go use the https://golang.org/pkg/testing/[Go Testing package]. The tests written in Python depend on http://nose.readthedocs.io/en/latest/[nosetests] and require a compiled and executable binary from the Go code. The python test run a beat with a specific config and params and either check if the output is as expected or if the correct things show up in the logs.
 
 For both of the above test suites so called integration tests exists. Integration tests in Beats are tests which require an external system like Elasticsearch to test if the integration with this service works as expected. Beats provides in its testsuite docker containers and docker-compose files to start these environments but a developer can run the required services also locally.
 
-==== Running Golang Tests
+==== Running Go Tests
 
-The golang tests can be executed in each Golang package by running `go test .`. This will execute all tests which don't don't require an external service to be running. To also run the Golang integration tests run `go test -tags=integration .`. It will require you to run the expected services on localhost.
+The Go tests can be executed in each Go package by running `go test .`. This will execute all tests which don't don't require an external service to be running. To also run the Go integration tests run `go test -tags=integration .`. It will require you to run the expected services on localhost.
 
 To run all non integration tests for a beat run `make unit`. This will execute all the tests which are not inside a `vendor` directory. If you want to have a coverage report for the tests which were run use `make unit-tests`. A coverage report will be generated under `build/coverage` directory.
 
-All Golang tests are in the same package as the tested code itself and have the postfix `_test` in the file name. Most of the tests are in the same package as the rest of the code. Some of the tests which should be separate from the rest of the code or should not use private variables go under `{packagename}_test`.
+All Go tests are in the same package as the tested code itself and have the postfix `_test` in the file name. Most of the tests are in the same package as the rest of the code. Some of the tests which should be separate from the rest of the code or should not use private variables go under `{packagename}_test`.
 
 
 ==== Running Python Tests
@@ -37,10 +37,10 @@ All Python tests are under `tests/system` directory.
 
 This is a quick summary of the available test commands:
 
-* `unit`: Golang tests
-* `unit-tests`: Golang tests with coverage reports
-* `integration-tests`: Golang tests with services in local docker
-* `integration-tests-environment`: Golang tests inside docker with service in docker
+* `unit`: Go tests
+* `unit-tests`: Go tests with coverage reports
+* `integration-tests`: Go tests with services in local docker
+* `integration-tests-environment`: Go tests inside docker with service in docker
 * `fast-system-tests`: Python tests
 * `system-tests`: Python tests with coverage report
 * `INTEGRATION_TESTS=1 system-tests`: Python tests with local services
@@ -50,7 +50,7 @@ This is a quick summary of the available test commands:
 
 There are two experimental test commands:
 
-* `benchmark-tests`: Running golang tests with `-bench` flag
+* `benchmark-tests`: Running Go tests with `-bench` flag
 * `load-tests`: Running system tests with `LOAD_TESTS=1` flag
 
 
@@ -60,7 +60,7 @@ If the tests were run to create a test coverage, the coverage report files can b
 
 ==== Race detection
 
-All tests can be run with the Golang race detector enabled by setting the environment variable `RACE_DETECTOR=1`. This applies to tests in Golang and Python. For Python the test binary has to be recompile when the flag is changed. Having the race detection enabled will slow down the tests.
+All tests can be run with the Go race detector enabled by setting the environment variable `RACE_DETECTOR=1`. This applies to tests in Go and Python. For Python the test binary has to be recompile when the flag is changed. Having the race detection enabled will slow down the tests.
 
 ==== Docker environment
 

filebeat/docs/autodiscover-hints.asciidoc

@@ -73,7 +73,7 @@ of supported processors.
 In order to provide ordering of the processor definition, numbers can be provided. If not, the hints builder will do
 arbitrary ordering:
 
-["source","yaml",subs="attributes"]
+["source","yaml"]
 -------------------------------------------------------------------------------------
 co.elastic.logs/processors.1.dissect.tokenizer: "%{key1} %{key2}"
 co.elastic.logs/processors.dissect.tokenizer: "%{key2} %{key1}"

filebeat/docs/getting-started.asciidoc

@@ -166,7 +166,7 @@ filebeat.inputs:
 +
 The input in this example harvests all files in the path `/var/log/*.log`, which means
 that Filebeat will harvest all files in the directory `/var/log/` that end with `.log`. All patterns supported
-by https://golang.org/pkg/path/filepath/#Glob[Golang Glob] are also supported here.
+by https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also supported here.
 +
 To fetch all files from a predefined level of subdirectories, the following pattern can be used:
 `/var/log/*/*.log`. This fetches all `.log` files from the subfolders of `/var/log`. It does not

filebeat/docs/inputs/input-common-file-options.asciidoc

@@ -348,9 +348,10 @@ specifying 10s for `max_backoff` means that, at the worst, a new line could be
 added to the log file if {beatname_uc} has backed off multiple times. The
 default is 10s.
 
-Requirement: max_backoff should always be set to `max_backoff <=
-scan_frequency`. In case `max_backoff` should be bigger, it is recommended to
-close the file handler instead let the {beatname_uc} pick up the file again.
+Requirement: Set `max_backoff` to be greater than or equal to `backoff` and
+less than or equal to `scan_frequency` (`backoff <= max_backoff <= scan_frequency`).
+If `max_backoff` needs to be higher, it is recommended to close the file handler
+instead and let {beatname_uc} pick up the file again.
 
 [float]
 ===== `backoff_factor`

filebeat/docs/inputs/input-log.asciidoc

@@ -7,10 +7,10 @@
 <titleabbrev>Log</titleabbrev>
 ++++
 
-Use the `log` input to read lines from log files. 
+Use the `log` input to read lines from log files.
 
 To configure this input, specify a list of glob-based <<input-paths,`paths`>>
-that must be crawled to locate and fetch the log lines. 
+that must be crawled to locate and fetch the log lines.
 
 Example configuration:
 
@@ -57,6 +57,10 @@ multiple input sections:
 IMPORTANT: Make sure a file is not defined more than once across all inputs
 because this can lead to unexpected behaviour.
 
+NOTE: When dealing with file rotation, avoid harvesting symlinks. Instead
+use the <<input-paths>> setting to point to the original file, and specify
+a pattern that matches the file you want to harvest and all of its rotated
+files.  
 
 [id="{beatname_lc}-input-{type}-options"]
 ==== Configuration options
@@ -69,7 +73,7 @@ The `log` input supports the following configuration options plus the
 ===== `paths`
 
 A list of glob-based paths that will be crawled and fetched. All patterns
-supported by https://golang.org/pkg/path/filepath/#Glob[Golang Glob] are also
+supported by https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also
 supported here. For example, to fetch all files from a predefined level of
 subdirectories, the following pattern can be used: `/var/log/*/*.log`. This
 fetches all `.log` files from the subfolders of `/var/log`. It does not

filebeat/docs/modules/kibana.asciidoc

@@ -23,16 +23,18 @@ include::../include/running-modules.asciidoc[]
 
 include::../include/configuring-intro.asciidoc[]
 
-//set the fileset name used in the included example
+//set the fileset name used in the included file
 :fileset_ex: log
 
 include::../include/config-option-intro.asciidoc[]
 
 [float]
-==== `{fileset}` log fileset settings
+==== `log` fileset settings
 
 include::../include/var-paths.asciidoc[]
 
+:fileset_ex!:
+
 
 [float]
 === Fields

filebeat/filebeat.reference.yml

@@ -1184,6 +1184,21 @@ output.elasticsearch:
   # never, once, and freely. Default is never.
   #ssl.renegotiation: never
 
+  # The number of times to retry publishing an event after a publishing failure.
+  # After the specified number of retries, the events are typically dropped.
+  # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
+  # and retry until all events are published.  Set max_retries to a value less
+  # than 0 to retry until all events are published. The default is 3.
+  #max_retries: 3
+
+  # The maximum number of events to bulk in a single Logstash request. The
+  # default is 2048.
+  #bulk_max_size: 2048
+
+  # The number of seconds to wait for responses from the Logstash server before
+  # timing out. The default is 30s.
+  #timeout: 30s
+
 #------------------------------- Kafka output ----------------------------------
 #output.kafka:
   # Boolean flag to enable or disable the output module.
@@ -1595,6 +1610,13 @@ output.elasticsearch:
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
 
+# A list of fields to be added to the template and Kibana index pattern. Also
+# specify setup.template.overwrite: true to overwrite the existing template.
+# This setting is experimental.
+#setup.template.append_fields:
+#- name: field_name
+#  type: field_type
+
 # Enable json template loading. If this is enabled, the fields.yml is ignored.
 #setup.template.json.enabled: false
 

filebeat/module/kibana/_meta/docs.asciidoc

@@ -18,12 +18,14 @@ include::../include/running-modules.asciidoc[]
 
 include::../include/configuring-intro.asciidoc[]
 
-//set the fileset name used in the included example
+//set the fileset name used in the included file
 :fileset_ex: log
 
 include::../include/config-option-intro.asciidoc[]
 
 [float]
-==== `{fileset}` log fileset settings
+==== `log` fileset settings
 
 include::../include/var-paths.asciidoc[]
+
+:fileset_ex!:

filebeat/scripts/module/_meta/docs.asciidoc

@@ -37,3 +37,7 @@ the relevant file. For example:
 ==== `{fileset}` log fileset settings
 
 include::../include/var-paths.asciidoc[]
+
+:fileset_ex!:
+
+:modulename!:

heartbeat/heartbeat.reference.yml

@@ -631,6 +631,21 @@ output.elasticsearch:
   # never, once, and freely. Default is never.
   #ssl.renegotiation: never
 
+  # The number of times to retry publishing an event after a publishing failure.
+  # After the specified number of retries, the events are typically dropped.
+  # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
+  # and retry until all events are published.  Set max_retries to a value less
+  # than 0 to retry until all events are published. The default is 3.
+  #max_retries: 3
+
+  # The maximum number of events to bulk in a single Logstash request. The
+  # default is 2048.
+  #bulk_max_size: 2048
+
+  # The number of seconds to wait for responses from the Logstash server before
+  # timing out. The default is 30s.
+  #timeout: 30s
+
 #------------------------------- Kafka output ----------------------------------
 #output.kafka:
   # Boolean flag to enable or disable the output module.
@@ -1042,6 +1057,13 @@ output.elasticsearch:
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
 
+# A list of fields to be added to the template and Kibana index pattern. Also
+# specify setup.template.overwrite: true to overwrite the existing template.
+# This setting is experimental.
+#setup.template.append_fields:
+#- name: field_name
+#  type: field_type
+
 # Enable json template loading. If this is enabled, the fields.yml is ignored.
 #setup.template.json.enabled: false
 

libbeat/_meta/config.reference.yml

@@ -417,6 +417,21 @@ output.elasticsearch:
   # never, once, and freely. Default is never.
   #ssl.renegotiation: never
 
+  # The number of times to retry publishing an event after a publishing failure.
+  # After the specified number of retries, the events are typically dropped.
+  # Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
+  # and retry until all events are published.  Set max_retries to a value less
+  # than 0 to retry until all events are published. The default is 3.
+  #max_retries: 3
+
+  # The maximum number of events to bulk in a single Logstash request. The
+  # default is 2048.
+  #bulk_max_size: 2048
+
+  # The number of seconds to wait for responses from the Logstash server before
+  # timing out. The default is 30s.
+  #timeout: 30s
+
 #------------------------------- Kafka output ----------------------------------
 #output.kafka:
   # Boolean flag to enable or disable the output module.
@@ -828,6 +843,13 @@ output.elasticsearch:
 # Path to fields.yml file to generate the template
 #setup.template.fields: "${path.config}/fields.yml"
 
+# A list of fields to be added to the template and Kibana index pattern. Also
+# specify setup.template.overwrite: true to overwrite the existing template.
+# This setting is experimental.
+#setup.template.append_fields:
+#- name: field_name
+#  type: field_type
+
 # Enable json template loading. If this is enabled, the fields.yml is ignored.
 #setup.template.json.enabled: false