From c7b196e46ac619c492e93620473913a7e8de7c8c Mon Sep 17 00:00:00 2001 From: Monica Sarbu Date: Sat, 23 Sep 2017 00:25:03 +0200 Subject: [PATCH] Update import/export dashboards section --- docs/devguide/newdashboards.asciidoc | 356 +++++++++++++-------------- 1 file changed, 172 insertions(+), 184 deletions(-) diff --git a/docs/devguide/newdashboards.asciidoc b/docs/devguide/newdashboards.asciidoc index df336935013..8757b0dc1a7 100644 --- a/docs/devguide/newdashboards.asciidoc +++ b/docs/devguide/newdashboards.asciidoc @@ -1,17 +1,18 @@ [[new-dashboards]] -== Creating New Kibana Dashboards for a Beat +== Creating New Kibana Dashboards for a Beat or a Beat module -NOTE: Starting with Beats 5.0.0, the Kibana dashboards are not released as part of the Beat package. They are released in a separate -package called `beats-dashboards`. +++++ +Creating New Kibana Dashboards +++++ -When contributing to Beats development, you may want to add new dashboards or modify existing ones. To make this easier, -we've created an `import_dashboards` script that you can use to <> from an -existing Beat into Kibana, where you can modify the dashboards or use them as a starting point to create new dashboards. -Kibana saves the dashboards, along with all the dependencies such as visualizations, searches, and index patterns, in -a special index in Elasticsearch. By default the index is `.kibana`, but you can specify a different index. - -When you're done making changes to the dashboards in Kibana, you can use the `export_dashboards` script to <>, along with all dependencies, to a local directory. +When contributing to Beats development, you may want to add new dashboards or +customize the existing ones. To get started, you can +<> that come with the official +Beats and use them as a starting point for your own dashboards. When you're done +making changes to the dashboards in Kibana, you can use the `export_dashboards` +script to <>, along with all +dependencies, to a local directory. To make sure the dashboards are compatible with the latest version of Kibana and Elasticsearch, we recommend that you use the virtual environment under @@ -30,103 +31,111 @@ The following topics provide more detail about importing and working with Beats [[import-dashboards]] === Importing Existing Beat Dashboards -You can use the `import_dashboards` script to import all the dashboards and the index pattern for a Beat, including the dependencies such as visualizations and searches. -The `import_dashboards` script is available under -https://github.com/elastic/beats/tree/master/dev-tools/cmd/import_dashboards[beats/dev-tools/cmd/import_dashboards]. It's also available in each Beat package under the `scripts` directory. +The official Beats come with Kibana dashboards, and starting with 6.0.0, they +are part of every Beat package. You can use the Beat executable to import all +the dashboards and the index pattern for a Beat, including the dependencies +such as visualizations and searches. + +You can use the Beat executable to import all the dashboards and the index pattern for a Beat, including the dependencies such as visualizations and searches. + +To import the dashboards, run the `setup` command. + + +[source,shell] +------------------------- +./metricbeat setup +------------------------- + +The `setup` phase loads: + +- Index mapping template in Elasticsearch +- Kibana dashboards +- Machine Learning jobs (if available) + +For more details about the `setup` command, run the following: + +[source,shell] +------------------------- +./metricbeat help setup + +This command does initial setup of the environment: + + * Index mapping template in Elasticsearch to ensure fields are mapped. + * Kibana dashboards (where available). + * ML jobs (where available). -There are a couple of common use cases for importing dashboards: +Usage: + filebeat setup [flags] -* Users who are getting started with Beats may want to <>. -* Community Beats developers may want to <> to use as a starting point for new dashboards. +Flags: + --dashboards Setup dashboards only + -h, --help help for setup + --machine-learning Setup machine learning job configurations only + --modules string List of enabled modules (comma separated) + --template Setup index template only +--------------------------- -[[import-single-beat]] -==== Import Dashboards and/or the Index Pattern for a Single Beat +The flags are useful when you don't want to load everything. For example, to +import only the dashboards, use the `--dashboards` flag: -Using the `import_dashboards` script from the Beat package, you can import the dashboards and the index pattern to -Elasticsearch running on localhost for a single Beat (eg. Metricbeat): +[source,shell] +--------------------- +./metricbeat setup --dashboards +------------------------------- + +Starting with Beats 6.0.0, the dashboards are no longer loaded directly into Elasticsearch. Instead, they are imported directly into Kibana. +Thus, if your Kibana instance is not listening on localhost, or you enabled +X-Pack for Kibana, you need to either configure the Kibana endpoint in +the config for the Beat, or pass the Kibana host and credentials as +arguments to the `setup` command. For example: + +[source,shell] +------------------------- +./metricbeat setup -E setup.kibana.host=192.168.3.206:5601 -E setup.kibana.username=elastic -E setup.kibana.password=secret +-------------------------- + +By default, the `setup` command imports the dashboards from the `kibana` +directory, which is available in the Beat package. + +NOTE: The format of the saved dashboards is not compatible between Kibana 5.x and 6.x. Thus, the Kibana 5.x dashboards are available in +the `5.x` directory, and the Kibana 6.0 dashboards, and older are in the `default` directory. + +In case you are using customized dashboards, you can import them: - from a local directory: + [source,shell] ---------------------------------------------------------------------- -./scripts/import_dashboards -dir kibana/metricbeat +./metricbeat setup -E setup.dashboards.directory=kibana ---------------------------------------------------------------------- - from a local zip archive: + [source,shell] ---------------------------------------------------------------------- -./scripts/import_dashboards -file metricbeat-dashboards-1.1.zip +./metricbeat setup -E setup.dashboards.file=metricbeat-dashboards-6.0.zip ---------------------------------------------------------------------- -- from the official zip archive available at {dashboards}: +- from a zip archive available online: + [source,shell] ---------------------------------------------------------------------- -./scripts/import_dashboards +./metricbeat setup -E setup.dashboards.url=path/to/url ---------------------------------------------------------------------- + -See <> for more details about the structure of dashboard archives. -- from a zip archive available online: -+ -[source,shell] ------------------------ -./scripts/import_dashboards -url https://github.com/monicasarbu/metricbeat-dashboards/archive/v1.1.zip ------------------------ - -If you don't specify the location of the archive, then by default it's set to the official zip archive containing the index pattern and the dashboards of the official Beats. +See <> for a description of the `setup.dashboards` configuration options. -To import only the index-pattern for a single Beat (eg. Metricbeat) use: -[source,shell] ------------------------ -./scripts/import_dashboards -only-index ------------------------ - -To import only the dashboards together with visualizations and searches for a single Beat (eg. Metricbeat) use: - -[source,shell] ------------------------ -./scripts/import_dashboards -only-dashboards ------------------------ - - -NOTE: When running the `import_dashboards` script from within the Beat package, the `-beat` option is set automatically to the Beat -name. - -See <> for a description of other import options. [[import-dashboards-for-development]] ==== Import Dashboards for Development -For development or community Beats, it's easier to run the `import_dashboards` script from the https://github.com/elastic/beats/tree/master/dev-tools/cmd/import_dashboards[beats/dev-tools/cmd/import_dashboards] directory. In this case, you need to first compile the script: - -[source,shell] ------------------------ -cd beats/dev-tools/cmd/import_dashboards -make ------------------------ - -And then you can import the index pattern and the dashboards together with visualizations and searches for a single -Beat, by passing the `-beat` option. - -For example, to import the Metricbeat dashboards together with visualizations, -searches, and the Metricbeat index pattern: - -[source,shell] ------------------ -beats/dev-tools/cmd/import_dashboards/import_dashboards -beat metricbeat ------------------ - -For this example, you must specify `-beat metricbeat`. If the `-beat` option is not -specified, the script imports the dashboards of all Beats. - -NOTE: You can make use of the Makefile from the Beat GitHub repository to import the +You can make use of the Makefile from the Beat GitHub repository to import the dashboards. If Elasticsearch is running on localhost, then you can run the following command from the Beat repository: [source,shell] -------------------------------- -make -C import-dashboards +make import-dashboards -------------------------------- If Elasticsearch is running on a different host, then you can use the `ES_URL` variable: @@ -137,120 +146,42 @@ ES_URL="http://192.168.3.206:9200" make import-dashboards ------------------------------- [[import-dashboard-options]] -==== Command Line Options +==== Kibana dashboards configuration -The `import_dashboards` script accepts the following command-line options. To see all the available options, read the descriptions below or run: +The configuration file (`*.reference.yml`) of each Beat contains the `setup.dashboards` section for configuring from where to get the Kibana dashboards, as well as the name of the index pattern. +Each of these configuration options can be overwritten with the command line options by using `-E` flag. -["source","sh",subs="attributes"] ----------------------------------------------------------------------- -./import_dashboards -h ----------------------------------------------------------------------- - -*`-beat `*:: -The Beat name. The Beat name is required when importing from a zip archive. When using `import_dashboards` from the Beat package, this option is set automatically with the name of -the Beat. When running the script from source, the default value is "", so you need to set this option in order to install the index pattern and -the dashboards for a single Beat. Otherwise the script imports the index pattern and the dashboards for all Beats. -*`-cacert `*:: -The Certificate Authority to use for server verification. +*`setup.dashboards.directory=`*:: +Local directory that contains the saved dashboards and their dependencies. +The default value is the `kibana` directory available in the Beat package. -*`-cert `*:: -The certificate to use for SSL client authentication. The certificate must be in -PEM format. +*`setup.dashboards.file=`*:: +Local zip archive with the dashboards. The archive can contain Kibana dashboards for a single Beat or for multiple Beats. The dashboards of each Beat are placed under a separate directory with the name of the Beat. -*`-dir `*:: -Local directory that contains the subdirectories: dashboard, visualization, search, and index-pattern. The default value is the current directory. - -*`-es `*:: -The Elasticsearch URL. The default value is http://localhost:9200. - -*`-file `*:: -Local zip archive with the dashboards. The archive can contain Kibana dashboards for a single Beat or for multiple Beats. +*`setup.dashboards.url=`*:: +Zip archive with the dashboards, available online. The archive can contain Kibana dashboards for a single Beat or for +multiple Beats. The dashboards for each Beat are placed under a separate directory with the name of the Beat. -*`-i `*:: +*`setup.dashboards.index `*:: You should only use this option if you want to change the index pattern name that's used by default. For example, if the default is `metricbeat-*`, you can change it to `custombeat-*`. -*`-insecure`*:: -If specified, "insecure" SSL connections are allowed. - -*`-k `*:: -The Elasticsearch index pattern where Kibana saves its configuration. The default value is `.kibana`. - -*`-key `*:: -The client certificate key. The key must be in PEM format. - -*`-only-dashboards`*:: -If specified, then only the dashboards, along with their visualizations and searches, are imported. The index pattern is -not imported. By default, this is false. - -*`-only-index`*:: -If specified, then only the index pattern is imported. The dashboards, along with their visualizations and searches, are not imported. By default, this is false. - -*`-pass `*:: -The password for authenticating the connection to Elasticsearch by using Basic Authentication. By default no username and password are used. - -*`-snapshot`*:: -Using `-snapshot` will import the snapshot dashboards build for the current version. This is mainly useful when running a snapshot Beat build for testing purpose. -+ -NOTE: When using `-snapshot`, `-url` will be ignored. - -*`-url `*:: -Zip archive with the dashboards, available online. The archive can contain Kibana dashboards for a single Beat or for -multiple Beats. - -*`-user `*:: -The username for authenticating the connection to Elasticsearch by using Basic Authentication. By default no username and password are used. - - -[[dashboards-archive-structure]] -==== Structure of the Dashboards Archive - -The zip archive contains dashboards for at least one Beat. The index pattern, dashboards, visualizations and searches -are available in a separate directory for each Beat, having the name of the Beat. For example the official zip archive (beats-dashboards-{stack-version}) has the following structure: - -[source,shell] -------------------------- - metricbeat/ - dashboard/ - search/ - visualization/ - index-pattern/ - packetbeat/ - dashboard/ - search/ - visualization/ - index-pattern/ - filebeat/ - index-pattern/ - winlogbeat/ - dashboard/ - search/ - visualization/ - index-pattern/ ------------------------- - [[build-dashboards]] === Building Your Own Beat Dashboards -For visualizing the dashboards of a Beat in Kibana you need to have configured: +NOTE: If you want to modify a dashboard that comes with a Beat, it's better to modify a copy of the dashboard because the Beat overwrites the dashboards during the setup phase in order to have the latest version. For duplicating a dashboard, just use the `Clone` button from the top of the page. -* the Beat index pattern, which specifies how Kibana should display the Beat fields -* the Beat dashboards, including the dependencies such as visualizations and searches - -For the Elastic Beats, the index pattern is available in the GitHub repository of each Beat under -`etc/kibana/index-pattern` or under the `beats-dashboards` zip archive, available for each Beat release. -For the community Beats, you can easily generate the index-pattern from the `etc/fields.yml` file. For more details -check the <> section. +Before building your own dashboards or customizing the existing ones, you need to load: -If you would like to build dashboards from scratch for any Elastic Beats, you can start by importing the same version of the index pattern as your Beat: +* the Beat index pattern, which specifies how Kibana should display the Beat fields +* the Beat dashboards that you want to customize -[source,shell] ---------------- -$ scripts/import_dashboards -only-index ---------------- +For the Elastic Beats, the index pattern is available in the Beat package under +`kibana/*/index-pattern`. The index-pattern is automatically generated from the `fields.yml` file, available in the Beat package. For more details +check the <> section. After creating your own dashboards in Kibana, you can <> to a local directory, and then <> in order to be able to share the dashboards with the community. @@ -258,15 +189,15 @@ directory, and then <> in order to be [[generate-index-pattern]] === Generating the Beat Index Pattern -If you change the fields exported by the Beat, you need to generate a new index pattern for your Beat. Otherwise -you can just use the index pattern available under the `etc/kibana/index-pattern` directory or in the `beats-dashboards` -archive for the Elastic Beats. +The index-pattern defines the format of each field, and it's used by Kibana to know how to display the field. +If you change the fields exported by the Beat, you need to generate a new index pattern for your Beat. Otherwise, you can just use the index pattern available under the `kibana/*/index-pattern` directory. -The Beat index pattern is generated from the `etc/fields.yml`, where all the fields for a Beat are defined. For each field, besides the `type`, you can configure the +The Beat index pattern is generated from the `fields.yml`, which contains all +the fields exported by the Beat. For each field, besides the `type`, you can configure the `format` field. The format informs Kibana about how to display a certain field. A good example is `percentage` or `bytes` to display fields as `50%` or `5MB`. -To generate the index pattern from the `etc/fields.yml`, you need to run the following command in the Beat repository: +To generate the index pattern from the `fields.yml`, you need to run the following command in the Beat repository: [source,shell] --------------- @@ -277,11 +208,13 @@ make update === Exporting New and Modified Beat Dashboards To export all the dashboards for any Elastic Beat or any community Beat, including any new or modified dashboards and all dependencies such as -visualizations, searches, you can use the Python script `export_dashboards.py` from -https://github.com/elastic/beats/tree/master/dev-tools[dev-tools]. See the dev-tools +visualizations, searches, you can use the Golang script `export_dashboards.go` from +https://github.com/elastic/beats/tree/master/dev-tools/cmd/dashboards[dev-tools] for exporting Kibana 6.0 dashboards or later, and the Python script `export_5x_dashboards.py` +for exporting Kibana 5.x dashboards. See the dev-tools https://github.com/elastic/beats/tree/master/dev-tools/README.md[readme] for more info. - +///////////////////// +NOT YET IMPLEMENTED NOTE: You can make use of the Makefile from the Beat GitHub repository to export all the Kibana dashboards for a Beat from your Elasticsearch. If Elasticsearch is running on localhost, then you just need to run the following command from the Beat repository: @@ -297,6 +230,61 @@ If Elasticsearch is running on a different host, then you can use the `ES_URL` v ES_URL="http://192.168.3.206:9200" make export-dashboards ---------------------------- +///////////////////// + +==== Exporting Kibana 6.0 dashboards and newer + +The `dev-tools/cmd/export_dashboards.go` script helps you export your customized Kibana 6.0 dashboards and newer. You might need to export a single dashboard or all the dashboards available for a module or Beat. + + +===== Export a single Kibana dashboard + +You can export a single dashboard by passing the dashboard ID in the `-dashboard` flag. + +NOTE: The dashboard ID is available in the dashboard URL. For example, in case the dashboard URL is +`app/kibana#/dashboard/7fea2930-478e-11e7-b1f0-cb29bac6bf8b?_g=()&_a=(description:'Overview%2...`, the dashboard ID is `7fea2930-478e-11e7-b1f0-cb29bac6bf8b`. + +[source,shell] +--------------- +cd filebeat/module/redis/_meta/kibana/default/dashboard +go run ../../../../../../../dev-tools/cmd/dashboards/export_dashboards.go -dashboard 7fea2930-478e-11e7-b1f0-cb29bac6bf8b -output Filebeat-redis.json +--------------- + +This generates the `Filebeat-redis.json` file that contains the dashboard for the Redis module of Filebeat, including the dependencies (visualizations and searches). + +===== Export all module/Beat dashboards + +Each module should contain a `module.yml` file with a list of all the dashboards available for the module. For the Beats that don't have support for modules (e.g. Packetbeat), +there is a `dashboards.yml` file that defines all the Packetbeat dashboards. + +Below, it's an example of the `module.yml` file for the system module in Metricbeat: + +[source,shell] +--------------- +dashboards: +- id: Metricbeat-system-overview + file: Metricbeat-system-overview.json + +- id: 79ffd6e0-faa0-11e6-947f-177f697178b8 + file: Metricbeat-host-overview.json + +- id: CPU-slash-Memory-per-container + file: Metricbeat-containers-overview.json +--------------- + + +Each dashboard is defined by an `id` and the name of json `file` where the dashboard is saved locally. + +By passing the yml file to the `export_dashboards.go` script, you can export all the dashboards defined: + +[source,shell] +------------------- +go run dev-tools/cmd/dashboards/export_dashboards.go -yml filebeat/module/system/module.yml +------------------- + + + +==== Exporting Kibana 5.x dashboards To export only some Kibana dashboards for an Elastic Beat or community Beat, you can simply pass a regular expression to the `export_dashboards.py` script to match the selected Kibana dashboards. @@ -313,14 +301,14 @@ For example, to export all Kibana dashboards that start with the **Packetbeat** [source,shell] ---------------------------------------------------------------------- -python ../dev-tools/export_dashboards.py --regex Packetbeat* +python ../dev-tools/cmd/dashboards/export_dashboards.py --regex Packetbeat* ---------------------------------------------------------------------- To see all the available options, read the descriptions below or run: [source,shell] ---------------------------------------------------------------------- -python ../dev-tools/export_dashboards.py -h +python ../dev-tools/cmd/dashboards/export_dashboards.py -h ---------------------------------------------------------------------- *`--url `*:: @@ -349,7 +337,7 @@ output/ [[archive-dashboards]] === Archiving Your Beat Dashboards -The Kibana dashboards for the Elastic Beats are saved under the `etc/kibana` directory. To create a zip archive with the +The Kibana dashboards for the Elastic Beats are saved under the `kibana` directory. To create a zip archive with the dashboards, including visualizations and searches and the index pattern, you can run the following command in the Beat repository: @@ -359,7 +347,7 @@ make package-dashboards -------------- The Makefile is part of libbeat, which means that community Beats contributors can use the commands shown here to -archive dashboards. The dashboards must be available under the `etc/kibana` directory. +archive dashboards. The dashboards must be available under the `kibana` directory. Another option would be to create a repository only with the dashboards, and use the GitHub release functionality to create a zip archive.