From df33aea5a18d2f15951862621200b5785f3b50cf Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Thu, 4 Jun 2020 08:10:57 -0700 Subject: [PATCH] docs: Restructure and deduplicate security docs (#18594) --- libbeat/docs/https.asciidoc | 156 ++++++++++++++---- libbeat/docs/security/api-keys.asciidoc | 23 ++- libbeat/docs/security/basic-auth.asciidoc | 67 -------- libbeat/docs/security/securing-beats.asciidoc | 41 ----- libbeat/docs/security/tls.asciidoc | 24 --- libbeat/docs/security/users.asciidoc | 41 +++-- libbeat/docs/shared-securing-beat.asciidoc | 51 ++++-- .../docs/shared-ssl-logstash-config.asciidoc | 6 +- 8 files changed, 206 insertions(+), 203 deletions(-) delete mode 100644 libbeat/docs/security/basic-auth.asciidoc delete mode 100644 libbeat/docs/security/securing-beats.asciidoc delete mode 100644 libbeat/docs/security/tls.asciidoc diff --git a/libbeat/docs/https.asciidoc b/libbeat/docs/https.asciidoc index 90623bda561..af5020453c0 100644 --- a/libbeat/docs/https.asciidoc +++ b/libbeat/docs/https.asciidoc @@ -10,62 +10,105 @@ //// This content is structured to be included as a whole file. ////////////////////////////////////////////////////////////////////////// -To secure the communication between {beatname_uc} and Elasticsearch, you can use -HTTPS and basic authentication. Basic authentication for Elasticsearch is -available when you enable {security} (see -{ref}/secure-cluster.html[Secure a cluster] and <>). -If you aren't using {security}, you can use a web proxy instead. +[role="xpack"] +[[securing-communication-elasticsearch]] +== Secure communication with Elasticsearch -Here is a sample configuration: +When sending data to a secured cluster through the `elasticsearch` +output, {beatname_uc} can use any of the following authentication methods: +* Basic authentication credentials (username and password). +* Token-based API authentication. +* A client certificate. + +Authentication is specified in the {beatname_uc} configuration file: + +* To use *basic authentication*, specify the `username` and `password` settings under `output.elasticsearch`. +For example: ++ +-- ["source","yaml",subs="attributes,callouts"] ---------------------------------------------------------------------- output.elasticsearch: - username: {beatname_lc} <1> - password: verysecret <2> - protocol: https <3> - hosts: ["elasticsearch.example.com:9200"] <4> + hosts: ["https://myEShost:9200"] + username: "{beat_default_index_prefix}_writer" <1> + password: "{pwd}" <2> ---------------------------------------------------------------------- -<1> The username to use for authenticating to Elasticsearch. -<2> The password to use for authenticating to Elasticsearch. -<3> This setting enables the HTTPS protocol. -<4> The IP and port of the Elasticsearch nodes. - -TIP: To obfuscate passwords and other sensitive settings, +<1> This user needs the privileges required to publish events to {es}. +To create a user like this, see <>. +<2> This example shows a hard-coded password, but you should store sensitive +values ifndef::serverless[] -use the <>. +in the <>. endif::[] ifdef::serverless[] -use environment variables. +in environment variables. endif::[] +-- -{beatname_uc} verifies the validity of the server certificates and only accepts trusted -certificates. Creating a correct SSL/TLS infrastructure is outside the scope of -this document. - -By default {beatname_uc} uses the list of trusted certificate authorities from the -operating system where {beatname_uc} is running. You can configure {beatname_uc} to use a specific list of -CA certificates instead of the list from the OS. You can also configure it to use client authentication -by specifying the certificate and key to use when the server requires the {beatname_uc} to authenticate. Here is an example -configuration: +* To use token-based *API key authentication*, specify the `api_key` under `output.elasticsearch`. +For example: ++ +-- +["source","yaml",subs="attributes,callouts"] +---------------------------------------------------------------------- +output.elasticsearch: + hosts: ["https://myEShost:9200"] + api_key: "KnR6yE41RrSowb0kQ0HWoA" <1> +---------------------------------------------------------------------- +<1> This API key must have the privileges required to publish events to {es}. +To create an API key like this, see <>. +-- +[[beats-tls]] +* To use *Public Key Infrastructure (PKI) certificates* to authenticate users, +specify the `certificate` and `key` settings under `output.elasticsearch`. +For example: ++ +-- +["source","yaml",subs="attributes,callouts"] +---------------------------------------------------------------------- +output.elasticsearch: + hosts: ["https://myEShost:9200"] + ssl.certificate: "/etc/pki/client/cert.pem" <1> + ssl.key: "/etc/pki/client/cert.key" <2> +---------------------------------------------------------------------- +<1> The path to the certificate for SSL client authentication +<2> The client certificate key +-- ++ +These settings assume that the +distinguished name (DN) in the certificate is mapped to the appropriate roles in +the `role_mapping.yml` file on each node in the {es} cluster. For more +information, see {ref}/mapping-roles.html#mapping-roles-file[Using role +mapping files]. ++ +By default, {beatname_uc} uses the list of trusted certificate authorities (CA) from the +operating system where {beatname_uc} is running. If the certificate authority that signed your node certificates +is not in the host system's trusted certificate authorities list, you need +to add the path to the `.pem` file that contains your CA's certificate to the +{beatname_uc} configuration. This will configure {beatname_uc} to use a specific list of +CA certificates instead of the default list from the OS. ++ +Here is an example configuration: ++ +-- ["source","yaml",subs="attributes,callouts"] ---------------------------------------------------------------------- output.elasticsearch: - username: {beatname_lc} - password: verysecret - protocol: https - hosts: ["elasticsearch.example.com:9200"] + hosts: ["https://myEShost:9200"] ssl.certificate_authorities: <1> - /etc/pki/my_root_ca.pem - /etc/pki/my_other_ca.pem ssl.certificate: "/etc/pki/client.pem" <2> ssl.key: "/etc/pki/key.pem" <3> ---------------------------------------------------------------------- -<1> The list of CA certificates to trust +<1> Specify the path to the local `.pem` file that contains your Certificate +Authority's certificate. This is needed if you use your own CA to sign your node certificates. <2> The path to the certificate for SSL client authentication <3> The client certificate key - +-- ++ NOTE: For any given connection, the SSL/TLS certificates must have a subject that matches the value specified for `hosts`, or the SSL handshake fails. For example, if you specify `hosts: ["foobar:9200"]`, the certificate MUST @@ -73,3 +116,50 @@ include `foobar` in the subject (`CN=foobar`) or as a subject alternative name (SAN). Make sure the hostname resolves to the correct IP address. If no DNS is available, then you can associate the IP address with your hostname in `/etc/hosts` (on Unix) or `C:\Windows\System32\drivers\etc\hosts` (on Windows). + +ifndef::no_dashboards[] +[role="xpack"] +[float] +[[securing-communication-kibana]] +=== Secure communication with the Kibana endpoint + +If you've configured the <>, +you can also specify credentials for authenticating with {kib} under `kibana.setup`. +If no credentials are specified, Kibana will use the configured authentication method +in the Elasticsearch output. + +For example, specify a unique username and password to connect to Kibana like this: + +-- +["source","yaml",subs="attributes,callouts"] +---- +setup.kibana: + host: "mykibanahost:5601" + username: "{beat_default_index_prefix}_kib_setup" <1> + password: "{pwd}" <2> +---- +<1> This user needs privileges required to set up dashboards. To create a user like this, +see <>. +<2> This example shows a hard-coded password, but you should store sensitive +values +ifndef::serverless[] +in the <>. +endif::[] +ifdef::serverless[] +in environment variables. +endif::[] +endif::no_dashboards[] +-- + +[role="xpack"] +[float] +[[securing-communication-learn-more]] +=== Learn more about secure communication + +More information on sending data to a secured cluster is available in the configuration reference: + +* <> +* <> +ifndef::no_dashboards[] +* <> +endif::no_dashboards[] diff --git a/libbeat/docs/security/api-keys.asciidoc b/libbeat/docs/security/api-keys.asciidoc index aef276417a4..403fd011122 100644 --- a/libbeat/docs/security/api-keys.asciidoc +++ b/libbeat/docs/security/api-keys.asciidoc @@ -1,18 +1,12 @@ [role="xpack"] [[beats-api-keys]] -=== Grant access using API keys +== Grant access using API keys Instead of using usernames and passwords, you can use API keys to grant access to {es} resources. You can set API keys to expire at a certain time, and you can explicitly invalidate them. Any user with the `manage_api_key` or `manage_own_api_key` cluster privilege can create API keys. -See the {es} API key documentation for more information: - -* {ref}/security-api-create-api-key.html[Create API key] -* {ref}/security-api-get-api-key.html[Get API key information] -* {ref}/security-api-invalidate-api-key.html[Invalidate API key] - {beatname_uc} instances typically send both collected data and monitoring information to {es}. If you are sending both to the same cluster, you can use the same API key. For different clusters, you need to use an API key per cluster. @@ -20,8 +14,9 @@ API key. For different clusters, you need to use an API key per cluster. NOTE: For security reasons, we recommend using a unique API key per {beatname_uc} instance. You can create as many API keys per user as necessary. +[float] [[beats-api-key-publish]] -==== Create an API key for publishing +=== Create an API key for publishing To create an API key to use for writing data to {es}, use the {ref}/security-api-create-api-key.html[Create API key API], for example: @@ -67,8 +62,9 @@ output.elasticsearch: -------------------- <1> Format is `id:api_key` (as returned by {ref}/security-api-create-api-key.html[Create API key]) +[float] [[beats-api-key-monitor]] -==== Create an API key for monitoring +=== Create an API key for monitoring To create an API key to use for sending monitoring data to {es}, use the {ref}/security-api-create-api-key.html[Create API key API], for example: @@ -114,3 +110,12 @@ monitoring.elasticsearch: -------------------- <1> Format is `id:api_key` (as returned by {ref}/security-api-create-api-key.html[Create API key]) +[[learn-more-api-keys]] +[float] +=== Learn more about API keys + +See the {es} API key documentation for more information: + +* {ref}/security-api-create-api-key.html[Create API key] +* {ref}/security-api-get-api-key.html[Get API key information] +* {ref}/security-api-invalidate-api-key.html[Invalidate API key] diff --git a/libbeat/docs/security/basic-auth.asciidoc b/libbeat/docs/security/basic-auth.asciidoc deleted file mode 100644 index 7c3d8ec5b52..00000000000 --- a/libbeat/docs/security/basic-auth.asciidoc +++ /dev/null @@ -1,67 +0,0 @@ -[role="xpack"] -[[beats-basic-auth]] -=== Configure authentication credentials - -When sending data to a secured cluster through the `elasticsearch` -output, {beatname_uc} must either provide basic authentication credentials -or present a client certificate. - -*Before you begin:* <>. - -You specify authentication credentials in the {beatname_uc} configuration -file: - -* To use basic authentication, specify the `username` and `password` settings -under `output.elasticsearch`. For example: -+ --- -["source","yaml",subs="attributes,callouts"] ----- -output.elasticsearch: - hosts: ["localhost:9200"] - username: "{beat_default_index_prefix}_writer" <1> - password: "{pwd}" <2> ----- -<1> Let's assume this user has the privileges required to publish events to -{es}. -<2> The example shows a hard-coded password, but you should store sensitive -values -ifndef::serverless[] -in the <>. -endif::[] -ifdef::serverless[] -in environment variables. -endif::[] --- -ifndef::apm-server[] -+ -If you've configured the {kib} endpoint, also specify credentials for -authenticating with {kib}. For example: -+ -["source","yaml",subs="attributes,callouts"] ----- -setup.kibana: - host: "mykibanahost:5601" - username: "{beat_default_index_prefix}_setup" <1> - password: "{pwd}" ----- -<1> Let's assume this user has the privileges required to set up dashboards. -endif::apm-server[] - -* To use Public Key Infrastructure (PKI) certificates to authenticate users, -configure the `certificate` and `key` settings. These settings assume that the -distinguished name (DN) in the certificate is mapped to the appropriate roles in -the `role_mapping.yml` file on each node in the {es} cluster. For more -information, see {ref}/mapping-roles.html#mapping-roles-file[Using role -mapping files]. -+ -["source","yaml",subs="attributes,callouts"] --------------------------------------------------- -output.elasticsearch: - hosts: ["localhost:9200"] - ssl.certificate: "/etc/pki/client/cert.pem" - ssl.key: "/etc/pki/client/cert.key" --------------------------------------------------- - -To learn more about {stack} security features and other types of -authentication, see {ref}/secure-cluster.html[Secure a cluster]. diff --git a/libbeat/docs/security/securing-beats.asciidoc b/libbeat/docs/security/securing-beats.asciidoc deleted file mode 100644 index f103efbf2f0..00000000000 --- a/libbeat/docs/security/securing-beats.asciidoc +++ /dev/null @@ -1,41 +0,0 @@ -[role="xpack"] -[[securing-beats]] -== Configure {beatname_uc} to use {security-features} - -[subs="attributes"] -++++ -Use {security-features} -++++ - -If you want {beatname_uc} to connect to a cluster that has -{security-features} enabled, there are extra configuration steps: - -. <>. -+ -You can use role-based access control to grant {beatname_uc} users access to -secured resources. - -. <>. -+ -You can use API keys to grant access to {es} more securely. - -. <>. -+ -To interact with a secured cluster, {beatname_uc} must either provide basic -authentication credentials or present a client certificate. - -. <>. -+ -If encryption is enabled on the cluster, you need to enable HTTPS in the -{beatname_uc} configuration. - -For more information about {security-features}, see -{ref}/secure-cluster.html[Secure a cluster]. - -include::users.asciidoc[] - -include::api-keys.asciidoc[] - -include::basic-auth.asciidoc[] - -include::tls.asciidoc[] diff --git a/libbeat/docs/security/tls.asciidoc b/libbeat/docs/security/tls.asciidoc deleted file mode 100644 index 10f7a60653e..00000000000 --- a/libbeat/docs/security/tls.asciidoc +++ /dev/null @@ -1,24 +0,0 @@ -[role="xpack"] -[[beats-tls]] -=== Configure {beatname_uc} to use encrypted connections - -If encryption is enabled on the {es} cluster, you need to connect to {es} via -HTTPS. If the certificate authority (CA) that signed your node certificates -is not in the host system's trusted certificate authorities list, you also need -to add the path to the `.pem` file that contains your CA's certificate to the -{beatname_uc} configuration. - -To configure a {beatname_uc} to connect to {es} via HTTPS, add the `https` -protocol to all host URLs: - -["source","js",subs="attributes,callouts"] --------------------------------------------------- -output.elasticsearch: - hosts: ["https://localhost:9200"] <1> - index: "{beatname_lc}" - ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] <2> --------------------------------------------------- -<1> Specify the `https` protocol to connect the {es} cluster. -<2> Specify the path to the local `.pem` file that contains your Certificate -Authority's certificate. This is generally only needed if you use your -own CA to sign your node certificates. diff --git a/libbeat/docs/security/users.asciidoc b/libbeat/docs/security/users.asciidoc index 72083671984..dbf731bc494 100644 --- a/libbeat/docs/security/users.asciidoc +++ b/libbeat/docs/security/users.asciidoc @@ -1,6 +1,6 @@ [role="xpack"] [[feature-roles]] -=== Grant users access to secured resources +== Grant users access to secured resources You can use role-based access control to grant users access to secured resources. The roles that you set up depend on your organization's security @@ -18,14 +18,21 @@ by {beatname_uc} create visualizations that access {beatname_uc} data -{security} provides {ref}/built-in-roles.html[built-in roles] that grant a +{es-security-features} provides {ref}/built-in-roles.html[built-in roles] that grant a subset of the privileges needed by {beatname_uc} users. When possible, use the built-in roles to minimize the affect of future changes on your security strategy. +Instead of using usernames and passwords, roles and privileges can be assigned to +API keys to grant access to Elasticsearch resources. See <> for +more information. [[privileges-to-setup-beats]] -==== Grant privileges and roles needed for setup +=== Grant privileges and roles needed for setup + +++++ +Create a _setup_ user +++++ IMPORTANT: Setting up {beatname_uc} is an admin-level task that requires extra privileges. As a best practice, grant the setup role to administrators only, and @@ -104,9 +111,13 @@ endif::has_central_config[] Omit any roles that aren't relevant in your environment. [[privileges-to-publish-monitoring]] -==== Grant privileges and roles needed for monitoring +=== Grant privileges and roles needed for monitoring + +++++ +Create a _monitoring_ user +++++ -{security} provides built-in users and roles for monitoring. The privileges and +{es-security-features} provides built-in users and roles for monitoring. The privileges and roles needed depend on the method used to collect monitoring data. [IMPORTANT] @@ -119,7 +130,7 @@ grant it the roles described in the following sections. ==== * If you're using <> to -collect metrics about {beatname_uc}, {security} provides +collect metrics about {beatname_uc}, {es-security-features} provides the +{beat_monitoring_user}+ {ref}/built-in-users.html[built-in user] and +{beat_monitoring_user}+ {ref}/built-in-roles.html[built-in role] to send monitoring information. You can use the built-in user, if it's available in your @@ -169,7 +180,7 @@ users who need to monitor {beatname_uc}: ifndef::serverless[] * If you're <> to collect -metrics about {beatname_uc}, {security} provides the `remote_monitoring_user` +metrics about {beatname_uc}, {es-security-features} provides the `remote_monitoring_user` {ref}/built-in-users.html[built-in user], and the `remote_monitoring_collector` and `remote_monitoring_agent` {ref}/built-in-roles.html[built-in roles] for collecting and sending monitoring information. You can use the built-in user, if @@ -211,7 +222,11 @@ information. endif::serverless[] [[privileges-to-publish-events]] -==== Grant privileges and roles needed for publishing +=== Grant privileges and roles needed for publishing + +++++ +Create a _publishing_ user +++++ Users who publish events to {es} need to create and write to {beatname_uc} indices. To minimize the privileges required by the writer role, use the @@ -283,7 +298,11 @@ endif::apm-server[] . Assign the *writer role* to users who will index events into {es}. [[kibana-user-privileges]] -==== Grant privileges and roles needed to read {beatname_uc} data from {kib} +=== Grant privileges and roles needed to read {beatname_uc} data from {kib} + +++++ +Create a _reader_ user +++++ {kib} users typically need to view dashboards and visualizations that contain {beatname_uc} data. These users might also need to create and edit dashboards @@ -353,9 +372,9 @@ data: |==== endif::apm-server[] - +// to do: THIS SHOULD GO IN ITS OWN FILE [[learn-more-security]] -==== Learn more about users and roles +=== Learn more about privileges, roles, and users Want to learn more about creating users and roles? See {ref}/secure-cluster.html[Secure a cluster]. Also see: diff --git a/libbeat/docs/shared-securing-beat.asciidoc b/libbeat/docs/shared-securing-beat.asciidoc index 67f38aeb50b..b8dcc3b1957 100644 --- a/libbeat/docs/shared-securing-beat.asciidoc +++ b/libbeat/docs/shared-securing-beat.asciidoc @@ -9,49 +9,66 @@ ++++ The following topics provide information about securing the {beatname_uc} -process and securing communication between {beatname_uc} and other products in -the Elastic stack: +process and connecting to a cluster that has {security-features} enabled. + +You can use role-based access control and optionally, API keys to grant {beatname_uc} users access to +secured resources. + +* <> +* <>. + +After privileged users have been created, use authentication to connect to a secured Elastic cluster. -ifdef::apm-server[] -* <> -endif::[] * <> ifndef::no-output-logstash[] * <> endif::[] -* <> + +ifdef::apm-server[] +For secure communication between APM Server and APM Agents, see <>. +endif::[] + ifndef::serverless[] +On Linux, {beatname_uc} can take advantage of secure computing mode to restrict the +system calls that a process can issue. + * <> endif::[] +// APM HTTPS information ifdef::beat-specific-security[] include::{beat-specific-security}[] endif::[] -- +// APM privileges ifdef::apm-server[] -include::{docdir}/secure-communication-agents.asciidoc[] +include::{docdir}/feature-roles.asciidoc[] endif::[] -//sets block macro for https.asciidoc included in next section +// Beat privileges +ifndef::apm-server[] +include::./security/users.asciidoc[] +endif::[] -[[securing-communication-elasticsearch]] -== Secure communication with Elasticsearch +// API Keys +include::./security/api-keys.asciidoc[] -include::./https.asciidoc[] +// APM Agent security +ifdef::apm-server[] +include::{docdir}/secure-communication-agents.asciidoc[] +endif::[] -//sets block macro for shared-ssl-logstash-config.asciidoc included in next section +// Elasticsearch security +include::./https.asciidoc[] +// Logstash security ifndef::no-output-logstash[] -[[configuring-ssl-logstash]] -== Secure communication with Logstash - include::./shared-ssl-logstash-config.asciidoc[] endif::[] -include::./security/securing-beats.asciidoc[] - +// Linux Seccomp ifndef::serverless[] include::./security/linux-seccomp.asciidoc[] endif::[] diff --git a/libbeat/docs/shared-ssl-logstash-config.asciidoc b/libbeat/docs/shared-ssl-logstash-config.asciidoc index 090b34cfc33..31cbf73fc62 100644 --- a/libbeat/docs/shared-ssl-logstash-config.asciidoc +++ b/libbeat/docs/shared-ssl-logstash-config.asciidoc @@ -9,6 +9,10 @@ //// include::../../libbeat/docs/shared-ssl-logstash-config.asciidoc[] ////////////////////////////////////////////////////////////////////////// +[role="xpack"] +[[configuring-ssl-logstash]] +== Secure communication with Logstash + You can use SSL mutual authentication to secure connections between {beatname_uc} and Logstash. This ensures that {beatname_uc} sends encrypted data to trusted Logstash servers only, and that the Logstash server receives data from trusted {beatname_uc} clients only. @@ -19,7 +23,7 @@ To use SSL mutual authentication: {beatname_uc} and Logstash. Creating a correct SSL/TLS infrastructure is outside the scope of this document. There are many online resources available that describe how to create certificates. + -TIP: If you are using {security}, you can use the +TIP: If you are using {security-features}, you can use the {ref}/certutil.html[elasticsearch-certutil tool] to generate certificates. . Configure {beatname_uc} to use SSL. In the +{beatname_lc}.yml+ config file, specify the following settings under