Add Okta module documentation, config cleanup, _id field (elastic#18953)

This add documentation for the Okta module. It contains descriptions of the configuration options and general information about the module. I fixed an issue with the module where it was not setting the _id field for Elasticsearch events. I also did some cleanup to the pipeline.js (indentation, semi-colons, strict equality checks). The module's manifest was updated to not duplicate httpjson's default values. The module was accepting configuration as JSON strings for some parameters (http_headers, http_request_body, pagination, rate_limit, ssl) which is inconsistent with how other parts of Beats are configured so I removed this. Now these options expect regular YAML objects for values. None of these options are required to use the module so the impact to users should be minimal.
melchiormoulin · Oct 14, 2020 · 725e309 · 725e309
1 parent 6c6869f
commit 725e309
Show file tree

Hide file tree

Showing 9 changed files with 279 additions and 97 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -42,6 +42,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 * iptables {pull}18756[18756]
 * Checkpoint {pull}18754[18754]
 - Preserve case of http.request.method.  ECS prior to 1.6 specified normalizing to lowercase, which lost information. Affects filesets: apache/access, elasticsearch/audit, iis/access, iis/error, nginx/access, nginx/ingress_controller, aws/elb, suricata/eve, zeek/http. {issue}18154[18154] {pull}18359[18359]
+- Okta module now requires objects instead of JSON strings for the `http_headers`, `http_request_body`, `pagination`, `rate_limit`, and `ssl` variables. {pull}18953[18953]
 
 *Heartbeat*
 
@@ -164,6 +165,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix `googlecloud.audit` pipeline to only take in fields that are explicitly defined by the dataset. {issue}18465[18465] {pull}18472[18472]
 - Fix `o365.audit` failing to ingest events when ip address is surrounded by square brackets. {issue}18587[18587] {pull}18591[18591]
 - Fix Kubernetes Watcher goroutine leaks when input config is invalid and `input.reload` is enabled. {issue}18629[18629] {pull}18630[18630]
+- Okta module now sets the Elasticsearch `_id` field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. {pull}18953[18953]
 
 *Heartbeat*
 

diff --git a/filebeat/docs/modules/okta.asciidoc b/filebeat/docs/modules/okta.asciidoc
@@ -12,15 +12,105 @@ This file is generated! See scripts/docs_collector.py
 
 beta[]
 
-This is a filebeat module for retrieving system logs from Okta (www.okta.com) via API. 
+The Okta module collects events from the
+https://developer.okta.com/docs/reference/[Okta API]. Specifically this supports
+reading from the https://developer.okta.com/docs/reference/api/system-log/[Okta
+System Log API].
 
-:has-dashboards!:
+:fileset_ex: system
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `system` fileset settings
+
+The Okta System Log records system events related to your organization in order
+to provide an audit trail that can be used to understand platform activity and
+to diagnose problems. This module is implemented using the
+<<filebeat-input-httpjson,httpjson>> input and is configured to paginate through
+the logs while honoring any
+https://developer.okta.com/docs/reference/rate-limits/[rate-limiting] headers
+sent by Okta.
+
+NOTE: This module does not persist the timestamp of the last read event in
+order to facilitate resuming on restart. This feature will be coming in a future
+version. When you restart the module will read events from the beginning of the
+log. To minimize duplicates documents the module uses the event's Okta UUID
+value as the Elasticsearch `_id`.
+
+This is an example configuration for the module.
+
+[source,yaml]
+----
+- module okta
+  system:
+    var.url: https://yourOktaDomain/api/v1/logs
+    var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
+----
+
+[float]
+===== Configuration options
+
+*`var.url`*::
+
+Specifies the URL to the Okta System Log API. Required.
++
+[source,yaml]
+----
+    var.url: https://mycompany.okta.com/api/v1/logs
+----
+
+*`var.api_key`*::
+
+Specifies the Okta API token to use in requests to the API. Required.
+The token is used in an HTTP `Authorization` header with the `SSWS` scheme.
+See https://developer.okta.com/docs/guides/create-an-api-token/create-the-token/[
+Create an API token] for information on how to obtain a token.
++
+[source,yaml]
+----
+    var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
+----
 
-This module comes with a sample dashboard. For example:
+*`var.http_client_timeout`*::
+
+Duration of the time limit on HTTP requests made by the module. Defaults to
+`60s`.
+
+*`var.interval`*::
+
+Duration between requests to the API. Defaults to `60s`.
+
+*`var.keep_original_message`*::
+
+Boolean flag indicating if the original JSON event string should be included in
+the `event.original` field. Defaults to `true`.
+
+*`var.ssl`*::
+
+Configuration options for SSL parameters like the certificate authority to use
+for HTTPS-based connections. If the `ssl` section is missing, the host CAs are
+used for HTTPS connections to Okta. See <<configuration-ssl>> for more
+information.
++
+[source,yaml]
+----
+    var.ssl:
+      supported_protocols: [TLSv1.2]
+----
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard:
 
 [role="screenshot"]
 image::./images/filebeat-okta-dashboard.png[]
 
+:has-dashboards!:
+
+:fileset_ex!:
+
 :modulename!:
 
 

diff --git a/x-pack/filebeat/filebeat.reference.yml b/x-pack/filebeat/filebeat.reference.yml
@@ -860,17 +860,10 @@ filebeat.modules:
 - module: okta
   system:
     enabled: true
-    # API key to access Okta
-    #var.api_key
-
-    # URL of the Okta REST API
-    #var.url
-
-    # Disable SSL verification 
-    #var.ssl: |-
-    #  {
-    #    "verification_mode": "none"
-    #  }
+    # You must configure the URL with your Okta domain and provide an
+    # API token to access the logs API.
+    #var.url: https://yourOktaDomain/api/v1/logs
+    #var.api_key: 'yourApiTokenHere'
 
 #------------------------------- Osquery Module -------------------------------
 - module: osquery

diff --git a/x-pack/filebeat/module/okta/_meta/config.yml b/x-pack/filebeat/module/okta/_meta/config.yml
@@ -1,14 +1,7 @@
 - module: okta
   system:
     enabled: true
-    # API key to access Okta
-    #var.api_key
-
-    # URL of the Okta REST API
-    #var.url
-
-    # Disable SSL verification 
-    #var.ssl: |-
-    #  {
-    #    "verification_mode": "none"
-    #  }
+    # You must configure the URL with your Okta domain and provide an
+    # API token to access the logs API.
+    #var.url: https://yourOktaDomain/api/v1/logs
+    #var.api_key: 'yourApiTokenHere'
diff --git a/x-pack/filebeat/module/okta/_meta/docs.asciidoc b/x-pack/filebeat/module/okta/_meta/docs.asciidoc
@@ -7,13 +7,103 @@
 
 beta[]
 
-This is a filebeat module for retrieving system logs from Okta (www.okta.com) via API. 
+The Okta module collects events from the
+https://developer.okta.com/docs/reference/[Okta API]. Specifically this supports
+reading from the https://developer.okta.com/docs/reference/api/system-log/[Okta
+System Log API].
 
-:has-dashboards!:
+:fileset_ex: system
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `system` fileset settings
+
+The Okta System Log records system events related to your organization in order
+to provide an audit trail that can be used to understand platform activity and
+to diagnose problems. This module is implemented using the
+<<filebeat-input-httpjson,httpjson>> input and is configured to paginate through
+the logs while honoring any
+https://developer.okta.com/docs/reference/rate-limits/[rate-limiting] headers
+sent by Okta.
+
+NOTE: This module does not persist the timestamp of the last read event in
+order to facilitate resuming on restart. This feature will be coming in a future
+version. When you restart the module will read events from the beginning of the
+log. To minimize duplicates documents the module uses the event's Okta UUID
+value as the Elasticsearch `_id`.
+
+This is an example configuration for the module.
+
+[source,yaml]
+----
+- module okta
+  system:
+    var.url: https://yourOktaDomain/api/v1/logs
+    var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
+----
+
+[float]
+===== Configuration options
+
+*`var.url`*::
+
+Specifies the URL to the Okta System Log API. Required.
++
+[source,yaml]
+----
+    var.url: https://mycompany.okta.com/api/v1/logs
+----
+
+*`var.api_key`*::
+
+Specifies the Okta API token to use in requests to the API. Required.
+The token is used in an HTTP `Authorization` header with the `SSWS` scheme.
+See https://developer.okta.com/docs/guides/create-an-api-token/create-the-token/[
+Create an API token] for information on how to obtain a token.
++
+[source,yaml]
+----
+    var.api_key: '00QCjAl4MlV-WPXM...0HmjFx-vbGua'
+----
 
-This module comes with a sample dashboard. For example:
+*`var.http_client_timeout`*::
+
+Duration of the time limit on HTTP requests made by the module. Defaults to
+`60s`.
+
+*`var.interval`*::
+
+Duration between requests to the API. Defaults to `60s`.
+
+*`var.keep_original_message`*::
+
+Boolean flag indicating if the original JSON event string should be included in
+the `event.original` field. Defaults to `true`.
+
+*`var.ssl`*::
+
+Configuration options for SSL parameters like the certificate authority to use
+for HTTPS-based connections. If the `ssl` section is missing, the host CAs are
+used for HTTPS connections to Okta. See <<configuration-ssl>> for more
+information.
++
+[source,yaml]
+----
+    var.ssl:
+      supported_protocols: [TLSv1.2]
+----
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard:
 
 [role="screenshot"]
 image::./images/filebeat-okta-dashboard.png[]
 
+:has-dashboards!:
+
+:fileset_ex!:
+
 :modulename!:
diff --git a/x-pack/filebeat/module/okta/system/config/input.yml b/x-pack/filebeat/module/okta/system/config/input.yml
@@ -1,19 +1,48 @@
 {{ if eq .input "httpjson" }}
 
 type: httpjson
+
+{{ if .api_key }}
 api_key: {{ .api_key }}
-authentication_scheme: {{.authentication_scheme}}
+{{ end }}
+
+authentication_scheme: {{ .authentication_scheme }}
+
+{{ if .http_client_timeout }}
 http_client_timeout: {{ .http_client_timeout }}
+{{ end }}
+
+{{ if .http_method }}
 http_method: {{ .http_method }}
-http_headers: {{ .http_headers }}
+{{ end }}
+
+{{ if .http_headers }}
+http_headers: {{ .http_headers | tojson }}
+{{ end }}
+
+{{ if .http_request_body }}
 http_request_body: {{ .http_request_body }}
-no_http_body: {{ .no_http_body }}
+{{ end }}
+
 interval: {{ .interval }}
+
+{{ if .json_objects_array }}
 json_objects_array: {{ .json_objects_array }}
-pagination: {{ .pagination }}
-rate_limit: {{ .rate_limit }}
+{{ end }}
+
+no_http_body: {{ .no_http_body }}
+
+pagination: {{ .pagination | tojson }}
+
+rate_limit: {{ .rate_limit | tojson }}
+
+{{ if .ssl }}
+ssl: {{ .ssl | tojson }}
+{{ end }}
+
+{{ if .url }}
 url: {{ .url }}
-ssl: {{ .ssl }}
+{{ end }}
 
 {{ else if eq .input "file" }}