The supported log output sink types are documented below.
This sink type causes logging data to be captured into log files in a configurable logging directory.
The configuration key under the sinks key in the YAML
configuration is file-groups. Example configuration:
// sinks: // file-groups: # file group configurations start here // health: # defines one group called "health" // channels: HEALTH
Each generated log file is prefixed by the name of the process,
followed by the name of the group, separated by a hyphen. For example,
the group health will generate files named cockroach-health.XXX.log,
assuming the process is named cockroach. (A user can influence the
prefix by renaming the program executable.)
The files are named so that a lexicographical sort of the directory contents presents the file in creation order.
A symlink (e.g. cockroach-health.log) for each group points to the latest generated log file.
Every new file group sink configured automatically inherits
the configurations set in the file-defaults section.
For example:
// file-defaults: // redactable: false # default: disable redaction markers // dir: logs // sinks: // file-groups: // health: // channels: HEALTH // # This sink has redactable set to false, // # as the setting is inherited from file-defaults // # unless overridden here. // # // # Example override: // dir: health-logs # override the default 'logs'
{{site.data.alerts.callout_success}}
Run cockroach debug check-log-config to verify the effect of defaults inheritance.
{{site.data.alerts.end}}
Type-specific configuration options:
| Field | Description |
|---|---|
channels |
the list of logging channels that use this sink. See the channel selection configuration section for details. |
dir |
specifies the output directory for files generated by this sink. Inherited from file-defaults.dir if not specified. |
max-file-size |
the approximate maximum size of individual files generated by this sink. If zero, there is no maximum size. Inherited from file-defaults.max-file-size if not specified. |
max-group-size |
the approximate maximum combined size of all files to be preserved for this sink. An asynchronous garbage collection removes files that cause the file set to grow beyond this specified size. If zero, old files are not removed. Inherited from file-defaults.max-group-size if not specified. |
file-permissions |
the "chmod-style" permissions the log files are created with as a 3-digit octal number. The executable bit must not be set. Defaults to 644 (readable by all, writable by owner). Inherited from file-defaults.file-permissions if not specified. |
buffered-writes |
specifies whether to buffer log entries. Setting this to false flushes log writes upon every entry. Inherited from file-defaults.buffered-writes if not specified. |
Configuration options shared across all sink types:
| Field | Description |
|---|---|
filter |
specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. |
format |
the entry format to use. |
format-options |
additional options for the format. |
redact |
whether to strip sensitive information before log events are emitted to this sink. |
redactable |
whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. |
exit-on-error |
whether the logging system should terminate the process if an error is encountered while writing to this sink. |
auditable |
translated to tweaks to the other settings for this sink during validation. For example, it enables exit-on-error and changes the format of files from crdb-v1 to crdb-v1-count. |
buffering |
configures buffering for this log sink, or NONE to explicitly disable. See the common buffering configuration section for details. |
This sink type causes logging data to be sent over the network to a log collector that can ingest log data in a Fluentd-compatible protocol.
{{site.data.alerts.callout_danger}} TLS is not supported yet: the connection to the log collector is neither authenticated nor encrypted. Given that logging events may contain sensitive information, care should be taken to keep the log collector and the CockroachDB node close together on a private network, or connect them using a secure VPN. TLS support may be added at a later date. {{site.data.alerts.end}}
At the time of this writing, a Fluent sink buffers at most one log entry and retries sending the event at most one time if a network error is encountered. This is just sufficient to tolerate a restart of the Fluentd collector after a configuration change under light logging activity. If the server is unavailable for too long, or if more than one error is encountered, an error is reported to the process's standard error output with a copy of the logging event and the logging event is dropped.
The configuration key under the sinks key in the YAML
configuration is fluent-servers. Example configuration:
// sinks: // fluent-servers: # fluent configurations start here // health: # defines one sink called "health" // channels: HEALTH // address: 127.0.0.1:5170
Every new server sink configured automatically inherits the configurations set in the fluent-defaults section.
For example:
// fluent-defaults: // redactable: false # default: disable redaction markers // sinks: // fluent-servers: // health: // channels: HEALTH // # This sink has redactable set to false, // # as the setting is inherited from fluent-defaults // # unless overridden here.
The default output format for Fluent sinks is
json-fluent-compact. The fluent variants of the JSON formats
include a tag field as required by the Fluentd protocol, which
the non-fluent JSON format variants do not include.
{{site.data.alerts.callout_info}}
Run cockroach debug check-log-config to verify the effect of defaults inheritance.
{{site.data.alerts.end}}
Type-specific configuration options:
| Field | Description |
|---|---|
channels |
the list of logging channels that use this sink. See the channel selection configuration section for details. |
net |
the protocol for the fluent server. Can be "tcp", "udp", "tcp4", etc. |
address |
the network address of the fluent server. The host/address and port parts are separated with a colon. IPv6 numeric addresses should be included within square brackets, e.g.: [::1]:1234. |
Configuration options shared across all sink types:
| Field | Description |
|---|---|
filter |
specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. |
format |
the entry format to use. |
format-options |
additional options for the format. |
redact |
whether to strip sensitive information before log events are emitted to this sink. |
redactable |
whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. |
exit-on-error |
whether the logging system should terminate the process if an error is encountered while writing to this sink. |
auditable |
translated to tweaks to the other settings for this sink during validation. For example, it enables exit-on-error and changes the format of files from crdb-v1 to crdb-v1-count. |
buffering |
configures buffering for this log sink, or NONE to explicitly disable. See the common buffering configuration section for details. |
This sink type causes logging data to be sent over the network as requests to an HTTP server.
The configuration key under the sinks key in the YAML
configuration is http-servers. Example configuration:
// sinks: // http-servers: // health: // channels: HEALTH // address: http://127.0.0.1
Every new server sink configured automatically inherits the configuration set in the http-defaults section.
For example:
// http-defaults: // redactable: false # default: disable redaction markers // sinks: // http-servers: // health: // channels: HEALTH // # This sink has redactable set to false, // # as the setting is inherited from fluent-defaults // # unless overridden here.
The default output format for HTTP sinks is
json-compact. Other supported formats.
{{site.data.alerts.callout_info}}
Run cockroach debug check-log-config to verify the effect of defaults inheritance.
{{site.data.alerts.end}}
Type-specific configuration options:
| Field | Description |
|---|---|
channels |
the list of logging channels that use this sink. See the channel selection configuration section for details. |
address |
the network address of the http server. The host/address and port parts are separated with a colon. IPv6 numeric addresses should be included within square brackets, e.g.: [::1]:1234. Inherited from http-defaults.address if not specified. |
method |
the HTTP method to be used. POST and GET are supported; defaults to POST. Inherited from http-defaults.method if not specified. |
unsafe-tls |
enables certificate authentication to be bypassed. Defaults to false. Inherited from http-defaults.unsafe-tls if not specified. |
timeout |
the HTTP timeout. Defaults to 0 for no timeout. Inherited from http-defaults.timeout if not specified. |
disable-keep-alives |
causes the logging sink to re-establish a new connection for every outgoing log message. This option is intended for testing only and can cause excessive network overhead in production systems. Inherited from http-defaults.disable-keep-alives if not specified. |
headers |
a list of headers to attach to each HTTP request Inherited from http-defaults.headers if not specified. |
file-based-headers |
a list of headers with filepaths whose contents are attached to each HTTP request Inherited from http-defaults.file-based-headers if not specified. |
compression |
can be "none" or "gzip" to enable gzip compression. Set to "gzip" by default. Inherited from http-defaults.compression if not specified. |
Configuration options shared across all sink types:
| Field | Description |
|---|---|
filter |
specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. |
format |
the entry format to use. |
format-options |
additional options for the format. |
redact |
whether to strip sensitive information before log events are emitted to this sink. |
redactable |
whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. |
exit-on-error |
whether the logging system should terminate the process if an error is encountered while writing to this sink. |
auditable |
translated to tweaks to the other settings for this sink during validation. For example, it enables exit-on-error and changes the format of files from crdb-v1 to crdb-v1-count. |
buffering |
configures buffering for this log sink, or NONE to explicitly disable. See the common buffering configuration section for details. |
The standard error output stream of the running cockroach
process.
The configuration key under the sinks key in the YAML configuration
is stderr. Example configuration:
// sinks: // stderr: # standard error sink configuration starts here // channels: DEV
{{site.data.alerts.callout_info}}
The server start-up messages are still emitted at the start of the standard error
stream even when logging to stderr is enabled. This makes it generally difficult
to automate integration of stderr with log analyzers. Generally, we recommend using
file logging or network logging
instead of stderr when integrating with automated monitoring software.
{{site.data.alerts.end}}
It is not possible to enable the redactable parameter on the stderr sink if
capture-stray-errors (i.e., capturing stray error information to files) is disabled.
This is because when capture-stray-errors is disabled, the process's standard error stream
can contain an arbitrary interleaving of logging events and stray
errors. It is possible for stray error output to interfere with redaction markers
and remove the guarantees that information outside of redaction markers does not
contain sensitive information.
For a similar reason, no guarantee of parsability of the output format is available
when capture-stray-errors is disabled, since the standard error stream can then
contain an arbitrary interleaving of non-formatted error data.
Type-specific configuration options:
| Field | Description |
|---|---|
channels |
the list of logging channels that use this sink. See the channel selection configuration section for details. |
no-color |
forces the omission of VT color codes in the output even when stderr is a terminal. This option is deprecated; its effects are equivalent to 'format-options: {colors: none}'. |
Configuration options shared across all sink types:
| Field | Description |
|---|---|
filter |
specifies the default minimum severity for log events to be emitted to this sink, when not otherwise specified by the 'channels' sink attribute. |
format |
the entry format to use. |
format-options |
additional options for the format. |
redact |
whether to strip sensitive information before log events are emitted to this sink. |
redactable |
whether to keep redaction markers in the sink's output. The presence of redaction markers makes it possible to strip sensitive data reliably. |
exit-on-error |
whether the logging system should terminate the process if an error is encountered while writing to this sink. |
auditable |
translated to tweaks to the other settings for this sink during validation. For example, it enables exit-on-error and changes the format of files from crdb-v1 to crdb-v1-count. |
buffering |
configures buffering for this log sink, or NONE to explicitly disable. See the common buffering configuration section for details. |
Each sink can select multiple channels. The names of selected channels can be specified as a YAML array or as a string.
Additionally, severity filters can be applied separately for different groups of channels.
Example configurations:
# Select just these two channels. Space is important.
# This uses the severity filter set by the separate 'filter' attribute
# in the sink configuration.
channels: [OPS, HEALTH]
# Select PERF at severity INFO, and HEALTH and OPS at severity WARNING.
# The 'filter' attribute in the sink configuration is ignored.
channels: {INFO: [PERF], WARNING: [HEALTH, OPS]}
# The brackets are optional when selecting a single channel.
channels: OPS
channels: {INFO: PERF}
# The selection is case-insensitive.
channels: [ops, HeAlTh]
# Same configuration, as a YAML string. Avoid space around comma
# if using the YAML "inline" format.
channels: OPS,HEALTH
# Same configuration, as a quoted string.
channels: 'OPS, HEALTH'
# Same configuration, as a multi-line YAML array.
channels:
- OPS
- HEALTH
channels:
INFO:
- PERF
WARNING:
- OPS
- HEALTH
It is also possible to select all channels, using the "all" keyword. For example:
channels: all
channels: 'all'
channels: [all]
channels: ['all']
Likewise:
channels: {INFO: all}
etc.
It is also possible to select all channels except for a subset, using the "all except" keyword prefix. This makes it possible to define sinks that capture "everything else". For example:
channels: all except ops,health
channels: all except [ops,health]
channels: 'all except ops, health'
channels: 'all except [ops, health]'
Likewise:
channels: {INFO: all except ops,health}
etc.
Buffering may be configured with the following fields. It may also be explicitly set to "NONE" to disable buffering. Example configuration:
// file-defaults: // dir: logs // buffering: // max-staleness: 20s // flush-trigger-size: 25KB // max-buffer-size: 10MB // sinks: // file-groups: // health: // channels: HEALTH // buffering: // max-staleness: 5s # Override max-staleness for this sink. // ops: // channels: OPS // buffering: NONE # Disable buffering for this sink.
| Field | Description |
|---|---|
max-staleness |
the maximum time a log message will sit in the buffer before a flush is triggered. |
flush-trigger-size |
the number of bytes that will trigger the buffer to flush. |
max-buffer-size |
the limit on the size of the messages that are buffered. If this limit is exceeded, messages are dropped. The limit is expected to be higher than FlushTriggerSize. A buffer is flushed as soon as FlushTriggerSize is reached, and a new buffer is created once the flushing is started. Only one flushing operation is active at a time. |
format |
describes how the buffer output should be formatted. Currently 2 options: newline: default option - separates buffer entries with newline char json-array: separates entries with ',' and wraps buffer contents in square brackets |