diff --git a/content/en/docs/concepts/cluster-administration/system-logs.md b/content/en/docs/concepts/cluster-administration/system-logs.md index 0466837356023..58f123a1c87da 100644 --- a/content/en/docs/concepts/cluster-administration/system-logs.md +++ b/content/en/docs/concepts/cluster-administration/system-logs.md @@ -22,14 +22,62 @@ generates log messages for the Kubernetes system components. For more information about klog configuration, see the [Command line tool reference](/docs/reference/command-line-tools-reference/). -An example of the klog native format: +Kubernetes is in the process of simplifying logging in its components. The +following klog command line flags [are +deprecated](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) +starting with Kubernetes 1.23 and will be removed in a future release: + +- `--add-dir-header` +- `--alsologtostderr` +- `--log-backtrace-at` +- `--log-dir` +- `--log-file` +- `--log-file-max-size` +- `--logtostderr` +- `--one-output` +- `--skip-headers` +- `--skip-log-headers` +- `--stderrthreshold` + +Output will always be written to stderr, regardless of the output +format. Output redirection is expected to be handled by the component which +invokes a Kubernetes component. This can be a POSIX shell or a tool like +systemd. + +In some cases, for example a distroless container or a Windows system service, +those options are not available. Then the +[`kube-log-runner`](https://github.com/kubernetes/kubernetes/blob/d2a8a81639fcff8d1221b900f66d28361a170654/staging/src/k8s.io/component-base/logs/kube-log-runner/README.md) +binary can be used as wrapper around a Kubernetes component to redirect +output. A prebuilt binary is included in several Kubernetes base images under +its traditional name as `/go-runner` and as `kube-log-runner` in server and +node release archives. + +This table shows how `kube-log-runner` invocations correspond to shell redirection: + +| Usage | POSIX shell (such as bash) | `kube-log-runner ` | +| -----------------------------------------|----------------------------|-------------------------------------------------------------| +| Merge stderr and stdout, write to stdout | `2>&1` | `kube-log-runner` (default behavior) | +| Redirect both into log file | `1>>/tmp/log 2>&1` | `kube-log-runner -log-file=/tmp/log` | +| Copy into log file and to stdout | `2>&1 \| tee -a /tmp/log` | `kube-log-runner -log-file=/tmp/log -also-stdout` | +| Redirect only stdout into log file | `>/tmp/log` | `kube-log-runner -log-file=/tmp/log -redirect-stderr=false` | + +### Klog output + +An example of the traditional klog native format: ``` I1025 00:15:15.525108 1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756] ``` +The message string may contain line breaks: +``` +I1025 00:15:15.525108 1 example.go:79] This is a message +which has a line break. +``` + + ### Structured Logging -{{< feature-state for_k8s_version="v1.19" state="alpha" >}} +{{< feature-state for_k8s_version="v1.23" state="beta" >}} {{< warning >}} Migration to structured log messages is an ongoing process. Not all log messages are structured in this version. When parsing log files, you must also handle unstructured log messages. @@ -38,9 +86,11 @@ Log formatting and value serialization are subject to change. {{< /warning>}} Structured logging introduces a uniform structure in log messages allowing for programmatic extraction of information. You can store and process structured logs with less effort and cost. -New message format is backward compatible and enabled by default. +The code which generates a log message determines whether it uses the traditional unstructured klog output +or structured logging. -Format of structured logs: +The default formatting of structured log messages is as text, with a format that +is backward compatible with traditional klog: ```ini "" ="" ="" ... @@ -52,6 +102,13 @@ Example: I1025 00:15:15.525108 1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready" ``` +Strings are quoted. Other values are formatted with +[`%+v`](https://pkg.go.dev/fmt#hdr-Printing), which may cause log messages to +continue on the next line [depending on the data](https://github.com/kubernetes/kubernetes/issues/106428). +``` +I1025 00:15:15.525108 1 example.go:116] "Example" data="This is text with a line break\nand \"quotation marks\"." someInt=1 someFloat=0.1 someStruct={StringField: First line, +second line.} +``` ### JSON log format @@ -82,7 +139,7 @@ Example of JSON log format (pretty printed): Keys with special meaning: * `ts` - timestamp as Unix time (required, float) -* `v` - verbosity (required, int, default 0) +* `v` - verbosity (only for info and not for error messages, int) * `err` - error string (optional, string) * `msg` - message (required, string) @@ -139,4 +196,5 @@ The `logrotate` tool rotates logs daily, or once the log size is greater than 10 * Read about the [Kubernetes Logging Architecture](/docs/concepts/cluster-administration/logging/) * Read about [Structured Logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1602-structured-logging) +* Read about [deprecation of klog flags](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) * Read about the [Conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md)