From d411d57e99057a7a00d547dfb58cd25b5a1d8eec Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 07:46:53 -0800 Subject: [PATCH 01/31] Moved from kubernetes/docs/proposals --- .../design-proposals/core-metrics-pipeline.md | 315 ++++++++++++++++++ 1 file changed, 315 insertions(+) create mode 100644 contributors/design-proposals/core-metrics-pipeline.md diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md new file mode 100644 index 00000000000..81de95e26b8 --- /dev/null +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -0,0 +1,315 @@ +# Core Metrics Pipeline in kubelet + +**Author**: David Ashpole (@dashpole) + +**Last Updated**: 12/21/2016 + +**Status**: Draft Proposal (WIP) + +This document proposes a design for an internal Core Metrics Pipeline. + + + +- [Core Metrics Pipeline in kubelet](#core-metrics-pipeline-in-kubelet) + - [Introduction](#introduction) + - [Background](#background) + - [Motivations](#motivations) + - [Proposal](#proposal) + - [Non Goals](#non-goals) + - [Design](#design) + - [Proposed Core Metrics API:](#proposed-core-metrics-api) + - [Core Machine Info:](#core-machine-info) + - [Implementation Plan](#implementation-plan) + - [Rollout Plan](#rollout-plan) + - [Implementation Status](#implementation-status) + + + +## Introduction + +### Background +The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. + +CAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds), and the kubelet then simply querries these cached metrics whenever it has a need for them. + +Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet summary API is published to the kubelet summary API endpoint. Some of the metrics provided by the summary API are consumed internally, but most are not used internally. + +### Motivations + +Giving the kubelet the role of both providing metrics for its own use, and providing metrics for users has a couple problems + - First, it is clearly inefficent to collect metrics and not use them. The kubelet uses only a small portion of the metrics it collects, and thus presents considerable extra overhead to any users who do not use them, or prefer a third party monitoring solution. + - Second, as the number of metrics collected grows over time, the kubelet will gain more and more overhead for collecting, processing, and publishing these metrics. Since the metrics users may want is unbounded, the kubelet's resource overhead could easily grow to unreasonable levels. + +It is very cumbersome to make changes or bugfixes in cAdvisor, because that then needs to be vendored back into kubernetes. + +CAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. + +### Proposal + +I propose to create a set of core metrics, collected by the kubelet, and used solely by internal kubernetes components. + +### Non Goals +Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. + +Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. + +## Design + +This design covers only the internal Core Metrics Pipeline. + +High level requirements for the design are as follows: + - Do not break existing users. We should continue to provide the full summary API by default. + - The kubelet collects the minimum possible number of metrics for full kubernetes functionality. + - Code for collecting core metrics resides in the kubernetes codebase. + - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. + +Metrics requirements, based on kubernetes component needs, are as follows: + - Kubelet + - Node-level capacity and availability metrics for Disk and Memory + - Pod-level usage metrics for Disk and Memory + - Scheduler + - Node-level capacity and availability metrics for Disk, CPU, and Memory + - Pod-level usage metrics for Disk, CPU, and Memory + - Container-level usage metrics for Disk, CPU, and Memory + - Horizontal-Pod-Autoscaler (HPA) + - Node-level capacity and availability metrics for CPU and Memory + - Pod-level usage metrics for CPU and Memory + +More details on how I intend to achieve these high level goals can be found in the Implementation Plan. + +In order to continue to provide the full summary API, either the kubelet or a stand-alone version of cAdvisor will need to publish these metrics. + +This Core Metrics API will be versioned to account for version-skew between kubernetes components. + +This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed for an internal use case, they can be added to the core metrics API. + +### Proposed Core Metrics API: + +An important difference between the current summary api and the proposed core metrics api is that per-pod stats in the core metrics api contain only usage data, and not capacity-related statistics. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how the kubelet uses the data. The kubelet finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of stats. + +```go +// CoreStats is a top-level container for holding NodeStats and PodStats. +type CoreStats struct { + // Overall node resource stats. + Node NodeResources `json:"node"` + // Per-pod usage stats. + Pods []PodUsage `json:"pods"` +} + +// NodeStats holds node-level stats. NodeStats contains capacity and availibility for Node Resources. +type NodeResources struct { + // The filesystem device used by node k8s components. + // +optional + KubeletFsDevice string `json:"kubeletfs"` + // The filesystem device used by node runtime components. + // +optional + RuntimeFsDevice string `json:"runtimefs"` + // Stats pertaining to cpu resources. + // +optional + CPU *CpuResources `json:"cpu,omitempty"` + // Stats pertaining to memory (RAM) resources. + // +optional + Memory *MemoryResources `json:"memory,omitempty"` + // Stats pertaining to node filesystem resources. + // +optional + Filesystems []DiskResources `json:"filesystems, omitempty" patchStrategy:"merge" patchMergeKey:"device"` +} + +// CpuResources containes data about cpu resource usage +type CpuResources struct { + // The number of cores in this machine. + NumCores int `json:"numcores"` + // The current Usage of CPU resources + TotalUsage *CpuUsage `json:"cpuusage,omitempty"` +} + +// MemoryResources contains data about memory resource usage. +type MemoryResources struct { + // The time at which these stats were updated. + Timestamp metav1.Time `json:"time"` + // The memory capacity, in bytes + CapacityBytes *uint64 `json:"capacitybytes,omitempty"` + // The available memory, in bytes + AvailableBytes *uint64 `json:"availablebytes,omitempty"` +} + +// DiskResources contains data about filesystem disk resources. +type DiskResources struct { + // The time at which these stats were updated. + Timestamp metav1.Time `json:"time"` + // The device that this filesystem is on + Device string `json:"device"` + // AvailableBytes represents the storage space available (bytes) for the filesystem. + // +optional + AvailableBytes *uint64 `json:"availableBytes,omitempty"` + // CapacityBytes represents the total capacity (bytes) of the filesystems underlying storage. + // +optional + CapacityBytes *uint64 `json:"capacityBytes,omitempty"` + // InodesFree represents the free inodes in the filesystem. + // +optional + InodesFree *uint64 `json:"inodesFree,omitempty"` + // Inodes represents the total inodes in the filesystem. + // +optional + Inodes *uint64 `json:"inodes,omitempty"` +} + +// PodUsage holds pod-level unprocessed sample stats. +type PodUsage struct { + // UID of the pod + PodUID string `json:"uid"` + // Stats pertaining to pod total usage of cpu + // This may include additional overhead not included in container usage statistics. + // +optional + CPU *CpuUsage `json:"cpu,omitempty"` + // Stats pertaining to pod total usage of system memory + // This may include additional overhead not included in container usage statistics. + // +optional + Memory *MemoryUsage `json:"memory,omitempty"` + // Stats of containers in the pod. + Containers []ContainerUsage `json:"containers" patchStrategy:"merge" patchMergeKey:"uid"` + // Stats pertaining to volume usage of filesystem resources. + // +optional + Volumes []VolumeUsage `json:"volume,omitempty" patchStrategy:"merge" patchMergeKey:"name"` +} + +// ContainerUsage holds container-level usage stats. +type ContainerUsage struct { + // UID of the container + ContainerUID string `json:"uid"` + // Stats pertaining to container usage of cpu + // +optional + CPU *CpuUsage `json:"memory,omitempty"` + // Stats pertaining to container usage of system memory + // +optional + Memory *MemoryUsage `json:"memory,omitempty"` + // Stats pertaining to container rootfs usage of disk. + // Rootfs.UsedBytes is the number of bytes used for the container write layer. + // +optional + Rootfs *DiskUsage `json:"rootfs,omitempty"` + // Stats pertaining to container logs usage of Disk. + // +optional + Logs *DiskUsage `json:"logs,omitempty"` +} + +// CpuUsage holds statistics about the amount of cpu time consumed +type CpuUsage struct { + // The time at which these stats were updated. + Timestamp metav1.Time `json:"time"` + // Total CPU usage (sum of all cores) averaged over the sample window. + // The "core" unit can be interpreted as CPU core-nanoseconds per second. + // +optional + UsageNanoCores *uint64 `json:"usageNanoCores,omitempty"` + // Cumulative CPU usage (sum of all cores) since object creation. + // +optional + UsageCoreNanoSeconds *uint64 `json:"usageCoreNanoSeconds,omitempty"` +} + +// MemoryUsage holds statistics about the quantity of memory consumed +type MemoryUsage struct { + // The time at which these stats were updated. + Timestamp metav1.Time `json:"time"` + // The amount of working set memory. This includes recently accessed memory, + // dirty memory, and kernel memory. + // +optional + WorkingSetBytes *uint64 `json:"workingSetBytes,omitempty"` +} + +// VolumeUsage holds statistics about the quantity of disk resources consumed for a volume +type VolumeUsage struct { + // Embedded DiskUsage + DiskUsage + // Name is the name given to the Volume + // +optional + Name string `json:"name,omitempty"` +} + +// DiskUsage holds statistics about the quantity of disk resources consumed +type DiskUsage struct { + // The time at which these stats were updated. + Timestamp metav1.Time `json:"time"` + // The device on which resources are consumed + Device string `json:"device"` + // UsedBytes represents the disk space consumed on the device, in bytes. + // +optional + UsedBytes *uint64 `json:"usedBytes,omitempty"` + // InodesUsed represents the inodes consumed on the device + // +optional + InodesUsed *uint64 `json:"inodesUsed,omitempty"` +} +``` + +### Core Machine Info: + +In addition to providing metrics, cAdvisor also provides machine info. While it is not neccessary to use this structure for reporting Machine Info, the following contains all of the information provided to the kubelet by cAdvisor, generally used at startup. + +The code that provides this data currently resides in cAdvisor. I propose moving this to the kubelet as well. + +```go +type CoreInfo struct { + // MachineID reported by the node. For unique machine identification + // in the cluster this field is prefered. Learn more from man(5) + // machine-id: http://man7.org/linux/man-pages/man5/machine-id.5.html + MachineID string `json:"machineID" protobuf:"bytes,1,opt,name=machineID"` + // SystemUUID reported by the node. For unique machine identification + // MachineID is prefered. This field is specific to Red Hat hosts + // https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/getting-system-uuid.html + SystemUUID string `json:"systemUUID" protobuf:"bytes,2,opt,name=systemUUID"` + // Boot ID reported by the node. + BootID string `json:"bootID" protobuf:"bytes,3,opt,name=bootID"` + // Kernel Version reported by the node from 'uname -r' (e.g. 3.16.0-0.bpo.4-amd64). + KernelVersion string `json:"kernelVersion" protobuf:"bytes,4,opt,name=kernelVersion"` + // OS Image reported by the node from /etc/os-release (e.g. Debian GNU/Linux 7 (wheezy)). + OSImage string `json:"osImage" protobuf:"bytes,5,opt,name=osImage"` + // Capacity represents the total resources of a node. + // More info: http://kubernetes.io/docs/user-guide/persistent-volumes#capacity for more details. + // +optional + // ContainerRuntime Version reported by the node through runtime remote API (e.g. docker://1.5.0). + ContainerRuntimeVersion string `json:"containerRuntimeVersion" protobuf:"bytes,6,opt,name=containerRuntimeVersion"` + // Cloud provider the machine belongs to. + CloudProvider CloudProvider `json:"cloud_provider"` + // ID of cloud instance (e.g. instance-1) given to it by the cloud provider. + InstanceID InstanceID `json:"instance_id"` +} + +type CloudProvider string + +const ( + GCE CloudProvider = "GCE" + AWS = "AWS" + Azure = "Azure" + Baremetal = "Baremetal" + UnknownProvider = "Unknown" +) +``` + +## Implementation Plan + +I will move all code pertaining to collection and processing of core metrics from cAdvisor into kubernetes. +I will vendor the new core metrics code back into cAdvisor. +I will modify volume stats collection so that it relies on this code. +I will modify the structure of stats collection code to be "On-Demand" + +Tenative future work, not included in this proposal: +Obtain all runtime-specific information needed to collect metrics from the CRI. +Create a third party metadata API, whose function is to provide third party monitoring solutions with kubernetes-specific data (pod-container relationships, for example). +Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. It will consume the above metadata API, and provide the summary API. +The kubelet no longer provides the summary API, and starts cAdvisor stand-alone by default. Include flag to not start cAdvisor. + +## Rollout Plan + +TBD + +## Implementation Status + +The implementation goals of the first milestone are outlined below. +- [ ] Create the proposal +- [ ] Move all code relevant for the collection and processing of core metrics from cAdvisor into kubernetes. +- [ ] Vendor the new core metrics code back into cAdvisor. +- [ ] Modify volume stats collection so that it relies on this code. +- [ ] Modify the structure of stats collection code to be "On-Demand" + + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/core-metrics-pipeline.md?pixel)]() + From 3bad1202c42561445ae41655d60378bff107ec3b Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 13:26:29 -0800 Subject: [PATCH 02/31] Addressed comments s/querries/queries s/CAdvisor/cAdvisor s/CoreInfo/MachineInfo --- contributors/design-proposals/core-metrics-pipeline.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 81de95e26b8..4659b97d670 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -30,7 +30,9 @@ This document proposes a design for an internal Core Metrics Pipeline. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. -CAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds), and the kubelet then simply querries these cached metrics whenever it has a need for them. + + +is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds), and the kubelet then simply queries these cached metrics whenever it has a need for them. Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet summary API is published to the kubelet summary API endpoint. Some of the metrics provided by the summary API are consumed internally, but most are not used internally. @@ -42,7 +44,7 @@ Giving the kubelet the role of both providing metrics for its own use, and provi It is very cumbersome to make changes or bugfixes in cAdvisor, because that then needs to be vendored back into kubernetes. -CAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. +cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal @@ -245,7 +247,7 @@ In addition to providing metrics, cAdvisor also provides machine info. While it The code that provides this data currently resides in cAdvisor. I propose moving this to the kubelet as well. ```go -type CoreInfo struct { +type MachineInfo struct { // MachineID reported by the node. For unique machine identification // in the cluster this field is prefered. Learn more from man(5) // machine-id: http://man7.org/linux/man-pages/man5/machine-id.5.html From 1e6945925ca2ddaaffd9fa99e51d524eff8a4145 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 14:06:56 -0800 Subject: [PATCH 03/31] More comments Addressed some nits, removed some of motivation, as it is covered by Architecture, s/DiskUsage/FilesystemUsage s/DiskResources/FilesystemResources --- .../design-proposals/core-metrics-pipeline.md | 52 ++++++++----------- 1 file changed, 23 insertions(+), 29 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 4659b97d670..6fd1ccd620d 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -30,27 +30,22 @@ This document proposes a design for an internal Core Metrics Pipeline. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. - - -is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds), and the kubelet then simply queries these cached metrics whenever it has a need for them. +cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds), and the kubelet then simply queries these cached metrics whenever it has a need for them. Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet summary API is published to the kubelet summary API endpoint. Some of the metrics provided by the summary API are consumed internally, but most are not used internally. ### Motivations -Giving the kubelet the role of both providing metrics for its own use, and providing metrics for users has a couple problems - - First, it is clearly inefficent to collect metrics and not use them. The kubelet uses only a small portion of the metrics it collects, and thus presents considerable extra overhead to any users who do not use them, or prefer a third party monitoring solution. - - Second, as the number of metrics collected grows over time, the kubelet will gain more and more overhead for collecting, processing, and publishing these metrics. Since the metrics users may want is unbounded, the kubelet's resource overhead could easily grow to unreasonable levels. - -It is very cumbersome to make changes or bugfixes in cAdvisor, because that then needs to be vendored back into kubernetes. +The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal -I propose to create a set of core metrics, collected by the kubelet, and used solely by internal kubernetes components. +I propose to use this set of core metrics, collected by the kubelet, and used solely by internal kubernetes components. ### Non Goals + Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. @@ -61,8 +56,7 @@ This design covers only the internal Core Metrics Pipeline. High level requirements for the design are as follows: - Do not break existing users. We should continue to provide the full summary API by default. - - The kubelet collects the minimum possible number of metrics for full kubernetes functionality. - - Code for collecting core metrics resides in the kubernetes codebase. + - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. Metrics requirements, based on kubernetes component needs, are as follows: @@ -76,6 +70,8 @@ Metrics requirements, based on kubernetes component needs, are as follows: - Horizontal-Pod-Autoscaler (HPA) - Node-level capacity and availability metrics for CPU and Memory - Pod-level usage metrics for CPU and Memory + - Master metrics API + - More details on how I intend to achieve these high level goals can be found in the Implementation Plan. @@ -102,10 +98,10 @@ type CoreStats struct { type NodeResources struct { // The filesystem device used by node k8s components. // +optional - KubeletFsDevice string `json:"kubeletfs"` + KubeletFsDevice string `json:"kubeletfs,omitempty"` // The filesystem device used by node runtime components. // +optional - RuntimeFsDevice string `json:"runtimefs"` + RuntimeFsDevice string `json:"runtimefs,omitempty"` // Stats pertaining to cpu resources. // +optional CPU *CpuResources `json:"cpu,omitempty"` @@ -114,7 +110,7 @@ type NodeResources struct { Memory *MemoryResources `json:"memory,omitempty"` // Stats pertaining to node filesystem resources. // +optional - Filesystems []DiskResources `json:"filesystems, omitempty" patchStrategy:"merge" patchMergeKey:"device"` + Filesystems []FilesystemResources `json:"filesystems, omitempty" patchStrategy:"merge" patchMergeKey:"device"` } // CpuResources containes data about cpu resource usage @@ -135,8 +131,8 @@ type MemoryResources struct { AvailableBytes *uint64 `json:"availablebytes,omitempty"` } -// DiskResources contains data about filesystem disk resources. -type DiskResources struct { +// FilesystemResources contains data about filesystem disk resources. +type FilesystemResources struct { // The time at which these stats were updated. Timestamp metav1.Time `json:"time"` // The device that this filesystem is on @@ -187,10 +183,10 @@ type ContainerUsage struct { // Stats pertaining to container rootfs usage of disk. // Rootfs.UsedBytes is the number of bytes used for the container write layer. // +optional - Rootfs *DiskUsage `json:"rootfs,omitempty"` + Rootfs *FilesystemUsage `json:"rootfs,omitempty"` // Stats pertaining to container logs usage of Disk. // +optional - Logs *DiskUsage `json:"logs,omitempty"` + Logs *FilesystemUsage `json:"logs,omitempty"` } // CpuUsage holds statistics about the amount of cpu time consumed @@ -218,15 +214,15 @@ type MemoryUsage struct { // VolumeUsage holds statistics about the quantity of disk resources consumed for a volume type VolumeUsage struct { - // Embedded DiskUsage - DiskUsage + // Embedded FilesystemUsage + FilesystemUsage // Name is the name given to the Volume // +optional Name string `json:"name,omitempty"` } -// DiskUsage holds statistics about the quantity of disk resources consumed -type DiskUsage struct { +// FilesystemUsage holds statistics about the quantity of disk resources consumed +type FilesystemUsage struct { // The time at which these stats were updated. Timestamp metav1.Time `json:"time"` // The device on which resources are consumed @@ -286,14 +282,12 @@ const ( ## Implementation Plan -I will move all code pertaining to collection and processing of core metrics from cAdvisor into kubernetes. -I will vendor the new core metrics code back into cAdvisor. +I will create a separate endpoint TBD to publish this set of core metrics. I will modify volume stats collection so that it relies on this code. -I will modify the structure of stats collection code to be "On-Demand" +I will modify the structure of stats collection code to be "On-Demand". Tenative future work, not included in this proposal: -Obtain all runtime-specific information needed to collect metrics from the CRI. -Create a third party metadata API, whose function is to provide third party monitoring solutions with kubernetes-specific data (pod-container relationships, for example). +Obtain all runtime-specific information needed to collect metrics from the CRI. Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. It will consume the above metadata API, and provide the summary API. The kubelet no longer provides the summary API, and starts cAdvisor stand-alone by default. Include flag to not start cAdvisor. @@ -305,8 +299,8 @@ TBD The implementation goals of the first milestone are outlined below. - [ ] Create the proposal -- [ ] Move all code relevant for the collection and processing of core metrics from cAdvisor into kubernetes. -- [ ] Vendor the new core metrics code back into cAdvisor. +- [ ] Implement collection and consumption of core metrics. +- [ ] Create Kubelet API endpoint for core metrics. - [ ] Modify volume stats collection so that it relies on this code. - [ ] Modify the structure of stats collection code to be "On-Demand" From d0aaf9adfde5130750a429cc8cd3051a0d6fe076 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 14:12:02 -0800 Subject: [PATCH 04/31] Changes cAdvisor vendoring no longer changed. Clarified that standalone cAdvisor will take over providing summary api. --- contributors/design-proposals/core-metrics-pipeline.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 6fd1ccd620d..1d7c757e2f0 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -75,7 +75,7 @@ Metrics requirements, based on kubernetes component needs, are as follows: More details on how I intend to achieve these high level goals can be found in the Implementation Plan. -In order to continue to provide the full summary API, either the kubelet or a stand-alone version of cAdvisor will need to publish these metrics. +In order to continue to provide the full summary API, eventually a stand-alone version of cAdvisor will need to publish these metrics. This Core Metrics API will be versioned to account for version-skew between kubernetes components. @@ -240,8 +240,6 @@ type FilesystemUsage struct { In addition to providing metrics, cAdvisor also provides machine info. While it is not neccessary to use this structure for reporting Machine Info, the following contains all of the information provided to the kubelet by cAdvisor, generally used at startup. -The code that provides this data currently resides in cAdvisor. I propose moving this to the kubelet as well. - ```go type MachineInfo struct { // MachineID reported by the node. For unique machine identification From 29a4bf8377d2a91c38a24ec8bcf3b3eb490587db Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 14:16:38 -0800 Subject: [PATCH 05/31] s/WorkingsetBytes/UsedBytes --- contributors/design-proposals/core-metrics-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 1d7c757e2f0..709f086be5a 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -209,7 +209,7 @@ type MemoryUsage struct { // The amount of working set memory. This includes recently accessed memory, // dirty memory, and kernel memory. // +optional - WorkingSetBytes *uint64 `json:"workingSetBytes,omitempty"` + UsageBytes *uint64 `json:"usageBytes,omitempty"` } // VolumeUsage holds statistics about the quantity of disk resources consumed for a volume From 25abf52882e246d38a3b04b6c9d02dc1ecd2e637 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 14:24:13 -0800 Subject: [PATCH 06/31] s/I/@dashpole --- contributors/design-proposals/core-metrics-pipeline.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 709f086be5a..5e79566fce4 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -280,9 +280,10 @@ const ( ## Implementation Plan -I will create a separate endpoint TBD to publish this set of core metrics. -I will modify volume stats collection so that it relies on this code. -I will modify the structure of stats collection code to be "On-Demand". +@dashpole will internally separate core metrics from summary metrics and make the kubelet use the core metrics. +@dashpole will create a separate endpoint TBD to publish this set of core metrics. +@dashpole will modify volume stats collection so that it relies on this code. +@dashpole will modify the structure of stats collection code to be "On-Demand". Tenative future work, not included in this proposal: Obtain all runtime-specific information needed to collect metrics from the CRI. From 11ea97f9dd87be40ea60395bf3d4aa876eb7de39 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 14:29:05 -0800 Subject: [PATCH 07/31] node-level cpu metrics in kubelet requirements --- contributors/design-proposals/core-metrics-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 5e79566fce4..25a557a3696 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -61,7 +61,7 @@ High level requirements for the design are as follows: Metrics requirements, based on kubernetes component needs, are as follows: - Kubelet - - Node-level capacity and availability metrics for Disk and Memory + - Node-level capacity and availability metrics for Disk, Memory, and CPU - Pod-level usage metrics for Disk and Memory - Scheduler - Node-level capacity and availability metrics for Disk, CPU, and Memory From 13eee17dd2014fa714255124568ecf0670f339a9 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 5 Jan 2017 14:58:40 -0800 Subject: [PATCH 08/31] Improve memory and cpu documentation --- contributors/design-proposals/core-metrics-pipeline.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 25a557a3696..4c7120c0426 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -128,6 +128,8 @@ type MemoryResources struct { // The memory capacity, in bytes CapacityBytes *uint64 `json:"capacitybytes,omitempty"` // The available memory, in bytes + // This is the number of bytes which are not included in the working set memory + // Working set memory includes recently accessed memory, dirty memory, and kernel memory. AvailableBytes *uint64 `json:"availablebytes,omitempty"` } @@ -195,11 +197,13 @@ type CpuUsage struct { Timestamp metav1.Time `json:"time"` // Total CPU usage (sum of all cores) averaged over the sample window. // The "core" unit can be interpreted as CPU core-nanoseconds per second. + // For example, a value of 5 means that cpu is consumed at a rate of + // 5 core-nanoseconds per second during the sample window. // +optional - UsageNanoCores *uint64 `json:"usageNanoCores,omitempty"` + UsageRateNanoCores *uint64 `json:"usageNanoCores,omitempty"` // Cumulative CPU usage (sum of all cores) since object creation. // +optional - UsageCoreNanoSeconds *uint64 `json:"usageCoreNanoSeconds,omitempty"` + AggregateUsageCoreNanoSeconds *uint64 `json:"usageCoreNanoSeconds,omitempty"` } // MemoryUsage holds statistics about the quantity of memory consumed From 919764e306b86cf31f62eb50bab5a45d01ebcc7d Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 6 Jan 2017 11:09:27 -0800 Subject: [PATCH 09/31] summary API change, cpu description --- .../design-proposals/core-metrics-pipeline.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 4c7120c0426..8fbd2a178c0 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -55,7 +55,7 @@ Integration with CRI will not be covered in this proposal. In future proposals, This design covers only the internal Core Metrics Pipeline. High level requirements for the design are as follows: - - Do not break existing users. We should continue to provide the full summary API by default. + - Do not break existing users. We should continue to provide the full summary API as an optional add-on. Once the monitoring pipeline is completed, the summary API will be provided by the monitoring pipeline, possibly through a stand-alone version of cAdvisor. - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. @@ -75,8 +75,6 @@ Metrics requirements, based on kubernetes component needs, are as follows: More details on how I intend to achieve these high level goals can be found in the Implementation Plan. -In order to continue to provide the full summary API, eventually a stand-alone version of cAdvisor will need to publish these metrics. - This Core Metrics API will be versioned to account for version-skew between kubernetes components. This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed for an internal use case, they can be added to the core metrics API. @@ -195,10 +193,10 @@ type ContainerUsage struct { type CpuUsage struct { // The time at which these stats were updated. Timestamp metav1.Time `json:"time"` - // Total CPU usage (sum of all cores) averaged over the sample window. - // The "core" unit can be interpreted as CPU core-nanoseconds per second. - // For example, a value of 5 means that cpu is consumed at a rate of - // 5 core-nanoseconds per second during the sample window. + // Average CPU usage rate over sample window (across all cores), in "cores". + // The "core" unit represents nanoseconds of CPU time consumed per second. + // For example, 5 nanocores means the process averaged 5 nanoseconds + // of cpu time per second during the sample window. // +optional UsageRateNanoCores *uint64 `json:"usageNanoCores,omitempty"` // Cumulative CPU usage (sum of all cores) since object creation. From 04dc0aa7a2b4360e19ae7723c438732a2030e218 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 6 Jan 2017 11:17:33 -0800 Subject: [PATCH 10/31] Include Resource Metrics API --- .../design-proposals/core-metrics-pipeline.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 8fbd2a178c0..30f390e39ec 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -63,15 +63,23 @@ Metrics requirements, based on kubernetes component needs, are as follows: - Kubelet - Node-level capacity and availability metrics for Disk, Memory, and CPU - Pod-level usage metrics for Disk and Memory - - Scheduler + - Scheduler (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) - Node-level capacity and availability metrics for Disk, CPU, and Memory - Pod-level usage metrics for Disk, CPU, and Memory - Container-level usage metrics for Disk, CPU, and Memory - - Horizontal-Pod-Autoscaler (HPA) + - Horizontal-Pod-Autoscaler (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md))) - Node-level capacity and availability metrics for CPU and Memory - Pod-level usage metrics for CPU and Memory - - Master metrics API - - + - Cluster Federation (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - kubectl top (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk, Memory, and CPU + - Container-level usage metrics for Disk, CPU, and Memory + - Kubernetes Dashboard (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk, Memory, and CPU + - Container-level usage metrics for Disk, CPU, and Memory More details on how I intend to achieve these high level goals can be found in the Implementation Plan. From fdcfed7a01c8a050ea98f951186af2952ee4db62 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 6 Jan 2017 11:25:37 -0800 Subject: [PATCH 11/31] Configurable interval --- contributors/design-proposals/core-metrics-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 30f390e39ec..9cc9064889f 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -30,7 +30,7 @@ This document proposes a design for an internal Core Metrics Pipeline. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. -cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds), and the kubelet then simply queries these cached metrics whenever it has a need for them. +cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet summary API is published to the kubelet summary API endpoint. Some of the metrics provided by the summary API are consumed internally, but most are not used internally. From 40a60f2810e688d6297be82133540fa6ad206bd2 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 6 Jan 2017 11:36:17 -0800 Subject: [PATCH 12/31] Define summary API --- contributors/design-proposals/core-metrics-pipeline.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 9cc9064889f..b847afee409 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -30,9 +30,9 @@ This document proposes a design for an internal Core Metrics Pipeline. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. -cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the Summary API. cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. +cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. -Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet summary API is published to the kubelet summary API endpoint. Some of the metrics provided by the summary API are consumed internally, but most are not used internally. +Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed internally, but most are not used internally. ### Motivations From d030ef9407f5e8f5257b2d85be407055508ab761 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 6 Jan 2017 11:42:34 -0800 Subject: [PATCH 13/31] removed proto annotation --- .../design-proposals/core-metrics-pipeline.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index b847afee409..69ef5677fd7 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -255,22 +255,22 @@ type MachineInfo struct { // MachineID reported by the node. For unique machine identification // in the cluster this field is prefered. Learn more from man(5) // machine-id: http://man7.org/linux/man-pages/man5/machine-id.5.html - MachineID string `json:"machineID" protobuf:"bytes,1,opt,name=machineID"` + MachineID string `json:"machineID"` // SystemUUID reported by the node. For unique machine identification // MachineID is prefered. This field is specific to Red Hat hosts // https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/getting-system-uuid.html - SystemUUID string `json:"systemUUID" protobuf:"bytes,2,opt,name=systemUUID"` + SystemUUID string `json:"systemUUID"` // Boot ID reported by the node. - BootID string `json:"bootID" protobuf:"bytes,3,opt,name=bootID"` + BootID string `json:"bootID"` // Kernel Version reported by the node from 'uname -r' (e.g. 3.16.0-0.bpo.4-amd64). - KernelVersion string `json:"kernelVersion" protobuf:"bytes,4,opt,name=kernelVersion"` + KernelVersion string `json:"kernelVersion"` // OS Image reported by the node from /etc/os-release (e.g. Debian GNU/Linux 7 (wheezy)). - OSImage string `json:"osImage" protobuf:"bytes,5,opt,name=osImage"` + OSImage string `json:"osImage"` // Capacity represents the total resources of a node. // More info: http://kubernetes.io/docs/user-guide/persistent-volumes#capacity for more details. // +optional // ContainerRuntime Version reported by the node through runtime remote API (e.g. docker://1.5.0). - ContainerRuntimeVersion string `json:"containerRuntimeVersion" protobuf:"bytes,6,opt,name=containerRuntimeVersion"` + ContainerRuntimeVersion string `json:"containerRuntimeVersion"` // Cloud provider the machine belongs to. CloudProvider CloudProvider `json:"cloud_provider"` // ID of cloud instance (e.g. instance-1) given to it by the cloud provider. From 54eaae8785d86d5ed5c24ec16ed64954d64bafde Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 6 Jan 2017 11:48:26 -0800 Subject: [PATCH 14/31] update tentative future plans --- contributors/design-proposals/core-metrics-pipeline.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 69ef5677fd7..a6dfb7e244c 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -295,10 +295,10 @@ const ( @dashpole will modify volume stats collection so that it relies on this code. @dashpole will modify the structure of stats collection code to be "On-Demand". -Tenative future work, not included in this proposal: -Obtain all runtime-specific information needed to collect metrics from the CRI. -Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. It will consume the above metadata API, and provide the summary API. -The kubelet no longer provides the summary API, and starts cAdvisor stand-alone by default. Include flag to not start cAdvisor. +Suggested, tentative future work, which may be covered by future proposals: + - Obtain all runtime-specific information needed to collect metrics from the CRI. + - Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. It will consume the above metadata API, and provide the summary API. + - The kubelet no longer provides the summary API, and starts cAdvisor stand-alone by default. Include flag to not start stand-alone cAdvisor. ## Rollout Plan From 53c3b18e163f01185133576afc297a2bdac08436 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 10 Jan 2017 09:59:45 -0800 Subject: [PATCH 15/31] Feature based --- .../design-proposals/core-metrics-pipeline.md | 83 ++++++------------- 1 file changed, 24 insertions(+), 59 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index a6dfb7e244c..6a5e0de06ba 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -17,8 +17,8 @@ This document proposes a design for an internal Core Metrics Pipeline. - [Proposal](#proposal) - [Non Goals](#non-goals) - [Design](#design) + - [Metric Requirements:](#metric-requirements) - [Proposed Core Metrics API:](#proposed-core-metrics-api) - - [Core Machine Info:](#core-machine-info) - [Implementation Plan](#implementation-plan) - [Rollout Plan](#rollout-plan) - [Implementation Status](#implementation-status) @@ -32,34 +32,46 @@ The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/mast cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. -Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed internally, but most are not used internally. +Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but most are not for this purpose. ### Motivations - The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. + + cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal - -I propose to use this set of core metrics, collected by the kubelet, and used solely by internal kubernetes components. +I propose to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system compenents to support resource feasibility checking and resource management on the node. ### Non Goals - Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. ## Design - -This design covers only the internal Core Metrics Pipeline. +This design covers only the Core Metrics Pipeline. High level requirements for the design are as follows: - Do not break existing users. We should continue to provide the full summary API as an optional add-on. Once the monitoring pipeline is completed, the summary API will be provided by the monitoring pipeline, possibly through a stand-alone version of cAdvisor. - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. -Metrics requirements, based on kubernetes component needs, are as follows: +More details on how I intend to achieve these high level goals can be found in the Implementation Plan. + +This Core Metrics API will be versioned to account for version-skew between kubernetes components. + +This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support resource feasibility checking or node resource management, they can be added to the core metrics API. + +### Metric Requirements +The core metrics api is designed to provide metrics for two use-cases within kubernetes: + - Resource Feasibility Checking + - Node Resource Management + +Many kubernetes system components currently support these features. Many more components that support these features are in development. +The following is meant not meant to be an exhaustive list, but gives the current set of use cases for these metrics. + +Metrics requirements for resource feasibility checking and node resource management, based on kubernetes component needs, are as follows: - Kubelet - Node-level capacity and availability metrics for Disk, Memory, and CPU - Pod-level usage metrics for Disk and Memory @@ -81,11 +93,6 @@ Metrics requirements, based on kubernetes component needs, are as follows: - Pod-level usage metrics for Disk, Memory, and CPU - Container-level usage metrics for Disk, CPU, and Memory -More details on how I intend to achieve these high level goals can be found in the Implementation Plan. - -This Core Metrics API will be versioned to account for version-skew between kubernetes components. - -This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed for an internal use case, they can be added to the core metrics API. ### Proposed Core Metrics API: @@ -246,53 +253,11 @@ type FilesystemUsage struct { } ``` -### Core Machine Info: - -In addition to providing metrics, cAdvisor also provides machine info. While it is not neccessary to use this structure for reporting Machine Info, the following contains all of the information provided to the kubelet by cAdvisor, generally used at startup. - -```go -type MachineInfo struct { - // MachineID reported by the node. For unique machine identification - // in the cluster this field is prefered. Learn more from man(5) - // machine-id: http://man7.org/linux/man-pages/man5/machine-id.5.html - MachineID string `json:"machineID"` - // SystemUUID reported by the node. For unique machine identification - // MachineID is prefered. This field is specific to Red Hat hosts - // https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/getting-system-uuid.html - SystemUUID string `json:"systemUUID"` - // Boot ID reported by the node. - BootID string `json:"bootID"` - // Kernel Version reported by the node from 'uname -r' (e.g. 3.16.0-0.bpo.4-amd64). - KernelVersion string `json:"kernelVersion"` - // OS Image reported by the node from /etc/os-release (e.g. Debian GNU/Linux 7 (wheezy)). - OSImage string `json:"osImage"` - // Capacity represents the total resources of a node. - // More info: http://kubernetes.io/docs/user-guide/persistent-volumes#capacity for more details. - // +optional - // ContainerRuntime Version reported by the node through runtime remote API (e.g. docker://1.5.0). - ContainerRuntimeVersion string `json:"containerRuntimeVersion"` - // Cloud provider the machine belongs to. - CloudProvider CloudProvider `json:"cloud_provider"` - // ID of cloud instance (e.g. instance-1) given to it by the cloud provider. - InstanceID InstanceID `json:"instance_id"` -} - -type CloudProvider string - -const ( - GCE CloudProvider = "GCE" - AWS = "AWS" - Azure = "Azure" - Baremetal = "Baremetal" - UnknownProvider = "Unknown" -) -``` - ## Implementation Plan -@dashpole will internally separate core metrics from summary metrics and make the kubelet use the core metrics. -@dashpole will create a separate endpoint TBD to publish this set of core metrics. -@dashpole will modify volume stats collection so that it relies on this code. +@dashpole will internally separate core metrics from summary metrics and make the kubelet use the core metrics. +@dashpole will create a separate endpoint TBD to publish this set of core metrics. +@dashpole will modify volume stats collection so that it relies on this code. @dashpole will modify the structure of stats collection code to be "On-Demand". Suggested, tentative future work, which may be covered by future proposals: From 2f1d746bf06b18434db957bc6c70f31643c878b8 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 10 Jan 2017 15:50:34 -0800 Subject: [PATCH 16/31] Add Definitions section. Add users description. --- .../design-proposals/core-metrics-pipeline.md | 20 ++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 6a5e0de06ba..c8cea8867e0 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -2,7 +2,7 @@ **Author**: David Ashpole (@dashpole) -**Last Updated**: 12/21/2016 +**Last Updated**: 1/10/2017 **Status**: Draft Proposal (WIP) @@ -12,6 +12,7 @@ This document proposes a design for an internal Core Metrics Pipeline. - [Core Metrics Pipeline in kubelet](#core-metrics-pipeline-in-kubelet) - [Introduction](#introduction) + - [Definitions](#definitions) - [Background](#background) - [Motivations](#motivations) - [Proposal](#proposal) @@ -27,25 +28,34 @@ This document proposes a design for an internal Core Metrics Pipeline. ## Introduction +### Definitions +"Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things. +["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. +["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. +["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). +"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. + ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. -cAdvisor is an open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. Kubernetes vendors cAdvisor into its codebase, and uses cAdvisor as a library with functions that enable it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs like pods to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. +Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but most are not for this purpose. ### Motivations The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. - +By publishing core metrics, the summary API is relieved of its responsibility to provide metrics to system components. This will allow the summary API to evolve into a much richer set of metrics for monitoring, since the overhead burden falls on the third party monitoring pipeline. cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal I propose to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system compenents to support resource feasibility checking and resource management on the node. +The target "Users" of this set of metrics are kubernetes components. This set of metrics does not neccessarily need to be user-friendly, but should be flexible enough to be turned into user friendly metrics. + ### Non Goals -Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. +Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc will not be covered in this proposal. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. @@ -75,7 +85,7 @@ Metrics requirements for resource feasibility checking and node resource managem - Kubelet - Node-level capacity and availability metrics for Disk, Memory, and CPU - Pod-level usage metrics for Disk and Memory - - Scheduler (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) + - Scheduler (Possibly through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) - Node-level capacity and availability metrics for Disk, CPU, and Memory - Pod-level usage metrics for Disk, CPU, and Memory - Container-level usage metrics for Disk, CPU, and Memory From b059fd428ae44a9776515606cdbb5b59734303ac Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 11 Jan 2017 10:50:44 -0800 Subject: [PATCH 17/31] cleanup language, formatting; rollout plan; --- .../design-proposals/core-metrics-pipeline.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index c8cea8867e0..119e7143cc6 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -29,11 +29,11 @@ This document proposes a design for an internal Core Metrics Pipeline. ## Introduction ### Definitions -"Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things. -["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. -["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. -["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). -"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. +"Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things. +["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. +["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. +["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). +"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. @@ -52,7 +52,7 @@ cAdvisor is structured to collect metrics on an interval, which is appropriate f ### Proposal I propose to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system compenents to support resource feasibility checking and resource management on the node. -The target "Users" of this set of metrics are kubernetes components. This set of metrics does not neccessarily need to be user-friendly, but should be flexible enough to be turned into user friendly metrics. +The target "Users" of this set of metrics are kubernetes components. This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. ### Non Goals Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc will not be covered in this proposal. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. @@ -106,7 +106,7 @@ Metrics requirements for resource feasibility checking and node resource managem ### Proposed Core Metrics API: -An important difference between the current summary api and the proposed core metrics api is that per-pod stats in the core metrics api contain only usage data, and not capacity-related statistics. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how the kubelet uses the data. The kubelet finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of stats. +An important difference between the current summary api and the proposed core metrics api is that per-pod stats in the core metrics api contain only usage data, and not capacity-related statistics. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how components use these metrics. The kubelet, for example, finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of stats. ```go // CoreStats is a top-level container for holding NodeStats and PodStats. @@ -273,11 +273,11 @@ type FilesystemUsage struct { Suggested, tentative future work, which may be covered by future proposals: - Obtain all runtime-specific information needed to collect metrics from the CRI. - Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. It will consume the above metadata API, and provide the summary API. - - The kubelet no longer provides the summary API, and starts cAdvisor stand-alone by default. Include flag to not start stand-alone cAdvisor. + - The kubelet no longer provides the summary API, and starts, by default, cAdvisor stand-alone (which provides the summary API). Include flag to disable running stand-alone cAdvisor. ## Rollout Plan - -TBD +The core metrics endpoint (TBD) will be added alongside the current Summary API for the upcoming release. This should allow concurrent developments of other portions of the system metrics pipeline (metrics-server, for example). Once this addition is made, all other changes will be internal, and will not require any API changes. +Once the [implementation work](#implementation-plan) is completed, @dashpole will start discussions on how to provide the summary API through a means separate from the kubelet. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. ## Implementation Status From 9e87a2f7ec30e89d7eefb60f73f66110ee076a76 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 11 Jan 2017 13:47:03 -0800 Subject: [PATCH 18/31] Addressed comments about language --- contributors/design-proposals/core-metrics-pipeline.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 119e7143cc6..141e37849c2 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -36,7 +36,7 @@ This document proposes a design for an internal Core Metrics Pipeline. "Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. ### Background -The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and how they will be collected on the node. +The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and by what means they are exposed to system components. Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. @@ -50,7 +50,7 @@ By publishing core metrics, the summary API is relieved of its responsibility to cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal -I propose to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system compenents to support resource feasibility checking and resource management on the node. +This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support resource feasibility checking and resource management on the node. The target "Users" of this set of metrics are kubernetes components. This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. @@ -67,7 +67,7 @@ High level requirements for the design are as follows: - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. -More details on how I intend to achieve these high level goals can be found in the Implementation Plan. +More details on how these high level goals will be achieved can be found in the Implementation Plan. This Core Metrics API will be versioned to account for version-skew between kubernetes components. @@ -79,7 +79,7 @@ The core metrics api is designed to provide metrics for two use-cases within kub - Node Resource Management Many kubernetes system components currently support these features. Many more components that support these features are in development. -The following is meant not meant to be an exhaustive list, but gives the current set of use cases for these metrics. +The following is not meant to be an exhaustive list, but gives the current set of use cases for these metrics. Metrics requirements for resource feasibility checking and node resource management, based on kubernetes component needs, are as follows: - Kubelet @@ -141,7 +141,7 @@ type CpuResources struct { // The number of cores in this machine. NumCores int `json:"numcores"` // The current Usage of CPU resources - TotalUsage *CpuUsage `json:"cpuusage,omitempty"` + Usage *CpuUsage `json:"cpuusage,omitempty"` } // MemoryResources contains data about memory resource usage. From a106dd6e7e445b3f99b0690cce169970583c9450 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 12 Jan 2017 13:46:41 -0800 Subject: [PATCH 19/31] address comments, remove json --- .../design-proposals/core-metrics-pipeline.md | 231 ++++++++---------- 1 file changed, 104 insertions(+), 127 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 141e37849c2..55c3c88f6db 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -30,22 +30,26 @@ This document proposes a design for an internal Core Metrics Pipeline. ### Definitions "Kubelet": The daemon that runs on every kubernetes node and controls pod and container lifecycle, among many other things. -["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of k8s constructs like pods or volumes. +["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of kubernetes constructs like pods or volumes. ["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. ["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). "Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. +"Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc). +"First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec. +"Metric": A measure of consumption of a Resource. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and by what means they are exposed to system components. -Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of k8s constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. +Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of kubernetes constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but most are not for this purpose. ### Motivations The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. -By publishing core metrics, the summary API is relieved of its responsibility to provide metrics to system components. This will allow the summary API to evolve into a much richer set of metrics for monitoring, since the overhead burden falls on the third party monitoring pipeline. +By publishing core metrics, the kubelet is relieved of its responsibility to provide metrics to system components. +The third party monitoring pipeline also is relieved of any responsibility to provide these metrics to system components. cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. @@ -63,7 +67,7 @@ Integration with CRI will not be covered in this proposal. In future proposals, This design covers only the Core Metrics Pipeline. High level requirements for the design are as follows: - - Do not break existing users. We should continue to provide the full summary API as an optional add-on. Once the monitoring pipeline is completed, the summary API will be provided by the monitoring pipeline, possibly through a stand-alone version of cAdvisor. + - Do not break existing users. We should continue to provide the full summary API as an optional add-on for the forseeable future. Once the monitoring pipeline is completed, the summary API, or a suitable replacement, will be provided by the monitoring pipeline, possibly through a stand-alone version of cAdvisor. - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. @@ -74,169 +78,145 @@ This Core Metrics API will be versioned to account for version-skew between kube This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support resource feasibility checking or node resource management, they can be added to the core metrics API. ### Metric Requirements -The core metrics api is designed to provide metrics for two use-cases within kubernetes: - - Resource Feasibility Checking - - Node Resource Management +The core metrics api is designed to provide metrics for "First Class Resource Isolation and Utilization Features" within kubernetes. Many kubernetes system components currently support these features. Many more components that support these features are in development. The following is not meant to be an exhaustive list, but gives the current set of use cases for these metrics. -Metrics requirements for resource feasibility checking and node resource management, based on kubernetes component needs, are as follows: +Metrics requirements for "First Class Resource Isolation and Utilization Features", based on kubernetes component needs, are as follows: - Kubelet - - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Node-level availability metrics for Disk, Memory, and CPU - Pod-level usage metrics for Disk and Memory - - Scheduler (Possibly through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) - - Node-level capacity and availability metrics for Disk, CPU, and Memory - - Pod-level usage metrics for Disk, CPU, and Memory - - Container-level usage metrics for Disk, CPU, and Memory - - Horizontal-Pod-Autoscaler (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md))) - - Node-level capacity and availability metrics for CPU and Memory - - Pod-level usage metrics for CPU and Memory - - Cluster Federation (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - kubectl top (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk, Memory, and CPU - - Container-level usage metrics for Disk, CPU, and Memory - - Kubernetes Dashboard (Exposed through [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)) - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk, Memory, and CPU - - Container-level usage metrics for Disk, CPU, and Memory + - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: + - Scheduler + - Node-level capacity and availability metrics for Disk, CPU, and Memory + - Pod-level usage metrics for Disk, CPU, and Memory + - Container-level usage metrics for Disk, CPU, and Memory + - Horizontal-Pod-Autoscaler + - Node-level capacity and availability metrics for CPU and Memory + - Pod-level usage metrics for CPU and Memory + - Cluster Federation + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - kubectl top and Kubernetes Dashboard + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk, Memory, and CPU + - Container-level usage metrics for Disk, CPU, and Memory ### Proposed Core Metrics API: -An important difference between the current summary api and the proposed core metrics api is that per-pod stats in the core metrics api contain only usage data, and not capacity-related statistics. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how components use these metrics. The kubelet, for example, finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of stats. +An important difference between the current summary api and the proposed core metrics api is that metrics in the core metrics api contain only usage data, and not capacity-related statistics. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how components use these metrics. The kubelet, for example, finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of metrics. ```go -// CoreStats is a top-level container for holding NodeStats and PodStats. -type CoreStats struct { - // Overall node resource stats. - Node NodeResources `json:"node"` - // Per-pod usage stats. - Pods []PodUsage `json:"pods"` +// CoreMetrics is a top-level container for holding NodeResources and PodUsage. +type CoreMetrics struct { + // Overall node resource metrics. + Node NodeResources + // Per-pod usage metrics. + Pods []PodUsage } -// NodeStats holds node-level stats. NodeStats contains capacity and availibility for Node Resources. -type NodeResources struct { - // The filesystem device used by node k8s components. - // +optional - KubeletFsDevice string `json:"kubeletfs,omitempty"` - // The filesystem device used by node runtime components. - // +optional - RuntimeFsDevice string `json:"runtimefs,omitempty"` - // Stats pertaining to cpu resources. - // +optional - CPU *CpuResources `json:"cpu,omitempty"` - // Stats pertaining to memory (RAM) resources. - // +optional - Memory *MemoryResources `json:"memory,omitempty"` - // Stats pertaining to node filesystem resources. - // +optional - Filesystems []FilesystemResources `json:"filesystems, omitempty" patchStrategy:"merge" patchMergeKey:"device"` +// NodeResources holds node-level metrics. NodeResources contains capacity and availibility for Node Resources. +type NodeResources struct { + // The filesystem used by node kubernetes components. + KubeletFilesystem string + // The filesystem used by node runtime components. + RuntimeFilesystem string + // Metrics pertaining to cpu resources. + CPU *CpuResources + // Metrics pertaining to memory (RAM) resources. + Memory *MemoryResources + // Metrics pertaining to node filesystem resources. + Filesystems []FilesystemResources } -// CpuResources containes data about cpu resource usage -type CpuResources struct { +// CpuResources containes data about overall cpu resource capacity and usage +type CpuResources struct { // The number of cores in this machine. - NumCores int `json:"numcores"` + NumCores int `json:"numcores"` // The current Usage of CPU resources - Usage *CpuUsage `json:"cpuusage,omitempty"` + Usage *CpuUsage } // MemoryResources contains data about memory resource usage. type MemoryResources struct { - // The time at which these stats were updated. - Timestamp metav1.Time `json:"time"` + // The time at which these metrics were updated. + Timestamp metav1.Time // The memory capacity, in bytes - CapacityBytes *uint64 `json:"capacitybytes,omitempty"` + CapacityBytes *uint64 // The available memory, in bytes // This is the number of bytes which are not included in the working set memory // Working set memory includes recently accessed memory, dirty memory, and kernel memory. - AvailableBytes *uint64 `json:"availablebytes,omitempty"` + AvailableBytes *uint64 } // FilesystemResources contains data about filesystem disk resources. type FilesystemResources struct { - // The time at which these stats were updated. - Timestamp metav1.Time `json:"time"` - // The device that this filesystem is on - Device string `json:"device"` + // The time at which these metrics were updated. + Timestamp metav1.Time + // The filesystem name, which uniquely identifies a filesystem + Filesystem string // AvailableBytes represents the storage space available (bytes) for the filesystem. - // +optional - AvailableBytes *uint64 `json:"availableBytes,omitempty"` + AvailableBytes *uint64 // CapacityBytes represents the total capacity (bytes) of the filesystems underlying storage. - // +optional - CapacityBytes *uint64 `json:"capacityBytes,omitempty"` + CapacityBytes *uint64 // InodesFree represents the free inodes in the filesystem. - // +optional - InodesFree *uint64 `json:"inodesFree,omitempty"` + InodesFree *uint64 // Inodes represents the total inodes in the filesystem. - // +optional - Inodes *uint64 `json:"inodes,omitempty"` + Inodes *uint64 } -// PodUsage holds pod-level unprocessed sample stats. +// PodUsage holds pod-level metrics. type PodUsage struct { // UID of the pod - PodUID string `json:"uid"` - // Stats pertaining to pod total usage of cpu + PodUID string + // Metrics pertaining to pod total usage of cpu // This may include additional overhead not included in container usage statistics. - // +optional - CPU *CpuUsage `json:"cpu,omitempty"` - // Stats pertaining to pod total usage of system memory + CPU *CpuUsage + // Metrics pertaining to pod total usage of system memory // This may include additional overhead not included in container usage statistics. - // +optional - Memory *MemoryUsage `json:"memory,omitempty"` - // Stats of containers in the pod. - Containers []ContainerUsage `json:"containers" patchStrategy:"merge" patchMergeKey:"uid"` - // Stats pertaining to volume usage of filesystem resources. - // +optional - Volumes []VolumeUsage `json:"volume,omitempty" patchStrategy:"merge" patchMergeKey:"name"` + Memory *MemoryUsage + // Metrics of containers in the pod. + Containers []ContainerUsage + // Metrics pertaining to volume usage of filesystem resources. + Volumes []VolumeUsage } -// ContainerUsage holds container-level usage stats. +// ContainerUsage holds container-level usage metrics. type ContainerUsage struct { - // UID of the container - ContainerUID string `json:"uid"` - // Stats pertaining to container usage of cpu - // +optional - CPU *CpuUsage `json:"memory,omitempty"` - // Stats pertaining to container usage of system memory - // +optional - Memory *MemoryUsage `json:"memory,omitempty"` - // Stats pertaining to container rootfs usage of disk. + // ID of the container + ContainerID string + // Metrics pertaining to container usage of cpu + CPU *CpuUsage + // Metrics pertaining to container usage of system memory + Memory *MemoryUsage + // Metrics pertaining to container rootfs usage of disk. // Rootfs.UsedBytes is the number of bytes used for the container write layer. - // +optional - Rootfs *FilesystemUsage `json:"rootfs,omitempty"` - // Stats pertaining to container logs usage of Disk. - // +optional - Logs *FilesystemUsage `json:"logs,omitempty"` + Rootfs *FilesystemUsage + // Metrics pertaining to container logs usage of Disk. + Logs *FilesystemUsage } // CpuUsage holds statistics about the amount of cpu time consumed type CpuUsage struct { - // The time at which these stats were updated. - Timestamp metav1.Time `json:"time"` - // Average CPU usage rate over sample window (across all cores), in "cores". + // The time at which these Metrics were updated. + Timestamp metav1.Time + // Average CPU usage rate over sample window (across all cores), in "nano cores". // The "core" unit represents nanoseconds of CPU time consumed per second. // For example, 5 nanocores means the process averaged 5 nanoseconds // of cpu time per second during the sample window. - // +optional - UsageRateNanoCores *uint64 `json:"usageNanoCores,omitempty"` + UsageRateNanoCores *uint64 // Cumulative CPU usage (sum of all cores) since object creation. - // +optional - AggregateUsageCoreNanoSeconds *uint64 `json:"usageCoreNanoSeconds,omitempty"` + AggregateUsageCoreNanoSeconds *uint64 } // MemoryUsage holds statistics about the quantity of memory consumed type MemoryUsage struct { - // The time at which these stats were updated. - Timestamp metav1.Time `json:"time"` + // The time at which these metrics were updated. + Timestamp metav1.Time // The amount of working set memory. This includes recently accessed memory, // dirty memory, and kernel memory. - // +optional - UsageBytes *uint64 `json:"usageBytes,omitempty"` + WorkingSetBytes *uint64 } // VolumeUsage holds statistics about the quantity of disk resources consumed for a volume @@ -244,35 +224,32 @@ type VolumeUsage struct { // Embedded FilesystemUsage FilesystemUsage // Name is the name given to the Volume - // +optional - Name string `json:"name,omitempty"` + Name string } // FilesystemUsage holds statistics about the quantity of disk resources consumed type FilesystemUsage struct { - // The time at which these stats were updated. - Timestamp metav1.Time `json:"time"` - // The device on which resources are consumed - Device string `json:"device"` - // UsedBytes represents the disk space consumed on the device, in bytes. - // +optional - UsedBytes *uint64 `json:"usedBytes,omitempty"` - // InodesUsed represents the inodes consumed on the device - // +optional - InodesUsed *uint64 `json:"inodesUsed,omitempty"` + // The time at which these metrics were updated. + Timestamp metav1.Time + // The name filesystme on which resources are consumed. This must uniquely identify the filesystem. + Filesystem string + // UsedBytes represents the disk space consumed on the filesystem, in bytes. + UsedBytes *uint64 + // InodesUsed represents the inodes consumed on the filesystem + InodesUsed *uint64 } ``` ## Implementation Plan -@dashpole will internally separate core metrics from summary metrics and make the kubelet use the core metrics. +@dashpole will create an interface over cAdvisor that provides core metrics. @dashpole will create a separate endpoint TBD to publish this set of core metrics. -@dashpole will modify volume stats collection so that it relies on this code. -@dashpole will modify the structure of stats collection code to be "On-Demand". +@dashpole will modify volume metrics collection so that it re-uses vendored cAdvisor libraries. +@dashpole will modify the structure of metrics collection code to be "On-Demand". Suggested, tentative future work, which may be covered by future proposals: - Obtain all runtime-specific information needed to collect metrics from the CRI. - - Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. It will consume the above metadata API, and provide the summary API. + - Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. - The kubelet no longer provides the summary API, and starts, by default, cAdvisor stand-alone (which provides the summary API). Include flag to disable running stand-alone cAdvisor. ## Rollout Plan @@ -285,8 +262,8 @@ The implementation goals of the first milestone are outlined below. - [ ] Create the proposal - [ ] Implement collection and consumption of core metrics. - [ ] Create Kubelet API endpoint for core metrics. -- [ ] Modify volume stats collection so that it relies on this code. -- [ ] Modify the structure of stats collection code to be "On-Demand" +- [ ] Modify volume metrics collection so that it re-uses vendored cAdvisor libraries. +- [ ] Modify the structure of metrics collection code to be "On-Demand" From da190b64e9842b733d4f7aa7f2bdd94783d585d3 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 12 Jan 2017 13:50:35 -0800 Subject: [PATCH 20/31] formatting --- .../design-proposals/core-metrics-pipeline.md | 45 ++++++++++--------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 55c3c88f6db..6400d28ed0c 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -34,9 +34,9 @@ This document proposes a design for an internal Core Metrics Pipeline. ["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. ["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). "Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. -"Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc). -"First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec. -"Metric": A measure of consumption of a Resource. +"Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc). +"First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec. +"Metric": A measure of consumption of a Resource. ### Background The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and by what means they are exposed to system components. @@ -83,25 +83,26 @@ The core metrics api is designed to provide metrics for "First Class Resource Is Many kubernetes system components currently support these features. Many more components that support these features are in development. The following is not meant to be an exhaustive list, but gives the current set of use cases for these metrics. -Metrics requirements for "First Class Resource Isolation and Utilization Features", based on kubernetes component needs, are as follows: - - Kubelet - - Node-level availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk and Memory - - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: - - Scheduler - - Node-level capacity and availability metrics for Disk, CPU, and Memory - - Pod-level usage metrics for Disk, CPU, and Memory - - Container-level usage metrics for Disk, CPU, and Memory - - Horizontal-Pod-Autoscaler - - Node-level capacity and availability metrics for CPU and Memory - - Pod-level usage metrics for CPU and Memory - - Cluster Federation - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - kubectl top and Kubernetes Dashboard - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk, Memory, and CPU - - Container-level usage metrics for Disk, CPU, and Memory - +Metrics requirements for "First Class Resource Isolation and Utilization Features", based on kubernetes component needs, are as follows: + +Kubelet + - Node-level availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk and Memory + +Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: + - Scheduler + - Node-level capacity and availability metrics for Disk, CPU, and Memory + - Pod-level usage metrics for Disk, CPU, and Memory + - Container-level usage metrics for Disk, CPU, and Memory + - Horizontal-Pod-Autoscaler + - Node-level capacity and availability metrics for CPU and Memory + - Pod-level usage metrics for CPU and Memory + - Cluster Federation + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - kubectl top and Kubernetes Dashboard + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk, Memory, and CPU + - Container-level usage metrics for Disk, CPU, and Memory ### Proposed Core Metrics API: From 6cf837ed9b459f81bebd7cdad7dc2e5d1cf4b1f5 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 12 Jan 2017 14:14:08 -0800 Subject: [PATCH 21/31] Clarified only usage --- contributors/design-proposals/core-metrics-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 6400d28ed0c..c9eb7096d32 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -106,7 +106,7 @@ Metrics Server (outlined in [Monitoring Architecture](https://github.com/kuberne ### Proposed Core Metrics API: -An important difference between the current summary api and the proposed core metrics api is that metrics in the core metrics api contain only usage data, and not capacity-related statistics. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how components use these metrics. The kubelet, for example, finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of metrics. +An important difference between the current summary api and the proposed core metrics api is that pod and container-level metrics in the core metrics api contain only usage data, and not capacity-related statistics. Node level resource metrics continue to provide capacity. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how components use these metrics. The kubelet, for example, finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of metrics. ```go // CoreMetrics is a top-level container for holding NodeResources and PodUsage. From d9fca71f21e9efbc5f0fd85a8b4a54442daff02d Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 12 Jan 2017 14:33:04 -0800 Subject: [PATCH 22/31] On-demand --- .../design-proposals/core-metrics-pipeline.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index c9eb7096d32..0eed10b6f09 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -20,6 +20,7 @@ This document proposes a design for an internal Core Metrics Pipeline. - [Design](#design) - [Metric Requirements:](#metric-requirements) - [Proposed Core Metrics API:](#proposed-core-metrics-api) + - [On-Demand Design:](#on-demand-design) - [Implementation Plan](#implementation-plan) - [Rollout Plan](#rollout-plan) - [Implementation Status](#implementation-status) @@ -71,8 +72,6 @@ High level requirements for the design are as follows: - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. -More details on how these high level goals will be achieved can be found in the Implementation Plan. - This Core Metrics API will be versioned to account for version-skew between kubernetes components. This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support resource feasibility checking or node resource management, they can be added to the core metrics API. @@ -241,10 +240,17 @@ type FilesystemUsage struct { } ``` -## Implementation Plan +### On-Demand Design +Interface: +The interface for exposing these metrics within the kubelet contains a method for getting this datastructure. This method contains a "recency" parameter which specifies how recently the metrics must have been computed. Kubelet components which require very up-to-date metrics (eviction, for example), use very low values. Other components use higher values. +Implementation: +To keep performance bounded while still offering metrics "On-Demand", all calls to get metrics are cached, and a minimum recency is established to prevent repeated metrics computation. Before computing new metrics, the previous metrics are checked to see if they meet the recency requirements of the caller. If the age of the metrics meet the recency requirements, then the cached metrics are returned. If not, then new metrics are computed and cached. +In the case where some metrics are not able to be computed instantly (time-averaged metrics, for example), then they must be recomputed on the "minimum recency" interval so that they always meet recency requirements. + +## Implementation Plan @dashpole will create an interface over cAdvisor that provides core metrics. -@dashpole will create a separate endpoint TBD to publish this set of core metrics. +@dashpole will create a separate, versioned endpoint TBD to publish this set of core metrics. @dashpole will modify volume metrics collection so that it re-uses vendored cAdvisor libraries. @dashpole will modify the structure of metrics collection code to be "On-Demand". From 5f636c314402d0b6f9e4d9e43106420cab309fee Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 12 Jan 2017 17:08:29 -0800 Subject: [PATCH 23/31] No longer about how metrics exposed; +formatting --- .../design-proposals/core-metrics-pipeline.md | 63 ++++++++++--------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 0eed10b6f09..8d793cab397 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -1,16 +1,16 @@ -# Core Metrics Pipeline in kubelet +# Core Metrics in kubelet **Author**: David Ashpole (@dashpole) -**Last Updated**: 1/10/2017 +**Last Updated**: 1/12/2017 -**Status**: Draft Proposal (WIP) +**Status**: Proposal -This document proposes a design for an internal Core Metrics Pipeline. +This document proposes a design for the set of metrics included in an eventual Core Metrics Pipeline. -- [Core Metrics Pipeline in kubelet](#core-metrics-pipeline-in-kubelet) +- [Core Metrics in kubelet](#core-metrics-in-kubelet) - [Introduction](#introduction) - [Definitions](#definitions) - [Background](#background) @@ -49,26 +49,28 @@ Currently, cAdvisor collects a large number of metrics related to system and con ### Motivations The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. -By publishing core metrics, the kubelet is relieved of its responsibility to provide metrics to system components. +By publishing core metrics, the kubelet is relieved of its responsibility to provide metrics for monitoring. The third party monitoring pipeline also is relieved of any responsibility to provide these metrics to system components. cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal -This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support resource feasibility checking and resource management on the node. +This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support resource feasibility checking and resource management on the node. This proposal is not designed to be an API published by the kubelet, but rather a set of metrics collected by the kubelet that will be transformed, and published in the future. -The target "Users" of this set of metrics are kubernetes components. This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. +The target "Users" of this set of metrics are kubernetes components (though not neccessarily directly). This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. ### Non Goals Everything covered in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) design doc will not be covered in this proposal. This includes the third party metrics pipeline, and the methods by which the metrics found in this proposal are provided to other kubernetes components. Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. +The kubelet API endpoint, including the format, url pattern, and name of the API will be the topic of a follow-up proposal to this proposal. + ## Design This design covers only the Core Metrics Pipeline. High level requirements for the design are as follows: - - Do not break existing users. We should continue to provide the full summary API as an optional add-on for the forseeable future. Once the monitoring pipeline is completed, the summary API, or a suitable replacement, will be provided by the monitoring pipeline, possibly through a stand-alone version of cAdvisor. + - Do not break existing users. We should continue to provide the full summary API as an optional add-on for the forseeable future. Once the monitoring pipeline is completed, the summary API, or a suitable replacement, will be provided by the third party monitoring pipeline, possibly through a stand-alone version of cAdvisor. - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. @@ -84,24 +86,23 @@ The following is not meant to be an exhaustive list, but gives the current set o Metrics requirements for "First Class Resource Isolation and Utilization Features", based on kubernetes component needs, are as follows: -Kubelet - - Node-level availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk and Memory - -Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: - - Scheduler - - Node-level capacity and availability metrics for Disk, CPU, and Memory - - Pod-level usage metrics for Disk, CPU, and Memory - - Container-level usage metrics for Disk, CPU, and Memory - - Horizontal-Pod-Autoscaler - - Node-level capacity and availability metrics for CPU and Memory - - Pod-level usage metrics for CPU and Memory - - Cluster Federation - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - kubectl top and Kubernetes Dashboard - - Node-level capacity and availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk, Memory, and CPU - - Container-level usage metrics for Disk, CPU, and Memory + - Kubelet + - Node-level availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk and Memory + - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: + - Scheduler + - Node-level capacity and availability metrics for Disk, CPU, and Memory + - Pod-level usage metrics for Disk, CPU, and Memory + - Container-level usage metrics for Disk, CPU, and Memory + - Horizontal-Pod-Autoscaler + - Node-level capacity and availability metrics for CPU and Memory + - Pod-level usage metrics for CPU and Memory + - Cluster Federation + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - kubectl top and Kubernetes Dashboard + - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Pod-level usage metrics for Disk, Memory, and CPU + - Container-level usage metrics for Disk, CPU, and Memory ### Proposed Core Metrics API: @@ -250,25 +251,25 @@ In the case where some metrics are not able to be computed instantly (time-avera ## Implementation Plan @dashpole will create an interface over cAdvisor that provides core metrics. -@dashpole will create a separate, versioned endpoint TBD to publish this set of core metrics. @dashpole will modify volume metrics collection so that it re-uses vendored cAdvisor libraries. @dashpole will modify the structure of metrics collection code to be "On-Demand". Suggested, tentative future work, which may be covered by future proposals: + - Publish these metrics in some form to a kubelet API endpoint - Obtain all runtime-specific information needed to collect metrics from the CRI. - Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. - - The kubelet no longer provides the summary API, and starts, by default, cAdvisor stand-alone (which provides the summary API). Include flag to disable running stand-alone cAdvisor. + - Kubernetes can be configured to run a default "third party metrics provider" as a daemonset. Possibly standalone cAdvisor. ## Rollout Plan +The work described here is entirely internal. However, publishing a kubelet API endpoint The core metrics endpoint (TBD) will be added alongside the current Summary API for the upcoming release. This should allow concurrent developments of other portions of the system metrics pipeline (metrics-server, for example). Once this addition is made, all other changes will be internal, and will not require any API changes. -Once the [implementation work](#implementation-plan) is completed, @dashpole will start discussions on how to provide the summary API through a means separate from the kubelet. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. +Once the [implementation work](#implementation-plan) is completed, @dashpole will start discussions on the future of the Summary API, and how to provide an out-of-the-box solution for the "third party monitoring" pipeline on the node. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. ## Implementation Status The implementation goals of the first milestone are outlined below. - [ ] Create the proposal - [ ] Implement collection and consumption of core metrics. -- [ ] Create Kubelet API endpoint for core metrics. - [ ] Modify volume metrics collection so that it re-uses vendored cAdvisor libraries. - [ ] Modify the structure of metrics collection code to be "On-Demand" From 2d5ebbb6e3b521d69965ed7e0559b683bd33a3fe Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 13 Jan 2017 15:41:41 -0800 Subject: [PATCH 24/31] changes --- contributors/design-proposals/core-metrics-pipeline.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 8d793cab397..6dffedb68cd 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -40,11 +40,11 @@ This document proposes a design for the set of metrics included in an eventual C "Metric": A measure of consumption of a Resource. ### Background -The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, and by what means they are exposed to system components. +The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal contains a blueprint for a set of metrics referred to as "Core Metrics". The purpose of this proposal is to specify what those metrics are, to enable work relating to the collection, by the kubelet, of the metrics. Kubernetes vendors cAdvisor into its codebase, and the kubelet uses cAdvisor as a library that enables it to collect metrics on containers. The kubelet can then combine container-level metrics from cAdvisor with the kubelet's knowledge of kubernetes constructs (e.g. pods) to produce the kubelet Summary statistics, which provides metrics for use by the kubelet, or by users through the [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go). cAdvisor works by collecting metrics at an interval (10 seconds, by default), and the kubelet then simply queries these cached metrics whenever it has a need for them. -Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but most are not for this purpose. +Currently, cAdvisor collects a large number of metrics related to system and container performance. However, only some of these metrics are consumed by the kubelet summary API, and many are not used. The kubelet [Summary API](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) is published to the kubelet summary API endpoint (stats/summary). Some of the metrics provided by the summary API are consumed by kubernetes system components, but many are included for the sole purpose of providing metrics for monitoring. ### Motivations The [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) proposal explains why a separate monitoring pipeline is required. From 005669dafcd767a744469772941ed52ce59eb086 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 13 Jan 2017 16:41:57 -0800 Subject: [PATCH 25/31] Refined language --- .../design-proposals/core-metrics-pipeline.md | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 6dffedb68cd..40c20e8118e 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -55,7 +55,7 @@ The third party monitoring pipeline also is relieved of any responsibility to pr cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal -This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support resource feasibility checking and resource management on the node. This proposal is not designed to be an API published by the kubelet, but rather a set of metrics collected by the kubelet that will be transformed, and published in the future. +This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support "First Class Resource Isolation and Utilization Features". This proposal is not designed to be an API published by the kubelet, but rather a set of metrics collected by the kubelet that will be transformed, and published in the future. The target "Users" of this set of metrics are kubernetes components (though not neccessarily directly). This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. @@ -64,18 +64,16 @@ Everything covered in the [Monitoring Architecture](https://github.com/kubernete Integration with CRI will not be covered in this proposal. In future proposals, integrating with CRI may provide a better abstraction of information required by the core metrics pipeline to collect metrics. -The kubelet API endpoint, including the format, url pattern, and name of the API will be the topic of a follow-up proposal to this proposal. +The kubelet API endpoint, including the format, url pattern, versioning strategy, and name of the API will be the topic of a follow-up proposal to this proposal. ## Design This design covers only the Core Metrics Pipeline. High level requirements for the design are as follows: - - Do not break existing users. We should continue to provide the full summary API as an optional add-on for the forseeable future. Once the monitoring pipeline is completed, the summary API, or a suitable replacement, will be provided by the third party monitoring pipeline, possibly through a stand-alone version of cAdvisor. - - The kubelet collects the minimum possible number of metrics for complete portable kubernetes functionalities. + - The kubelet collects the minimum possible number of metrics to provide "First Class Resource Isolation and Utilization Features". + - Do not break existing users. We should continue to provide the full summary API by default. Once the monitoring pipeline is completed, the summary API, or a suitable replacement, will be provided by the third party monitoring pipeline, possibly through a stand-alone version of cAdvisor. - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. -This Core Metrics API will be versioned to account for version-skew between kubernetes components. - This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support resource feasibility checking or node resource management, they can be added to the core metrics API. ### Metric Requirements @@ -250,14 +248,12 @@ To keep performance bounded while still offering metrics "On-Demand", all calls In the case where some metrics are not able to be computed instantly (time-averaged metrics, for example), then they must be recomputed on the "minimum recency" interval so that they always meet recency requirements. ## Implementation Plan -@dashpole will create an interface over cAdvisor that provides core metrics. -@dashpole will modify volume metrics collection so that it re-uses vendored cAdvisor libraries. +@dashpole will create an interface over cAdvisor that provides core metrics. @dashpole will modify the structure of metrics collection code to be "On-Demand". Suggested, tentative future work, which may be covered by future proposals: - Publish these metrics in some form to a kubelet API endpoint - Obtain all runtime-specific information needed to collect metrics from the CRI. - - Modify cAdvisor to be "stand alone", and run in a seperate binary from the kubelet. - Kubernetes can be configured to run a default "third party metrics provider" as a daemonset. Possibly standalone cAdvisor. ## Rollout Plan @@ -270,7 +266,6 @@ Once the [implementation work](#implementation-plan) is completed, @dashpole wil The implementation goals of the first milestone are outlined below. - [ ] Create the proposal - [ ] Implement collection and consumption of core metrics. -- [ ] Modify volume metrics collection so that it re-uses vendored cAdvisor libraries. - [ ] Modify the structure of metrics collection code to be "On-Demand" From cec5de5378b872b6a49daf71f1b45284cf0f3636 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 13 Jan 2017 16:45:35 -0800 Subject: [PATCH 26/31] removed capacity from requirements --- contributors/design-proposals/core-metrics-pipeline.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 40c20e8118e..aa7b3db9508 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -89,16 +89,16 @@ Metrics requirements for "First Class Resource Isolation and Utilization Feature - Pod-level usage metrics for Disk and Memory - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: - Scheduler - - Node-level capacity and availability metrics for Disk, CPU, and Memory + - Node-level availability metrics for Disk, CPU, and Memory - Pod-level usage metrics for Disk, CPU, and Memory - Container-level usage metrics for Disk, CPU, and Memory - Horizontal-Pod-Autoscaler - - Node-level capacity and availability metrics for CPU and Memory + - Node-level availability metrics for CPU and Memory - Pod-level usage metrics for CPU and Memory - Cluster Federation - - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Node-level availability metrics for Disk, Memory, and CPU - kubectl top and Kubernetes Dashboard - - Node-level capacity and availability metrics for Disk, Memory, and CPU + - Node-level availability metrics for Disk, Memory, and CPU - Pod-level usage metrics for Disk, Memory, and CPU - Container-level usage metrics for Disk, CPU, and Memory From ea0e796351856e301357f5f19a80efd26731f0ed Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Fri, 13 Jan 2017 16:48:55 -0800 Subject: [PATCH 27/31] internal to kubelet --- contributors/design-proposals/core-metrics-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index aa7b3db9508..132ce03c255 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -257,7 +257,7 @@ Suggested, tentative future work, which may be covered by future proposals: - Kubernetes can be configured to run a default "third party metrics provider" as a daemonset. Possibly standalone cAdvisor. ## Rollout Plan -The work described here is entirely internal. However, publishing a kubelet API endpoint +The work described here is entirely internal to the kubelet. However, publishing a kubelet API endpoint The core metrics endpoint (TBD) will be added alongside the current Summary API for the upcoming release. This should allow concurrent developments of other portions of the system metrics pipeline (metrics-server, for example). Once this addition is made, all other changes will be internal, and will not require any API changes. Once the [implementation work](#implementation-plan) is completed, @dashpole will start discussions on the future of the Summary API, and how to provide an out-of-the-box solution for the "third party monitoring" pipeline on the node. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. From 38e2036edf5e16813403971c2ca0a9e0ff09bc77 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 18 Jan 2017 11:36:58 -0800 Subject: [PATCH 28/31] narrow scope of metrics. --- .../design-proposals/core-metrics-pipeline.md | 148 +++--------------- 1 file changed, 23 insertions(+), 125 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 132ce03c255..d883319717d 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -85,157 +85,55 @@ The following is not meant to be an exhaustive list, but gives the current set o Metrics requirements for "First Class Resource Isolation and Utilization Features", based on kubernetes component needs, are as follows: - Kubelet - - Node-level availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk and Memory + - Node-level usage metrics for Filesystems, CPU, and Memory + - Pod-level usage metrics for Filesystems and Memory - Metrics Server (outlined in [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md)), which exposes the [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) to the following system components: - Scheduler - - Node-level availability metrics for Disk, CPU, and Memory - - Pod-level usage metrics for Disk, CPU, and Memory - - Container-level usage metrics for Disk, CPU, and Memory + - Node-level usage metrics for Filesystems, CPU, and Memory + - Pod-level usage metrics for Filesystems, CPU, and Memory + - Container-level usage metrics for Filesystems, CPU, and Memory - Horizontal-Pod-Autoscaler - - Node-level availability metrics for CPU and Memory + - Node-level usage metrics for CPU and Memory - Pod-level usage metrics for CPU and Memory - Cluster Federation - - Node-level availability metrics for Disk, Memory, and CPU + - Node-level usage metrics for Filesystems, CPU, and Memory - kubectl top and Kubernetes Dashboard - - Node-level availability metrics for Disk, Memory, and CPU - - Pod-level usage metrics for Disk, Memory, and CPU - - Container-level usage metrics for Disk, CPU, and Memory + - Node-level usage metrics for Filesystems, CPU, and Memory + - Pod-level usage metrics for Filesystems, CPU, and Memory + - Container-level usage metrics for Filesystems, CPU, and Memory -### Proposed Core Metrics API: - -An important difference between the current summary api and the proposed core metrics api is that pod and container-level metrics in the core metrics api contain only usage data, and not capacity-related statistics. Node level resource metrics continue to provide capacity. This is more accurate since a pod's resource capacity is really defined by its "requests" and "limits", and it is a better reflection of how components use these metrics. The kubelet, for example, finds which resources are constrained using node-level capacity and availability data, and then chooses which pods to take action on based on the pod's usage of the constrained resource. If neccessary, capacity for resources a pod consumes can still be correlated with node-level resources using this format of metrics. +### Proposed Core Metrics: +This section defines "usage metrics" for filesystems, CPU, and Memory. +As stated in Non-Goals, this proposal does not attempt to define the specific format by which these are exposed. For convenience, it may be neccessary to include static information such as start time, node capacities for CPU, Memory, or filesystems, and more. ```go -// CoreMetrics is a top-level container for holding NodeResources and PodUsage. -type CoreMetrics struct { - // Overall node resource metrics. - Node NodeResources - // Per-pod usage metrics. - Pods []PodUsage -} - -// NodeResources holds node-level metrics. NodeResources contains capacity and availibility for Node Resources. -type NodeResources struct { - // The filesystem used by node kubernetes components. - KubeletFilesystem string - // The filesystem used by node runtime components. - RuntimeFilesystem string - // Metrics pertaining to cpu resources. - CPU *CpuResources - // Metrics pertaining to memory (RAM) resources. - Memory *MemoryResources - // Metrics pertaining to node filesystem resources. - Filesystems []FilesystemResources -} - -// CpuResources containes data about overall cpu resource capacity and usage -type CpuResources struct { - // The number of cores in this machine. - NumCores int `json:"numcores"` - // The current Usage of CPU resources - Usage *CpuUsage -} - -// MemoryResources contains data about memory resource usage. -type MemoryResources struct { - // The time at which these metrics were updated. - Timestamp metav1.Time - // The memory capacity, in bytes - CapacityBytes *uint64 - // The available memory, in bytes - // This is the number of bytes which are not included in the working set memory - // Working set memory includes recently accessed memory, dirty memory, and kernel memory. - AvailableBytes *uint64 -} - -// FilesystemResources contains data about filesystem disk resources. -type FilesystemResources struct { - // The time at which these metrics were updated. - Timestamp metav1.Time - // The filesystem name, which uniquely identifies a filesystem - Filesystem string - // AvailableBytes represents the storage space available (bytes) for the filesystem. - AvailableBytes *uint64 - // CapacityBytes represents the total capacity (bytes) of the filesystems underlying storage. - CapacityBytes *uint64 - // InodesFree represents the free inodes in the filesystem. - InodesFree *uint64 - // Inodes represents the total inodes in the filesystem. - Inodes *uint64 -} - -// PodUsage holds pod-level metrics. -type PodUsage struct { - // UID of the pod - PodUID string - // Metrics pertaining to pod total usage of cpu - // This may include additional overhead not included in container usage statistics. - CPU *CpuUsage - // Metrics pertaining to pod total usage of system memory - // This may include additional overhead not included in container usage statistics. - Memory *MemoryUsage - // Metrics of containers in the pod. - Containers []ContainerUsage - // Metrics pertaining to volume usage of filesystem resources. - Volumes []VolumeUsage -} - -// ContainerUsage holds container-level usage metrics. -type ContainerUsage struct { - // ID of the container - ContainerID string - // Metrics pertaining to container usage of cpu - CPU *CpuUsage - // Metrics pertaining to container usage of system memory - Memory *MemoryUsage - // Metrics pertaining to container rootfs usage of disk. - // Rootfs.UsedBytes is the number of bytes used for the container write layer. - Rootfs *FilesystemUsage - // Metrics pertaining to container logs usage of Disk. - Logs *FilesystemUsage -} - // CpuUsage holds statistics about the amount of cpu time consumed type CpuUsage struct { // The time at which these Metrics were updated. Timestamp metav1.Time - // Average CPU usage rate over sample window (across all cores), in "nano cores". - // The "core" unit represents nanoseconds of CPU time consumed per second. - // For example, 5 nanocores means the process averaged 5 nanoseconds - // of cpu time per second during the sample window. - UsageRateNanoCores *uint64 // Cumulative CPU usage (sum of all cores) since object creation. - AggregateUsageCoreNanoSeconds *uint64 + CumulativeUsageNanoSeconds *uint64 } // MemoryUsage holds statistics about the quantity of memory consumed type MemoryUsage struct { // The time at which these metrics were updated. Timestamp metav1.Time - // The amount of working set memory. This includes recently accessed memory, + // The amount of "working set" memory. This includes recently accessed memory, // dirty memory, and kernel memory. - WorkingSetBytes *uint64 -} - -// VolumeUsage holds statistics about the quantity of disk resources consumed for a volume -type VolumeUsage struct { - // Embedded FilesystemUsage - FilesystemUsage - // Name is the name given to the Volume - Name string -} + UsageBytes *uint64 +} -// FilesystemUsage holds statistics about the quantity of disk resources consumed +// FilesystemUsage holds statistics about the quantity of local storage (e.g. disk) resources consumed type FilesystemUsage struct { // The time at which these metrics were updated. Timestamp metav1.Time - // The name filesystme on which resources are consumed. This must uniquely identify the filesystem. - Filesystem string - // UsedBytes represents the disk space consumed on the filesystem, in bytes. + // This must uniquely identify the storage resource that is consumed. + StorageIdentifier string + // UsedBytes represents the disk space consumed, in bytes. UsedBytes *uint64 - // InodesUsed represents the inodes consumed on the filesystem - InodesUsed *uint64 + // UsageInodes represents the inodes consumed + UsageInodes *uint64 } ``` From 15e80b487151d2096005cb80194f7c2b786686de Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Thu, 19 Jan 2017 14:37:01 -0800 Subject: [PATCH 29/31] final changes --- .../design-proposals/core-metrics-pipeline.md | 22 ++++++++----------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index d883319717d..68ed9e87df9 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -2,7 +2,7 @@ **Author**: David Ashpole (@dashpole) -**Last Updated**: 1/12/2017 +**Last Updated**: 1/19/2017 **Status**: Proposal @@ -19,7 +19,7 @@ This document proposes a design for the set of metrics included in an eventual C - [Non Goals](#non-goals) - [Design](#design) - [Metric Requirements:](#metric-requirements) - - [Proposed Core Metrics API:](#proposed-core-metrics-api) + - [Proposed Core Metrics:](#proposed-core-metrics) - [On-Demand Design:](#on-demand-design) - [Implementation Plan](#implementation-plan) - [Rollout Plan](#rollout-plan) @@ -34,7 +34,7 @@ This document proposes a design for the set of metrics included in an eventual C ["cAdvisor":](https://github.com/google/cadvisor) An open source container monitoring solution which only monitors containers, and has no concept of kubernetes constructs like pods or volumes. ["Summary API":](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/stats/types.go) A kubelet API which currently exposes node metrics for use by both system components and monitoring systems. ["CRI":](https://github.com/kubernetes/community/blob/master/contributors/devel/container-runtime-interface.md) The Container Runtime Interface designed to provide an abstraction over runtimes (docker, rkt, etc). -"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide system components with metrics for the purpose of [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) or node resource management. +"Core Metrics": A set of metrics described in the [Monitoring Architecture](https://github.com/kubernetes/kubernetes/blob/master/docs/design/monitoring_architecture.md) whose purpose is to provide metrics for first-class resource isolation and untilization features, including [resource feasibility checking](https://github.com/eBay/Kubernetes/blob/master/docs/design/resources.md#the-resource-model) and node resource management. "Resource": A consumable element of a node (e.g. memory, disk space, CPU time, etc). "First-class Resource": A resource critical for scheduling, whose requests and limits can be (or soon will be) set via the Pod/Container Spec. "Metric": A measure of consumption of a Resource. @@ -55,7 +55,7 @@ The third party monitoring pipeline also is relieved of any responsibility to pr cAdvisor is structured to collect metrics on an interval, which is appropriate for a stand-alone metrics collector. However, many functions in the kubelet are latency-sensitive (eviction, for example), and would benifit from a more "On-Demand" metrics collection design. ### Proposal -This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support "First Class Resource Isolation and Utilization Features". This proposal is not designed to be an API published by the kubelet, but rather a set of metrics collected by the kubelet that will be transformed, and published in the future. +This proposal is to use this set of core metrics, collected by the kubelet, and used solely by kubernetes system components to support "First-Class Resource Isolation and Utilization Features". This proposal is not designed to be an API published by the kubelet, but rather a set of metrics collected by the kubelet that will be transformed, and published in the future. The target "Users" of this set of metrics are kubernetes components (though not neccessarily directly). This set of metrics itself is not designed to be user-facing, but is designed to be general enough to support user-facing components. @@ -67,14 +67,13 @@ Integration with CRI will not be covered in this proposal. In future proposals, The kubelet API endpoint, including the format, url pattern, versioning strategy, and name of the API will be the topic of a follow-up proposal to this proposal. ## Design -This design covers only the Core Metrics Pipeline. +This design covers only metrics to be included in the Core Metrics Pipeline. High level requirements for the design are as follows: - - The kubelet collects the minimum possible number of metrics to provide "First Class Resource Isolation and Utilization Features". - - Do not break existing users. We should continue to provide the full summary API by default. Once the monitoring pipeline is completed, the summary API, or a suitable replacement, will be provided by the third party monitoring pipeline, possibly through a stand-alone version of cAdvisor. + - The kubelet collects the minimum possible number of metrics to provide "First-Class Resource Isolation and Utilization Features". - Metrics can be fetched "On Demand", giving the kubelet more up-to-date stats. -This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support resource feasibility checking or node resource management, they can be added to the core metrics API. +This proposal purposefully omits many metrics that may eventually become core metrics. This is by design. Once metrics are needed to support First-Class Resource Isolation and Utilization Features, they can be added to the core metrics API. ### Metric Requirements The core metrics api is designed to provide metrics for "First Class Resource Isolation and Utilization Features" within kubernetes. @@ -139,14 +138,12 @@ type FilesystemUsage struct { ### On-Demand Design Interface: -The interface for exposing these metrics within the kubelet contains a method for getting this datastructure. This method contains a "recency" parameter which specifies how recently the metrics must have been computed. Kubelet components which require very up-to-date metrics (eviction, for example), use very low values. Other components use higher values. +The interface for exposing these metrics within the kubelet contains methods for fetching each relevant metric. These methods contains a "recency" parameter which specifies how recently the metrics must have been computed. Kubelet components which require very up-to-date metrics (eviction, for example), use very low values. Other components use higher values. Implementation: To keep performance bounded while still offering metrics "On-Demand", all calls to get metrics are cached, and a minimum recency is established to prevent repeated metrics computation. Before computing new metrics, the previous metrics are checked to see if they meet the recency requirements of the caller. If the age of the metrics meet the recency requirements, then the cached metrics are returned. If not, then new metrics are computed and cached. -In the case where some metrics are not able to be computed instantly (time-averaged metrics, for example), then they must be recomputed on the "minimum recency" interval so that they always meet recency requirements. -## Implementation Plan -@dashpole will create an interface over cAdvisor that provides core metrics. +## Implementation Plan @dashpole will modify the structure of metrics collection code to be "On-Demand". Suggested, tentative future work, which may be covered by future proposals: @@ -163,7 +160,6 @@ Once the [implementation work](#implementation-plan) is completed, @dashpole wil The implementation goals of the first milestone are outlined below. - [ ] Create the proposal -- [ ] Implement collection and consumption of core metrics. - [ ] Modify the structure of metrics collection code to be "On-Demand" From 6c43ae8da2dc08e9ccd78e8cb2606ce6b8d56716 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Mon, 23 Jan 2017 16:55:03 -0800 Subject: [PATCH 30/31] s/UsageInodes/UsedInodes; updated rollout --- contributors/design-proposals/core-metrics-pipeline.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index 68ed9e87df9..f5a77cccea8 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -131,8 +131,8 @@ type FilesystemUsage struct { StorageIdentifier string // UsedBytes represents the disk space consumed, in bytes. UsedBytes *uint64 - // UsageInodes represents the inodes consumed - UsageInodes *uint64 + // UsedInodes represents the inodes consumed + UsedInodes *uint64 } ``` @@ -152,9 +152,8 @@ Suggested, tentative future work, which may be covered by future proposals: - Kubernetes can be configured to run a default "third party metrics provider" as a daemonset. Possibly standalone cAdvisor. ## Rollout Plan -The work described here is entirely internal to the kubelet. However, publishing a kubelet API endpoint -The core metrics endpoint (TBD) will be added alongside the current Summary API for the upcoming release. This should allow concurrent developments of other portions of the system metrics pipeline (metrics-server, for example). Once this addition is made, all other changes will be internal, and will not require any API changes. -Once the [implementation work](#implementation-plan) is completed, @dashpole will start discussions on the future of the Summary API, and how to provide an out-of-the-box solution for the "third party monitoring" pipeline on the node. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. +Once this set of metrics is accepted, @dashpole will begin discussions on the format, and design of the endpoint that exposes them. The node resource metrics endpoint (TBD) will be added alongside the current Summary API in an upcoming release. This should allow concurrent developments of other portions of the system metrics pipeline (metrics-server, for example). Once this addition is made, all other changes will be internal, and will not require any API changes. +@dashpole will also start discussions on integrating with the CRI, and discussions on how to provide an out-of-the-box solution for the "third party monitoring" pipeline on the node. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. ## Implementation Status From 33160bede67e7355f123d8c3ba121f71a7a01411 Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Tue, 24 Jan 2017 09:28:58 -0800 Subject: [PATCH 31/31] addressed timstclair changes --- .../design-proposals/core-metrics-pipeline.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/contributors/design-proposals/core-metrics-pipeline.md b/contributors/design-proposals/core-metrics-pipeline.md index f5a77cccea8..9feceeb7090 100644 --- a/contributors/design-proposals/core-metrics-pipeline.md +++ b/contributors/design-proposals/core-metrics-pipeline.md @@ -23,7 +23,6 @@ This document proposes a design for the set of metrics included in an eventual C - [On-Demand Design:](#on-demand-design) - [Implementation Plan](#implementation-plan) - [Rollout Plan](#rollout-plan) - - [Implementation Status](#implementation-status) @@ -90,6 +89,9 @@ Metrics requirements for "First Class Resource Isolation and Utilization Feature - Scheduler - Node-level usage metrics for Filesystems, CPU, and Memory - Pod-level usage metrics for Filesystems, CPU, and Memory + - Vertical-Pod-Autoscaler + - Node-level usage metrics for Filesystems, CPU, and Memory + - Pod-level usage metrics for Filesystems, CPU, and Memory - Container-level usage metrics for Filesystems, CPU, and Memory - Horizontal-Pod-Autoscaler - Node-level usage metrics for CPU and Memory @@ -127,7 +129,8 @@ type MemoryUsage struct { type FilesystemUsage struct { // The time at which these metrics were updated. Timestamp metav1.Time - // This must uniquely identify the storage resource that is consumed. + // StorageIdentifier must uniquely identify the node-level storage resource that is consumed. + // It may utilize device, partition, filesystem id, or other identifiers. StorageIdentifier string // UsedBytes represents the disk space consumed, in bytes. UsedBytes *uint64 @@ -155,13 +158,6 @@ Suggested, tentative future work, which may be covered by future proposals: Once this set of metrics is accepted, @dashpole will begin discussions on the format, and design of the endpoint that exposes them. The node resource metrics endpoint (TBD) will be added alongside the current Summary API in an upcoming release. This should allow concurrent developments of other portions of the system metrics pipeline (metrics-server, for example). Once this addition is made, all other changes will be internal, and will not require any API changes. @dashpole will also start discussions on integrating with the CRI, and discussions on how to provide an out-of-the-box solution for the "third party monitoring" pipeline on the node. One current idea is a standalone verison of cAdvisor, but any third party metrics solution could serve this function as well. -## Implementation Status - -The implementation goals of the first milestone are outlined below. -- [ ] Create the proposal -- [ ] Modify the structure of metrics collection code to be "On-Demand" - - [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/core-metrics-pipeline.md?pixel)]()