This plugin provides a HTTP server, exposing metrics via REST API. Conforming to API of Kubelet this publisher can be used as an alternate backend for Heapster monitoring tool.
- Supported REST server endpoints
- Configuration parameters
- Custom metrics
- Examples
- Known limitations
- Roadmap
This plugin needs to be used with snap-plugin-collector-docker.
Docker collector is necessary, because it delivers containers' metrics. Its metrics define the structure for published data. In particular docker collector delivers metrics for root container ('/') which receives custom metrics by default.
- Linux/amd64
- Darwin/amd64
You can get the pre-built binaries for your OS and architecture at snap's GitHub Releases page.
Fork https://github.com/intelsdi-x/snap-plugin-publisher-heapster
Clone repo into $GOPATH/src/github.com/intelsdi-x/:
$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-heapster.gitBuild the plugin by running make within the cloned repo:
$ makeIt may take a while to pull dependencies if you haven't had them already.
This builds the plugin in /build/rootfs/
- Set up the snap framework
- POST
/stats/container/- compatible with Kubelet, endpoint supports JSON requests. Example request:
{
"containerName":"/",
"num_stats":10,
"start": "2016-09-09T12:57:05+02:00",
"end": "2016-09-12T12:57:05+02:00",
"subcontainers": true
}As a matter of fact all parameters are optional, but not all should be left out (ie.: it would be wise to limit nubmer of stats per container).
Here is the copy of documentation for request copied from source code exchange/exchange.go:
// StatsRequest is a struct representing HTTP request for stats.
type StatsRequest struct {
// ContainerName is name of the container to retrieve stats for
// (defaults to root or '/').
ContainerName string `json:"containerName,omitempty"`
// NumStats defines amount of stats to return per container, maximum.
// Ignored if start and end time is given.
NumStats int `json:"num_stats,omitempty"`
// Start defines time window limiting the stats to fetch per container.
// Defaults to beginning of time.
Start time.Time `json:"start,omitempty"`
// End defines time window limiting the stats to fetch per container.
// Defaults to current time.
End time.Time `json:"end,omitempty"`
// Subcontainers controls depth of query, allowing to fetch stats for
// subcontainers.
Subcontainers bool `json:"subcontainers,omitempty"`
}Plugin supports several parameters in task manifest (defaults given below):
plugin_name: "heapster"
config:
stats_depth: 0
stats_span: "10m"
server_addr: "127.0.0.1"
server_port: 8777
verbose_at: ""
silent_at: ""
mute_at: ""All of them are optional. Their meaning is as follows:
stats_depth- max. number of stats elements stored per each containerstats_span- max. time difference between oldest and most recent stats elements per each container, e.g.:"10m"(10 minutes at most)server_addr- address the server should bind toserver_port- port number the server should listen atverbose_at- space delimited list of modules which should be logging at verbose level, e.g.:"/heapster /server"silent_at- space delimited list of modules which should be logging at silent level (severity at least warning)mute_at- list of modules which should have logging muted (only errors logged)
List of modules in the publisher:
/heapster- main module of the plugin,/server- module implementing REST server,/processor/main- module handling incoming docker metrics,/processor/custom- module handling incoming custom metrics.
Heapster publisher can expose custom metrics in addition to those reported for containers. It can work with any number of additional collectors and integrate their metrics into the published stats. This can be accomplished purely by configuration.
Heapster recognizes specific tags on collected metrics and can expose them via REST API. Tags may be set via task manifest with a per-metric granularity. You can read more about tags in snap documentation for tasks.
Here's the list of tags supported by heapster publisher with a description:
custom_metric_name- name for the custom metric to appear under custom metrics list in the stats defaults to original namespace of the incoming metriccustom_metric_type- type of the custom metric, as understood by Cadvisor, see cadvisor/info/v1/metric.go; defaults togaugecustom_metric_format- format of the metric data type, as understood by Cadvisor, see cadvisor/info/v1/metric.go; defaults tointcustom_metric_units- purely informative purpose, describes the units of the metric data; defaults tononecustom_metric_container_path- id of the container that should have those custom metrics assigned; defaults to/, or root container. There is also support for asterisk*, meaning: propagate to all containers.
A metric must be tagged with at least one of the tags above to be recognized as custom metric.
An example manifest for a task exercising custom metrics may be found in file examples/tasks/heapster-custom-metrics.yaml.
You can find example task manifests for using this plugin in folder examples/tasks/
Set up the Snap framework
Ensure Snap daemon is running in one of the following ways:
- initd:
service snap-telemetry start - systemd:
systemctl start snap-telemetry - manually:
sudo snapteld -l 1 -t 0 &
Start snap in one terminal window:
$ snapteld -l 1 -t 0In another terminal window load required plugins - docker and heapster (replace fully-qualified paths with actual plugin binaries if you downloaded them):
$ cd $GOPATH/src/github.com/intelsdi-x/
$ snaptel plugin load ./snap-plugin-collector-docker/build/rootfs/snap-plugin-collector-docker
$ snaptel plugin load ./snap-plugin-collector-docker/build/rootfs/snap-plugin-publisher-heapsterCreate task manifest referencing docker- and heapster- plugins:
version: 1
schedule:
type: "simple"
interval: "10s"
workflow:
collect:
metrics:
/intel/docker/*: {}
publish:
-
plugin_name: "heapster"
config:
stats_span: "10m"
server_addr: "127.0.0.1"
server_port: 8777Create a task by the following command:
cd $GOPATH/src/github.com/intelsdi-x/
$ snaptel task create -t ./snap-plugin-publisher-heapster/examples/tasks/heapster-with-docker.yaml
Using task manifest to create task
Task created
ID: 9b26dc87-a3f8-4e8a-9303-ef88cb12e3a2
Name: Task-9b26dc87-a3f8-4e8a-9303-ef8cb12e3a2
State: RunningVerify that heapster publisher correctly exposes metrics:
$ curl -H 'Content-Type: application/json' -X POST -d '{"num_stats": 1}' 127.0.0.1:8777/stats/container/ |python -m json.toolYou should see output similar to this one:
{
"/": {
"id": "/",
"name": "/",
"spec": {
"cpu": {
"limit": 0,
"max_limit": 0
},
"creation_time": "2016-06-16T13:05:16.10346462+02:00",
"custom_metrics": [
],
"has_cpu": false,
"has_custom_metrics": true,
"has_diskio": false,
"has_filesystem": true,
"has_memory": false,
"has_network": true,
"memory": {}
},
"stats": [
{
"cpu": {
"load_average": 0,
"usage": {
"per_cpu_usage": [
9409993187764,
5362499917666,
9168442203791,
6403390387149,
9431651754669,
9164956796928,
4759647789316,
5411972414128
],
"system": 10640710000000,
"total": 59112554451411,
"user": 50338490000000
}
},
[...] Rest ommittedWhen using tags for binding metrics to specific containers only the Id
of a container is recognized. That is, although a tag is named custom_metric_container_path,
this is understood and matched only against container Ids. In case there
is a container Id duplicate in system, custom metrics will appear only
for one of the containers.
There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release. If you have a feature request, please add it as an issue and/or submit a pull request.
This repository is one of many plugins in the Snap framework: a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework
We love contributions!
There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.
Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.
- Author: Marcin Olszewski
Thank you! Your contribution is incredibly important to us.