forked from cloudposse/prometheus-to-cloudwatch
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.yaml
214 lines (162 loc) · 11.7 KB
/
README.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
# Name of this project
name: prometheus-to-cloudwatch
# License of this project
license: APACHE2
# Canonical GitHub repo
github_repo: cloudposse/prometheus-to-cloudwatch
# Logo for this project
#logo: docs/logo.png
# Badges to display
badges:
- name: "Build Status"
image: "https://travis-ci.org/cloudposse/prometheus-to-cloudwatch.svg?branch=master"
url: "https://travis-ci.org/cloudposse/prometheus-to-cloudwatch"
- name: "Latest Release"
image: "https://img.shields.io/github/release/cloudposse/prometheus-to-cloudwatch.svg"
url: "https://github.com/cloudposse/prometheus-to-cloudwatch/releases/latest"
- name: "Slack Community"
image: "https://slack.cloudposse.com/badge.svg"
url: "https://slack.cloudposse.com"
related:
- name: "Prometheus Operator"
description: "Prometheus Operator creates/configures/manages Prometheus clusters atop Kubernetes"
url: "https://github.com/cloudposse/prometheus-operator"
- name: "terraform-aws-cloudwatch-logs"
description: "Terraform Module to Provide a CloudWatch Logs Endpoint"
url: "https://github.com/cloudposse/terraform-aws-cloudwatch-logs"
- name: "terraform-aws-ecs-web-app"
description: "Terraform module that implements a web app on ECS and supports autoscaling, CI/CD, monitoring, ALB integration, and much more."
url: "https://github.com/cloudposse/terraform-aws-ecs-web-app"
description: |-
Utility for scraping Prometheus metrics from a Prometheus client endpoint and publishing them to CloudWatch
usage: |-
__NOTE__: The module accepts parameters as command-line arguments or as ENV variables (or any combination of command-line arguments and ENV vars).
Command-line arguments take precedence over ENV vars
| Command-line argument | ENV var | Description |
|--------------------------------|--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| aws_access_key_id | AWS_ACCESS_KEY_ID | AWS access key Id with permissions to publish CloudWatch metrics |
| aws_secret_access_key | AWS_SECRET_ACCESS_KEY | AWS secret access key with permissions to publish CloudWatch metrics |
| cloudwatch_namespace | CLOUDWATCH_NAMESPACE | CloudWatch Namespace |
| cloudwatch_region | CLOUDWATCH_REGION | CloudWatch AWS Region |
| cloudwatch_publish_timeout | CLOUDWATCH_PUBLISH_TIMEOUT | CloudWatch publish timeout in seconds |
| prometheus_scrape_interval | PROMETHEUS_SCRAPE_INTERVAL | Prometheus scrape interval in seconds |
| prometheus_scrape_url | PROMETHEUS_SCRAPE_URL | The URL to scrape Prometheus metrics from |
| cert_path | CERT_PATH | Path to SSL Certificate file (when using SSL for `prometheus_scrape_url`) |
| keyPath | KEY_PATH | Path to Key file (when using SSL for `prometheus_scrape_url`) |
| accept_invalid_cert | ACCEPT_INVALID_CERT | Accept any certificate during TLS handshake. Insecure, use only for testing |
| additional_dimension | ADDITIONAL_DIMENSION | Additional dimension specified by NAME=VALUE |
| replace_dimensions | REPLACE_DIMENSIONS | Replace dimensions specified by NAME=VALUE,... |
| include_metrics | INCLUDE_METRICS | Only publish the specified metrics (comma-separated list of glob patterns) |
| exclude_metrics | EXCLUDE_METRICS | Never publish the specified metrics (comma-separated list of glob patterns) |
| include_dimensions_for_metrics | INCLUDE_DIMENSIONS_FOR_METRICS | Only publish the specified dimensions for metrics (semi-colon-separated key values of comma-separated dimensions of METRIC=dim1,dim2;, e.g. 'flink_jobmanager=job_id') |
| exclude_dimensions_for_metrics | EXCLUDE_DIMENSIONS_FOR_METRICS | Never publish the specified dimensions for metrics (semi-colon-separated key values of comma-separated dimensions of METRIC=dim1,dim2;, e.g. 'flink_jobmanager=job,host;zk_up=host,pod;') |
| force_high_res | FORCE_HIGH_RES | Whether publish all metrics with high resolution to Cloudwatch or only those labeled with `__cw_high_res`. |
| listenAddress | LISTEN_ADDRESS | Address to expose metrics. Default: 9698 |
| metricsPath | METRICS_PATH | Path under which to expose metrics. Default: /metrics |
| environmentName | environmentName | Environment name from where the CloudWatch metrics come from |
| infraShortname | infraShortname | Infrastructure shortname from where the CloudWatch metrics come from. |
__NOTE__: If AWS credentials are not provided in the command-line arguments (`aws_access_key_id` and `aws_secret_access_key`)
or ENV variables (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`),
the chain of credential providers will search for credentials in the shared credential file and EC2 Instance Roles.
This is useful when deploying the module in AWS on Kubernetes with [`kube2iam`](https://github.com/jtblin/kube2iam),
which will provide IAM credentials to containers running inside a Kubernetes cluster, allowing the module to assume an IAM Role with permissions
to publish metrics to CloudWatch.
monitoring: |-
### Internal Metrics
The service is able to expose its own metrics over `http://$someIP:9698/metrics`
* prometheus_to_cloudwatch_publishes_total
* prometheus_to_cloudwatch_publish_errors_total
* prometheus_to_cloudwatch_publish_duration_seconds
* prometheus_to_cloudwatch_metrics_total
examples: |-
### Build Go program
```sh
go get
CGO_ENABLED=0 go build -v -o "./dist/bin/prometheus-to-cloudwatch" *.go
```
### Run locally
```sh
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
export CLOUDWATCH_NAMESPACE=kube-state-metrics
export CLOUDWATCH_REGION=us-east-1
export CLOUDWATCH_PUBLISH_TIMEOUT=5
export PROMETHEUS_SCRAPE_INTERVAL=30
export PROMETHEUS_SCRAPE_URL=http://xxxxxxxxxxxx:8080/metrics
export CERT_PATH=""
export KEY_PATH=""
export ACCEPT_INVALID_CERT=true
# Optionally, restrict the subset of metrics to be exported to CloudWatch
# export INCLUDE_METRICS='jvm_*'
# export EXCLUDE_METRICS='jvm_memory_*,jvm_buffer_*'
# export INCLUDE_DIMENSIONS_FOR_METRICS='jvm_memory_*=pod_id'
# export EXCLUDE_DIMENSIONS_FOR_METRICS='jvm_memory_*=pod;jvm_buffer=job,pod'
./dist/bin/prometheus-to-cloudwatch
```
### Build Docker image
__NOTE__: it will download all `Go` dependencies and then build the program inside the container (see [`Dockerfile`](Dockerfile))
```sh
docker build --tag prometheus-to-cloudwatch --no-cache=true .
```
### Run in a Docker container
```sh
docker run -i --rm \
-e AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXXXXX \
-e AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
-e CLOUDWATCH_NAMESPACE=kube-state-metrics \
-e CLOUDWATCH_REGION=us-east-1 \
-e CLOUDWATCH_PUBLISH_TIMEOUT=5 \
-e PROMETHEUS_SCRAPE_INTERVAL=30 \
-e PROMETHEUS_SCRAPE_URL=http://xxxxxxxxxxxx:8080/metrics \
-e CERT_PATH="" \
-e KEY_PATH="" \
-e ACCEPT_INVALID_CERT=true \
-e INCLUDE_METRICS="" \
-e EXCLUDE_METRICS="" \
-e INCLUDE_DIMENSIONS_FOR_METRICS="" \
-e EXCLUDE_DIMENSIONS_FOR_METRICS="" \
prometheus-to-cloudwatch
```
### Run on Kubernetes
To run on `Kubernetes`, we will deploy two [`Helm`](https://helm.sh/) [charts](https://docs.helm.sh/developing_charts/)
1. [kube-state-metrics](https://github.com/kubernetes/charts/tree/master/stable/kube-state-metrics) - to generates metrics about the state of various objects inside the cluster, such as deployments, nodes and pods
2. [prometheus-to-cloudwatch](chart) - to scrape metrics from `kube-state-metrics` and publish them to CloudWatch
Install `kube-state-metrics` chart
```sh
helm install stable/kube-state-metrics
```
Find the running services
```sh
kubectl get services
```
Copy the name of the `kube-state-metrics` service (e.g. `gauche-turtle-kube-state-metrics`) into the ENV var `PROMETHEUS_SCRAPE_URL` in [values.yaml](chart/values.yaml).
It should look like this:
```sh
PROMETHEUS_SCRAPE_URL: "http://gauche-turtle-kube-state-metrics:8080/metrics"
```
Deploy `prometheus-to-cloudwatch` chart
```sh
cd chart
helm install .
```
`prometheus-to-cloudwatch` will start scraping the `/metrics` endpoint of the `kube-state-metrics` service and send the Prometheus metrics to CloudWatch
screenshots:
- name: "kube-state-metrics-to-cloudwatch"
url: "images/kube-state-metrics-to-cloudwatch.png"
description: "kube-state-metrics to CloudWatch"
# Contributors to this project
contributors:
- name: "Erik Osterman"
github: "osterman"
- name: "Andriy Knysh"
github: "aknysh"
- name: "Igor Rodionov"
github: "goruha"
- name: "yufukui-m"
github: "yufukui-m"
- name: "Satadru Biswas"
github: "sbiswas-suplari"
- name: "Austin ce"
github: "austince"