From fd946dd8e062b76cb43f05a0a05cf0c143e14bdf Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Wed, 25 Nov 2020 19:24:58 -0500 Subject: [PATCH 01/19] adding README --- .../examples/README.md | 180 ++++++++++++++++++ .../examples/docker-compose.yml | 12 ++ 2 files changed, 192 insertions(+) create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md new file mode 100644 index 0000000000..da62c3dfdd --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md @@ -0,0 +1,180 @@ +# OpenTelemetry Python SDK Prometheus Remote Write Exporter +This package contains an exporter to send [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md) +metrics from the Python SDK directly to a Prometheus Remote Write integrated +backend (such as Cortex or Thanos) without having to run an instance of the +Prometheus server. The image below shows the two Prometheus exporters in the OpenTelemetry Python SDK. + + +Pipeline 1 illustrates the setup required for a Prometheus "pull" exporter. + + +Pipeline 2 illustrates the setup required for the Prometheus Remote Write exporter. + +![Prometheus SDK pipelines](https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png) + + +The Prometheus Remote Write Exporter is a "push" based exporter and only works with the OpenTelemetry [push controller](https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py). +The controller periodically collects data and passes it to the exporter. This +exporter then converts the data into [`timeseries`](https://prometheus.io/docs/concepts/data_model/) and sends it to the Remote Write integrated backend through HTTP +POST requests. The metrics collection datapath is shown below: + +![Metrics collection datapath](https://user-images.githubusercontent.com/20804975/100286063-157f2a80-2f40-11eb-819d-f0aa46c1c2c8.png) + + + + +See the `example` folder for a demo usage of this exporter + + + +# Table of Contents + * [Summary](#opentelemetry-python-sdk-prometheus-remote-write-exporter) + * [Table of Contents](#table-of-contents) + * [Installation](#installation) + * [Quickstart](#quickstart) + * [Configuring the Exporter](#configuring-the-exporter) + * [Securing the Exporter](#securing-the-exporter) + * [Authentication](#authentication) + * [TLS](#tls) + * [Supported Aggregators](#supported-aggregators) + * [Error Handling](#error-handling) + * [Retry Logic](#retry-logic) + * [Contributing](#contributing) + * [Design Doc](#design-doc) + +## Installation + +* To install from the latest PyPi release, + run `pip install opentelemetry-exporter-prometheus-remote-write` +* To install from the local repository, run + `pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/` in + the project root + +## Quickstart + +```python +from opentelemetry import metrics +from opentelemetry.sdk.metrics import MeterProvider +from opentelemetry.exporter.prometheus_remote_write import ( + PrometheusRemoteWriteMetricsExporter +) + +# Sets the global MeterProvider instance +metrics.set_meter_provider(MeterProvider()) + +# The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here. +meter = metrics.get_meter(__name__) + +exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed + +metrics.get_meter_provider().start_pipeline(meter, exporter, 5) +``` +## Configuring the Exporter + +The exporter can be configured through parameters passed to the constructor. +Here are all the options: + +* `endpoint`: url where data will be sent **(Required)** +* `basic_auth`: username and password for authentication **(Optional)** +* `headers`: additional headers for remote write request as determined by the remote write backend's API **(Optional)** +* `timeout`: timeout for requests to the remote write endpoint in seconds **(Optional)** +* `proxies`: dict mapping request proxy protocols to proxy urls **(Optional)** +* `tls_config`: configuration for remote write TLS settings **(Optional)** + +Example with all the configuration options: + +```python +exporter = PrometheusRemoteWriteMetricsExporter( + endpoint="http://localhost:9009/api/prom/push", + timeout=30, + basic_auth={ + "username": "user", + "password": "pass123", + }, + headers={ + "X-Scope-Org-ID": "5", + "Authorization": "Bearer mytoken123", + }, + proxies={ + "http": "http://10.10.1.10:3000", + "https": "http://10.10.1.10:1080", + }, + tls_config={ + "cert_file": "path/to/file", + "key_file": "path/to/file", + } +) + +``` +## Securing the Exporter + +### Authentication + +The exporter provides two forms of authentication which are shown below. Users +can add their own custom authentication by setting the appropriate values in the `headers` dictionary + +1. Basic Authentication +Basic authentication sets a HTTP Authorization header containing a base64 encoded username/password pair. See [RFC 7617](https://tools.ietf.org/html/rfc7617) for more information. This + +```python +exporter = PrometheusRemoteWriteMetricsExporter( + basic_auth={"username": "base64user", "password": "base64pass"} +) +``` +2. Bearer Token Authentication +This custom configuration can be achieved by passing in a custom `header` to +the constructor. See [RFC 6750](https://tools.ietf.org/html/rfc6750) for more information. + + +```python +header = { + "Authorization": "Bearer mytoken123" +} +``` + +### TLS +Users can add TLS to the exporter's HTTP Client by providing certificate and key files in the `tls_config` parameter. + +## Supported Aggregators + +* Sum +* MinMaxSumCount +* Histogram +* LastValue +* ValueObserver +* Summary + +## Error Handling +In general, errors are returned to the calling function. The exception is for +the exporter's `export()` method where any error status code is logged as a +warning instead. + +This is because the exporter does not implement any retry logic +as it sends cumulative metrics data. This means that data will be preserved even if some exports fail. + +For example, consider a situation where a user increments a Counter instrument 5 times and an export happens between each increment. If the exports happen like so: +``` +SUCCESS FAIL FAIL SUCCESS SUCCESS +1 2 3 4 5 +``` +Then the recieved data will be: +``` +1 4 5 +``` +The end result is the same since the aggregations are cumulative +## Contributing + +This exporter's datapath is as follows: + +![Exporter datapath](https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png) +*Entites with `*` after their name are not actual classes but rather logical +groupings of functions within the exporter.* + +If you would like to learn more about the exporter's structure and design decisions please view the design document below + +### Design Doc + +[Design Document](TODO: add link) + +This document is not in this repo as it contains large images which will +significantly increase the size this repo. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml new file mode 100644 index 0000000000..15ccd3d705 --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml @@ -0,0 +1,12 @@ + cortex: + image: quay.io/cortexproject/cortex:v1.5.0 + command: + - -config.file=./config/cortex-config.yml + volumes: + - ./prometheus-remote-write-cortex/cortex-config.yml:/config/cortex-config.yml:ro + ports: + - 9009:9009 + grafana: + image: grafana/grafana:latest + ports: + - 3000:3000 From 4392645e9eef06548de7a8594da04b423cfbeadd Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Fri, 27 Nov 2020 12:21:46 -0500 Subject: [PATCH 02/19] adding sample app --- .../examples/Dockerfile | 8 + .../examples/README.md | 6 + .../examples/cortex-config.yml | 100 ++++++++++++ .../examples/docker-compose.yml | 23 ++- .../examples/requirements.txt | 7 + .../examples/sampleapp.py | 154 ++++++++++++++++++ .../setup.cfg | 4 +- 7 files changed, 299 insertions(+), 3 deletions(-) create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/examples/Dockerfile create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/examples/cortex-config.yml create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/examples/requirements.txt create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/Dockerfile b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/Dockerfile new file mode 100644 index 0000000000..09ce8cc323 --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/Dockerfile @@ -0,0 +1,8 @@ +FROM python:3.7 +WORKDIR /code + +COPY . . +RUN apt-get update -y && apt-get install libsnappy-dev -y +RUN pip install -e . +RUN pip install -r ./examples/requirements.txt +CMD ["python", "./examples/sampleapp.py"] diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md index da62c3dfdd..7df0e76a5a 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md @@ -32,6 +32,7 @@ See the `example` folder for a demo usage of this exporter * [Table of Contents](#table-of-contents) * [Installation](#installation) * [Quickstart](#quickstart) + * [Example](#example) * [Configuring the Exporter](#configuring-the-exporter) * [Securing the Exporter](#securing-the-exporter) * [Authentication](#authentication) @@ -69,6 +70,11 @@ exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add metrics.get_meter_provider().start_pipeline(meter, exporter, 5) ``` + +## Example + +* clone core +* TODO: change dockerfile if release 0.16 is public ## Configuring the Exporter The exporter can be configured through parameters passed to the constructor. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/cortex-config.yml b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/cortex-config.yml new file mode 100644 index 0000000000..37bd6473d6 --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/cortex-config.yml @@ -0,0 +1,100 @@ +# This Cortex Config is copied from the Cortex Project documentation +# Source: https://github.com/cortexproject/cortex/blob/master/docs/configuration/single-process-config.yaml + +# Configuration for running Cortex in single-process mode. +# This configuration should not be used in production. +# It is only for getting started and development. + +# Disable the requirement that every request to Cortex has a +# X-Scope-OrgID header. `fake` will be substituted in instead. +auth_enabled: false + +server: + http_listen_port: 9009 + + # Configure the server to allow messages up to 100MB. + grpc_server_max_recv_msg_size: 104857600 + grpc_server_max_send_msg_size: 104857600 + grpc_server_max_concurrent_streams: 1000 + +distributor: + shard_by_all_labels: true + pool: + health_check_ingesters: true + +ingester_client: + grpc_client_config: + # Configure the client to allow messages up to 100MB. + max_recv_msg_size: 104857600 + max_send_msg_size: 104857600 + use_gzip_compression: true + +ingester: + # We want our ingesters to flush chunks at the same time to optimise + # deduplication opportunities. + spread_flushes: true + chunk_age_jitter: 0 + + walconfig: + wal_enabled: true + recover_from_wal: true + wal_dir: /tmp/cortex/wal + + lifecycler: + # The address to advertise for this ingester. Will be autodiscovered by + # looking up address on eth0 or en0; can be specified if this fails. + # address: 127.0.0.1 + + # We want to start immediately and flush on shutdown. + join_after: 0 + min_ready_duration: 0s + final_sleep: 0s + num_tokens: 512 + tokens_file_path: /tmp/cortex/wal/tokens + + # Use an in memory ring store, so we don't need to launch a Consul. + ring: + kvstore: + store: inmemory + replication_factor: 1 + +# Use local storage - BoltDB for the index, and the filesystem +# for the chunks. +schema: + configs: + - from: 2019-07-29 + store: boltdb + object_store: filesystem + schema: v10 + index: + prefix: index_ + period: 1w + +storage: + boltdb: + directory: /tmp/cortex/index + + filesystem: + directory: /tmp/cortex/chunks + + delete_store: + store: boltdb + +purger: + object_store_type: filesystem + +frontend_worker: + # Configure the frontend worker in the querier to match worker count + # to max_concurrent on the queriers. + match_max_concurrent: true + +# Configure the ruler to scan the /tmp/cortex/rules directory for prometheus +# rules: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules +ruler: + enable_api: true + enable_sharding: false + storage: + type: local + local: + directory: /tmp/cortex/rules + diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml index 15ccd3d705..61e6f4981e 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml @@ -1,12 +1,33 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3.8" + +services: cortex: image: quay.io/cortexproject/cortex:v1.5.0 command: - -config.file=./config/cortex-config.yml volumes: - - ./prometheus-remote-write-cortex/cortex-config.yml:/config/cortex-config.yml:ro + - ./cortex-config.yml:/config/cortex-config.yml:ro ports: - 9009:9009 grafana: image: grafana/grafana:latest ports: - 3000:3000 + sample_app: + build: + context: ../ + dockerfile: ./examples/Dockerfile diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/requirements.txt b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/requirements.txt new file mode 100644 index 0000000000..c77b8b32ce --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/requirements.txt @@ -0,0 +1,7 @@ +psutil +protobuf>=3.13.0 +requests==2.25.0 +python-snappy +opentelemetry-api +opentelemetry-sdk +opentelemetry-proto diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py new file mode 100644 index 0000000000..2adc586609 --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py @@ -0,0 +1,154 @@ +from logging import INFO +from opentelemetry.sdk.metrics.export.aggregate import MinMaxSumCountAggregator +import psutil +import time +import random +import logging +import sys +from opentelemetry import metrics +from opentelemetry.sdk.metrics import MeterProvider +from opentelemetry.exporter.prometheus_remote_write import ( + PrometheusRemoteWriteMetricsExporter +) +from opentelemetry.sdk.metrics.export import ConsoleMetricsExporter +from opentelemetry.sdk.metrics.view import View, ViewConfig + +from opentelemetry.sdk.metrics.export.aggregate import ( + HistogramAggregator, + LastValueAggregator, + MinMaxSumCountAggregator, + SumAggregator, +) + +logging.basicConfig(stream=sys.stdout, level=logging.INFO) +logger = logging.getLogger(__name__) +metrics.set_meter_provider(MeterProvider()) +meter = metrics.get_meter(__name__) +exporter = PrometheusRemoteWriteMetricsExporter( + endpoint="http://cortex:9009/api/prom/push", + headers={"X-Scope-Org-ID": "5"} +) +metrics.get_meter_provider().start_pipeline(meter, exporter, 1) +testing_labels = {"environment": "testing"} + + +# Callback to gather cpu usage +def get_cpu_usage_callback(observer): + for (number, percent) in enumerate(psutil.cpu_percent(percpu=True)): + labels = {"cpu_number": str(number)} + observer.observe(percent, labels) + + +# Callback to gather RAM usage +def get_ram_usage_callback(observer): + ram_percent = psutil.virtual_memory().percent + observer.observe(ram_percent, {}) + + +requests_counter = meter.create_counter( + name="requests", + description="number of requests", + unit="1", + value_type=int, +) + +request_min_max = meter.create_counter( + name="requests_min_max", + description="min max sum count of requests", + unit="1", + value_type=int, +) + +request_last_value = meter.create_counter( + name="requests_last_value", + description="last value number of requests", + unit="1", + value_type=int, +) + +requests_size = meter.create_valuerecorder( + name="requests_size", + description="size of requests", + unit="1", + value_type=int, +) + +requests_size_histogram = meter.create_valuerecorder( + name="requests_size_histogram", + description="histogram of request_size", + unit="1", + value_type=int, +) +requests_active = meter.create_updowncounter( + name="requests_active", + description="number of active requests", + unit="1", + value_type=int, +) + +meter.register_sumobserver( + callback=get_ram_usage_callback, + name="ram_usage", + description="ram usage", + unit="1", + value_type=float, +) + +meter.register_valueobserver( + callback=get_cpu_usage_callback, + name="cpu_percent", + description="per-cpu usage", + unit="1", + value_type=float, +) + + +counter_view1 = View( + requests_counter, + SumAggregator, + label_keys=["environment"], + view_config=ViewConfig.LABEL_KEYS, +) +counter_view2 = View( + request_min_max, + MinMaxSumCountAggregator, + label_keys=["os_type"], + view_config=ViewConfig.LABEL_KEYS, +) +# This view has ViewConfig set to UNGROUPED, meaning all recorded metrics take +# the labels directly without and consideration for label_keys +counter_view3 = View( + request_last_value, + LastValueAggregator, + label_keys=["environment"], # is not used due to ViewConfig.UNGROUPED + view_config=ViewConfig.UNGROUPED, +) +size_view = View( + requests_size_histogram, + HistogramAggregator, + label_keys=["environment"], # is not used due to ViewConfig.UNGROUPED + aggregator_config={"bounds": [20, 40, 60, 80, 100]}, + view_config=ViewConfig.UNGROUPED, +) +meter.register_view(counter_view1) +meter.register_view(counter_view2) +meter.register_view(counter_view3) +meter.register_view(size_view) + +# Load generator +num = random.randint(0, 1000) +while(True): + # counters + requests_counter.add(num % 131 + 200, testing_labels) + request_min_max.add(num % 181 + 200, testing_labels) + request_last_value.add(num % 101 + 200, testing_labels) + + # updown counter + requests_active.add(num % 7231 + 200, testing_labels) + + # value observers + requests_size.record(num % 6101 + 100, testing_labels) + requests_size_histogram.record(num % 113, testing_labels) + logger.log(level=INFO, msg="completed metrics collection cycle") + time.sleep(1) + num += 9791 diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/setup.cfg b/exporter/opentelemetry-exporter-prometheus-remote-write/setup.cfg index 7d2869022b..046bf0d01f 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/setup.cfg +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/setup.cfg @@ -39,8 +39,8 @@ package_dir= =src packages=find_namespace: install_requires = - opentelemetry-api == 0.16.dev0 - opentelemetry-sdk == 0.16.dev0 + opentelemetry-api >= 0.16b0 + opentelemetry-sdk >= 0.16b0 [options.packages.find] where = src From d943098a11c38436881534c84d45ee497ae82f9d Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Fri, 27 Nov 2020 14:58:15 -0500 Subject: [PATCH 03/19] adding examples readme --- .../README.md | 176 +++++++++++++++ .../examples/README.md | 208 +++--------------- .../examples/sampleapp.py | 7 +- 3 files changed, 211 insertions(+), 180 deletions(-) create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/README.md diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md new file mode 100644 index 0000000000..92a37f6314 --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md @@ -0,0 +1,176 @@ +# OpenTelemetry Python SDK Prometheus Remote Write Exporter +This package contains an exporter to send [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md) +metrics from the Python SDK directly to a Prometheus Remote Write integrated +backend (such as Cortex or Thanos) without having to run an instance of the +Prometheus server. The image below shows the two Prometheus exporters in the OpenTelemetry Python SDK. + + +Pipeline 1 illustrates the setup required for a Prometheus "pull" exporter. + + +Pipeline 2 illustrates the setup required for the Prometheus Remote Write exporter. + +![Prometheus SDK pipelines](https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png) + + +The Prometheus Remote Write Exporter is a "push" based exporter and only works with the OpenTelemetry [push controller](https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py). +The controller periodically collects data and passes it to the exporter. This +exporter then converts the data into [`timeseries`](https://prometheus.io/docs/concepts/data_model/) and sends it to the Remote Write integrated backend through HTTP +POST requests. The metrics collection datapath is shown below: + +![Metrics collection datapath](https://user-images.githubusercontent.com/20804975/100286063-157f2a80-2f40-11eb-819d-f0aa46c1c2c8.png) + +See the `example` folder for a demo usage of this exporter + +# Table of Contents + * [Summary](#opentelemetry-python-sdk-prometheus-remote-write-exporter) + * [Table of Contents](#table-of-contents) + * [Installation](#installation) + * [Quickstart](#quickstart) + * [Configuring the Exporter](#configuring-the-exporter) + * [Securing the Exporter](#securing-the-exporter) + * [Authentication](#authentication) + * [TLS](#tls) + * [Supported Aggregators](#supported-aggregators) + * [Error Handling](#error-handling) + * [Retry Logic](#retry-logic) + * [Contributing](#contributing) + * [Design Doc](#design-doc) + +## Installation + +* To install from the latest PyPi release, + run `pip install opentelemetry-exporter-prometheus-remote-write` +* To install from the local repository, run + `pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/` in + the project root + +## Quickstart + +```python +from opentelemetry import metrics +from opentelemetry.sdk.metrics import MeterProvider +from opentelemetry.exporter.prometheus_remote_write import ( + PrometheusRemoteWriteMetricsExporter +) + +# Sets the global MeterProvider instance +metrics.set_meter_provider(MeterProvider()) + +# The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here. +meter = metrics.get_meter(__name__) + +exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed + +metrics.get_meter_provider().start_pipeline(meter, exporter, 5) +``` + +## Configuring the Exporter + +The exporter can be configured through parameters passed to the constructor. +Here are all the options: + +* `endpoint`: url where data will be sent **(Required)** +* `basic_auth`: username and password for authentication **(Optional)** +* `headers`: additional headers for remote write request as determined by the remote write backend's API **(Optional)** +* `timeout`: timeout for requests to the remote write endpoint in seconds **(Optional)** +* `proxies`: dict mapping request proxy protocols to proxy urls **(Optional)** +* `tls_config`: configuration for remote write TLS settings **(Optional)** + +Example with all the configuration options: + +```python +exporter = PrometheusRemoteWriteMetricsExporter( + endpoint="http://localhost:9009/api/prom/push", + timeout=30, + basic_auth={ + "username": "user", + "password": "pass123", + }, + headers={ + "X-Scope-Org-ID": "5", + "Authorization": "Bearer mytoken123", + }, + proxies={ + "http": "http://10.10.1.10:3000", + "https": "http://10.10.1.10:1080", + }, + tls_config={ + "cert_file": "path/to/file", + "key_file": "path/to/file", + } +) + +``` +## Securing the Exporter + +### Authentication + +The exporter provides two forms of authentication which are shown below. Users +can add their own custom authentication by setting the appropriate values in the `headers` dictionary + +1. Basic Authentication +Basic authentication sets a HTTP Authorization header containing a base64 encoded username/password pair. See [RFC 7617](https://tools.ietf.org/html/rfc7617) for more information. This + +```python +exporter = PrometheusRemoteWriteMetricsExporter( + basic_auth={"username": "base64user", "password": "base64pass"} +) +``` +2. Bearer Token Authentication +This custom configuration can be achieved by passing in a custom `header` to +the constructor. See [RFC 6750](https://tools.ietf.org/html/rfc6750) for more information. + + +```python +header = { + "Authorization": "Bearer mytoken123" +} +``` + +### TLS +Users can add TLS to the exporter's HTTP Client by providing certificate and key files in the `tls_config` parameter. + +## Supported Aggregators + +* Sum +* MinMaxSumCount +* Histogram +* LastValue +* ValueObserver +* Summary + +## Error Handling +In general, errors are returned to the calling function. The exception is for +the exporter's `export()` method where any error status code is logged as a +warning instead. + +This is because the exporter does not implement any retry logic +as it sends cumulative metrics data. This means that data will be preserved even if some exports fail. + +For example, consider a situation where a user increments a Counter instrument 5 times and an export happens between each increment. If the exports happen like so: +``` +SUCCESS FAIL FAIL SUCCESS SUCCESS +1 2 3 4 5 +``` +Then the recieved data will be: +``` +1 4 5 +``` +The end result is the same since the aggregations are cumulative +## Contributing + +This exporter's datapath is as follows: + +![Exporter datapath](https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png) +*Entites with `*` after their name are not actual classes but rather logical +groupings of functions within the exporter.* + +If you would like to learn more about the exporter's structure and design decisions please view the design document below + +### Design Doc + +[Design Document](TODO: add link) + +This document is not in this repo as it contains large images which will +significantly increase the size this repo. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md index 7df0e76a5a..91f7ead578 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md @@ -1,186 +1,42 @@ -# OpenTelemetry Python SDK Prometheus Remote Write Exporter -This package contains an exporter to send [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md) -metrics from the Python SDK directly to a Prometheus Remote Write integrated -backend (such as Cortex or Thanos) without having to run an instance of the -Prometheus server. The image below shows the two Prometheus exporters in the OpenTelemetry Python SDK. +# Prometheus Remote Write Exporter Example +This example uses [Docker Compose](https://docs.docker.com/compose/) to set up: +1. A Python program that creates 5 instruments with 5 unique +aggregators and a randomized load generator +2. An instance of [Cortex](https://cortexmetrics.io/) to recieve the metrics +data +3. An instance of [Grafana](https://grafana.com/) to visualizse the exported +data -Pipeline 1 illustrates the setup required for a Prometheus "pull" exporter. +## Requirements +* Have Docker Compose [installed](https://docs.docker.com/compose/install/) +*Users do not need to install Python as the app will be run in the Docker Container* -Pipeline 2 illustrates the setup required for the Prometheus Remote Write exporter. +## Instructions +1. Run `docker-compose up -d` in the the `examples/` directory -![Prometheus SDK pipelines](https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png) +The `-d` flag causes all services to run in detached mode and frees up your +terminal session. This also causes no logs to show up. Users can attach themselves to the service's logs manually using `docker logs ${CONTAINER_ID} --follow` +2. Log into the Grafana instance at [http://localhost:3000](http://localhost:3000) + * login credentials are `username: admin` and `password: admin` + * There may be an additional screen on setting a new password. This can be skipped and is optional -The Prometheus Remote Write Exporter is a "push" based exporter and only works with the OpenTelemetry [push controller](https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py). -The controller periodically collects data and passes it to the exporter. This -exporter then converts the data into [`timeseries`](https://prometheus.io/docs/concepts/data_model/) and sends it to the Remote Write integrated backend through HTTP -POST requests. The metrics collection datapath is shown below: +3. Navigate to the `Data Sources` page + * Look for a gear icon on the left sidebar and select `Data Sources` -![Metrics collection datapath](https://user-images.githubusercontent.com/20804975/100286063-157f2a80-2f40-11eb-819d-f0aa46c1c2c8.png) +4. Add a new Prometheus Data Source + * Use `http://cortex:9009/api/prom` as the URL + * (OPTIONAl) set the scrape interval to `2s` to make updates appear quickly + * click `Save & Test` +5. Go to `Metrics Explore` to query metrics + * Look for a compass icon on the left sidebar + * click `Metrics` for a dropdown list of all the available metrics + * (OPTIONAL) Adjust time range by clicking the `Last 6 hours` button on the upper right side of the graph + * (OPTIONAL) Set up auto-refresh by selecting an option under the dropdown next to the refresh button on the upper right side of the graph + * Click the refresh button and data should show up on hte graph - - -See the `example` folder for a demo usage of this exporter - - - -# Table of Contents - * [Summary](#opentelemetry-python-sdk-prometheus-remote-write-exporter) - * [Table of Contents](#table-of-contents) - * [Installation](#installation) - * [Quickstart](#quickstart) - * [Example](#example) - * [Configuring the Exporter](#configuring-the-exporter) - * [Securing the Exporter](#securing-the-exporter) - * [Authentication](#authentication) - * [TLS](#tls) - * [Supported Aggregators](#supported-aggregators) - * [Error Handling](#error-handling) - * [Retry Logic](#retry-logic) - * [Contributing](#contributing) - * [Design Doc](#design-doc) - -## Installation - -* To install from the latest PyPi release, - run `pip install opentelemetry-exporter-prometheus-remote-write` -* To install from the local repository, run - `pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/` in - the project root - -## Quickstart - -```python -from opentelemetry import metrics -from opentelemetry.sdk.metrics import MeterProvider -from opentelemetry.exporter.prometheus_remote_write import ( - PrometheusRemoteWriteMetricsExporter -) - -# Sets the global MeterProvider instance -metrics.set_meter_provider(MeterProvider()) - -# The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here. -meter = metrics.get_meter(__name__) - -exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed - -metrics.get_meter_provider().start_pipeline(meter, exporter, 5) -``` - -## Example - -* clone core -* TODO: change dockerfile if release 0.16 is public -## Configuring the Exporter - -The exporter can be configured through parameters passed to the constructor. -Here are all the options: - -* `endpoint`: url where data will be sent **(Required)** -* `basic_auth`: username and password for authentication **(Optional)** -* `headers`: additional headers for remote write request as determined by the remote write backend's API **(Optional)** -* `timeout`: timeout for requests to the remote write endpoint in seconds **(Optional)** -* `proxies`: dict mapping request proxy protocols to proxy urls **(Optional)** -* `tls_config`: configuration for remote write TLS settings **(Optional)** - -Example with all the configuration options: - -```python -exporter = PrometheusRemoteWriteMetricsExporter( - endpoint="http://localhost:9009/api/prom/push", - timeout=30, - basic_auth={ - "username": "user", - "password": "pass123", - }, - headers={ - "X-Scope-Org-ID": "5", - "Authorization": "Bearer mytoken123", - }, - proxies={ - "http": "http://10.10.1.10:3000", - "https": "http://10.10.1.10:1080", - }, - tls_config={ - "cert_file": "path/to/file", - "key_file": "path/to/file", - } -) - -``` -## Securing the Exporter - -### Authentication - -The exporter provides two forms of authentication which are shown below. Users -can add their own custom authentication by setting the appropriate values in the `headers` dictionary - -1. Basic Authentication -Basic authentication sets a HTTP Authorization header containing a base64 encoded username/password pair. See [RFC 7617](https://tools.ietf.org/html/rfc7617) for more information. This - -```python -exporter = PrometheusRemoteWriteMetricsExporter( - basic_auth={"username": "base64user", "password": "base64pass"} -) -``` -2. Bearer Token Authentication -This custom configuration can be achieved by passing in a custom `header` to -the constructor. See [RFC 6750](https://tools.ietf.org/html/rfc6750) for more information. - - -```python -header = { - "Authorization": "Bearer mytoken123" -} -``` - -### TLS -Users can add TLS to the exporter's HTTP Client by providing certificate and key files in the `tls_config` parameter. - -## Supported Aggregators - -* Sum -* MinMaxSumCount -* Histogram -* LastValue -* ValueObserver -* Summary - -## Error Handling -In general, errors are returned to the calling function. The exception is for -the exporter's `export()` method where any error status code is logged as a -warning instead. - -This is because the exporter does not implement any retry logic -as it sends cumulative metrics data. This means that data will be preserved even if some exports fail. - -For example, consider a situation where a user increments a Counter instrument 5 times and an export happens between each increment. If the exports happen like so: -``` -SUCCESS FAIL FAIL SUCCESS SUCCESS -1 2 3 4 5 -``` -Then the recieved data will be: -``` -1 4 5 -``` -The end result is the same since the aggregations are cumulative -## Contributing - -This exporter's datapath is as follows: - -![Exporter datapath](https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png) -*Entites with `*` after their name are not actual classes but rather logical -groupings of functions within the exporter.* - -If you would like to learn more about the exporter's structure and design decisions please view the design document below - -### Design Doc - -[Design Document](TODO: add link) - -This document is not in this repo as it contains large images which will -significantly increase the size this repo. +6. Shutdown the services when finished + * Run `docker-compose down` in the examples directory \ No newline at end of file diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py index 2adc586609..8dc25f3440 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py @@ -8,9 +8,8 @@ from opentelemetry import metrics from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.exporter.prometheus_remote_write import ( - PrometheusRemoteWriteMetricsExporter + PrometheusRemoteWriteMetricsExporter, ) -from opentelemetry.sdk.metrics.export import ConsoleMetricsExporter from opentelemetry.sdk.metrics.view import View, ViewConfig from opentelemetry.sdk.metrics.export.aggregate import ( @@ -26,7 +25,7 @@ meter = metrics.get_meter(__name__) exporter = PrometheusRemoteWriteMetricsExporter( endpoint="http://cortex:9009/api/prom/push", - headers={"X-Scope-Org-ID": "5"} + headers={"X-Scope-Org-ID": "5"}, ) metrics.get_meter_provider().start_pipeline(meter, exporter, 1) testing_labels = {"environment": "testing"} @@ -137,7 +136,7 @@ def get_ram_usage_callback(observer): # Load generator num = random.randint(0, 1000) -while(True): +while True: # counters requests_counter.add(num % 131 + 200, testing_labels) request_min_max.add(num % 181 + 200, testing_labels) From cc4aa7c8d253f28e3492ee3337407d8a06d2e401 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Fri, 27 Nov 2020 17:16:15 -0500 Subject: [PATCH 04/19] fixing lint errors --- .flake8 | 1 + .../README.md | 11 ++++----- .../examples/sampleapp.py | 24 +++++++++---------- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/.flake8 b/.flake8 index d0e523dbc3..416eb2dec1 100644 --- a/.flake8 +++ b/.flake8 @@ -20,6 +20,7 @@ exclude = exporter/opentelemetry-exporter-jaeger/src/opentelemetry/exporter/jaeger/gen/ exporter/opentelemetry-exporter-jaeger/build/* exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen + exporter/opentelemetry-exporter-prometheus-remote-write/examples/* docs/examples/opentelemetry-example-app/src/opentelemetry_example_app/grpc/gen/ docs/examples/opentelemetry-example-app/build/* opentelemetry-python-core/ diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md index 92a37f6314..8f8889bc35 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md @@ -18,7 +18,7 @@ The controller periodically collects data and passes it to the exporter. This exporter then converts the data into [`timeseries`](https://prometheus.io/docs/concepts/data_model/) and sends it to the Remote Write integrated backend through HTTP POST requests. The metrics collection datapath is shown below: -![Metrics collection datapath](https://user-images.githubusercontent.com/20804975/100286063-157f2a80-2f40-11eb-819d-f0aa46c1c2c8.png) +![controller_datapath_final](https://user-images.githubusercontent.com/20804975/100486582-79d1f380-30d2-11eb-8d17-d3e58e5c34e9.png) See the `example` folder for a demo usage of this exporter @@ -138,11 +138,10 @@ Users can add TLS to the exporter's HTTP Client by providing certificate and key * Histogram * LastValue * ValueObserver -* Summary ## Error Handling -In general, errors are returned to the calling function. The exception is for -the exporter's `export()` method where any error status code is logged as a +In general, errors are raised by the calling function. The exception is for +failed requests where any error status code is logged as a warning instead. This is because the exporter does not implement any retry logic @@ -172,5 +171,5 @@ If you would like to learn more about the exporter's structure and design decisi [Design Document](TODO: add link) -This document is not in this repo as it contains large images which will -significantly increase the size this repo. +This document is stored elsewhere as it contains large images which will +significantly increase the size of this repo. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py index 8dc25f3440..69f7a068ea 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/examples/sampleapp.py @@ -1,26 +1,27 @@ -from logging import INFO -from opentelemetry.sdk.metrics.export.aggregate import MinMaxSumCountAggregator -import psutil -import time -import random import logging +import random import sys +import time +from logging import INFO + +import psutil + from opentelemetry import metrics -from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.exporter.prometheus_remote_write import ( PrometheusRemoteWriteMetricsExporter, ) -from opentelemetry.sdk.metrics.view import View, ViewConfig - +from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export.aggregate import ( HistogramAggregator, LastValueAggregator, MinMaxSumCountAggregator, SumAggregator, ) +from opentelemetry.sdk.metrics.view import View, ViewConfig logging.basicConfig(stream=sys.stdout, level=logging.INFO) logger = logging.getLogger(__name__) + metrics.set_meter_provider(MeterProvider()) meter = metrics.get_meter(__name__) exporter = PrometheusRemoteWriteMetricsExporter( @@ -114,18 +115,17 @@ def get_ram_usage_callback(observer): label_keys=["os_type"], view_config=ViewConfig.LABEL_KEYS, ) -# This view has ViewConfig set to UNGROUPED, meaning all recorded metrics take -# the labels directly without and consideration for label_keys + counter_view3 = View( request_last_value, LastValueAggregator, - label_keys=["environment"], # is not used due to ViewConfig.UNGROUPED + label_keys=["environment"], view_config=ViewConfig.UNGROUPED, ) size_view = View( requests_size_histogram, HistogramAggregator, - label_keys=["environment"], # is not used due to ViewConfig.UNGROUPED + label_keys=["environment"], aggregator_config={"bounds": [20, 40, 60, 80, 100]}, view_config=ViewConfig.UNGROUPED, ) From d25c75442ad1cda8a90f1c11bc3e0e8076298f9d Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 12:49:52 -0500 Subject: [PATCH 05/19] linting examples --- .flake8 | 1 - 1 file changed, 1 deletion(-) diff --git a/.flake8 b/.flake8 index 416eb2dec1..d0e523dbc3 100644 --- a/.flake8 +++ b/.flake8 @@ -20,7 +20,6 @@ exclude = exporter/opentelemetry-exporter-jaeger/src/opentelemetry/exporter/jaeger/gen/ exporter/opentelemetry-exporter-jaeger/build/* exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen - exporter/opentelemetry-exporter-prometheus-remote-write/examples/* docs/examples/opentelemetry-example-app/src/opentelemetry_example_app/grpc/gen/ docs/examples/opentelemetry-example-app/build/* opentelemetry-python-core/ From 59b9aa01bbfba08250e6a172a634e5b71a739723 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 12:52:33 -0500 Subject: [PATCH 06/19] updating readme tls_config example --- .../opentelemetry-exporter-prometheus-remote-write/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md index 8f8889bc35..857671c2a9 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md @@ -98,6 +98,8 @@ exporter = PrometheusRemoteWriteMetricsExporter( tls_config={ "cert_file": "path/to/file", "key_file": "path/to/file", + "ca_file": "path_to_file", + "insecure_skip_verify": true, # for developing purposes } ) From 2f1b6479d272cea527808ea26c0578181ea97628 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 13:39:17 -0500 Subject: [PATCH 07/19] excluding examples --- .flake8 | 3 ++- .pylintrc | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.flake8 b/.flake8 index d0e523dbc3..4c80218872 100644 --- a/.flake8 +++ b/.flake8 @@ -4,7 +4,7 @@ ignore = F401 # unused import, defer to pylint W503 # allow line breaks before binary ops W504 # allow line breaks after binary ops - E203 # allow whitespace before ':' (https://github.com/psf/black#slices) + E203 # allow whitespace before ':' (https://github.com/psf/black#slices) exclude = .bzr .git @@ -20,6 +20,7 @@ exclude = exporter/opentelemetry-exporter-jaeger/src/opentelemetry/exporter/jaeger/gen/ exporter/opentelemetry-exporter-jaeger/build/* exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen + exporter/opentelemetry-exporter-prometheus-remote-write/examples docs/examples/opentelemetry-example-app/src/opentelemetry_example_app/grpc/gen/ docs/examples/opentelemetry-example-app/build/* opentelemetry-python-core/ diff --git a/.pylintrc b/.pylintrc index 866f003776..95c8358490 100644 --- a/.pylintrc +++ b/.pylintrc @@ -192,7 +192,7 @@ ignored-classes=optparse.Values,thread._local,_thread._local # (useful for modules/projects where namespaces are manipulated during runtime # and thus existing member attributes cannot be deduced by static analysis. It # supports qualified module names, as well as Unix pattern matching. -ignored-modules= +ignored-modules=docker-compose,README,Dockerfile,cortex-config,requirements # Show a hint with possible names when a member name was not found. The aspect # of finding the hint is based on edit distance. From 98dac1f171384c0a64ff469941440358c611b47c Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:00:49 -0500 Subject: [PATCH 08/19] adding examples to exclude in all linters --- .pylintrc | 2 +- pyproject.toml | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.pylintrc b/.pylintrc index 95c8358490..cb6bb41742 100644 --- a/.pylintrc +++ b/.pylintrc @@ -165,7 +165,7 @@ contextmanager-decorators=contextlib.contextmanager # List of members which are set dynamically and missed by pylint inference # system, and so shouldn't trigger E1101 when accessed. Python regular # expressions are accepted. -generated-members=*_pb2.py +generated-members=*_pb2.py,*.yml,*.md # Tells whether missing members accessed in mixin class should be ignored. A # mixin class is detected if its name ends with "mixin" (case insensitive). diff --git a/pyproject.toml b/pyproject.toml index 5dbe095fe8..bb4e91fcbc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,8 @@ exclude = ''' ( /( reference| # original files from DataDog - exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen + exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen| + exporter/opentelemetry-exporter-prometheus-remote-write/examples )/ ) ''' From 32f70146d6f5305801d7c2bd00bac0c96ee9b3af Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:07:11 -0500 Subject: [PATCH 09/19] adding isort.cfg skip --- .isort.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.isort.cfg b/.isort.cfg index 47fc6468cb..62d04f0587 100644 --- a/.isort.cfg +++ b/.isort.cfg @@ -13,6 +13,6 @@ line_length=79 ; docs: https://github.com/timothycrosley/isort#multi-line-output-modes multi_line_output=3 skip=target -skip_glob=**/gen/*,.venv*/*,venv*/*,reference*/*,opentelemetry-python-core/* +skip_glob=**/gen/*,.venv*/*,venv*/*,reference*/*,opentelemetry-python-core/*,*.yml,*.md,Dockerfile known_first_party=opentelemetry known_third_party=psutil,pytest,redis,redis_opentracing From caf76ab6cc5ae7e189ccc3beb51033544a905160 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:15:36 -0500 Subject: [PATCH 10/19] changing isort to path --- .isort.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.isort.cfg b/.isort.cfg index 62d04f0587..22eb7fac51 100644 --- a/.isort.cfg +++ b/.isort.cfg @@ -13,6 +13,6 @@ line_length=79 ; docs: https://github.com/timothycrosley/isort#multi-line-output-modes multi_line_output=3 skip=target -skip_glob=**/gen/*,.venv*/*,venv*/*,reference*/*,opentelemetry-python-core/*,*.yml,*.md,Dockerfile +skip_glob=**/gen/*,.venv*/*,venv*/*,reference*/*,opentelemetry-python-core/*,**/examples/* known_first_party=opentelemetry known_third_party=psutil,pytest,redis,redis_opentracing From e9586ec06dbfb11e07462e70e3f5e6dc095a0158 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:36:38 -0500 Subject: [PATCH 11/19] ignoring yml only --- .pylintrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pylintrc b/.pylintrc index cb6bb41742..5044afabce 100644 --- a/.pylintrc +++ b/.pylintrc @@ -11,7 +11,7 @@ ignore=CVS,gen # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. -ignore-patterns= +ignore-patterns=*.yml # Python code to execute, usually for sys.path manipulation such as # pygtk.require(). From fcedce246bad291737837aa137569276819cf9c2 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:39:15 -0500 Subject: [PATCH 12/19] adding it to excluded directories in pylintrc --- .pylintrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pylintrc b/.pylintrc index 5044afabce..f390b99d3f 100644 --- a/.pylintrc +++ b/.pylintrc @@ -7,7 +7,7 @@ extension-pkg-whitelist= # Add list of files or directories to be excluded. They should be base names, not # paths. -ignore=CVS,gen +ignore=CVS,gen,examples # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. From 1990ad8cf3cd921b19a20a844c16a7c6edadb669 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:44:10 -0500 Subject: [PATCH 13/19] only adding exclude to directory --- .flake8 | 1 - .isort.cfg | 2 +- .pylintrc | 4 ++-- pyproject.toml | 3 +-- 4 files changed, 4 insertions(+), 6 deletions(-) diff --git a/.flake8 b/.flake8 index 4c80218872..b7b9326368 100644 --- a/.flake8 +++ b/.flake8 @@ -20,7 +20,6 @@ exclude = exporter/opentelemetry-exporter-jaeger/src/opentelemetry/exporter/jaeger/gen/ exporter/opentelemetry-exporter-jaeger/build/* exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen - exporter/opentelemetry-exporter-prometheus-remote-write/examples docs/examples/opentelemetry-example-app/src/opentelemetry_example_app/grpc/gen/ docs/examples/opentelemetry-example-app/build/* opentelemetry-python-core/ diff --git a/.isort.cfg b/.isort.cfg index 22eb7fac51..47fc6468cb 100644 --- a/.isort.cfg +++ b/.isort.cfg @@ -13,6 +13,6 @@ line_length=79 ; docs: https://github.com/timothycrosley/isort#multi-line-output-modes multi_line_output=3 skip=target -skip_glob=**/gen/*,.venv*/*,venv*/*,reference*/*,opentelemetry-python-core/*,**/examples/* +skip_glob=**/gen/*,.venv*/*,venv*/*,reference*/*,opentelemetry-python-core/* known_first_party=opentelemetry known_third_party=psutil,pytest,redis,redis_opentracing diff --git a/.pylintrc b/.pylintrc index f390b99d3f..59150fe510 100644 --- a/.pylintrc +++ b/.pylintrc @@ -11,7 +11,7 @@ ignore=CVS,gen,examples # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. -ignore-patterns=*.yml +ignore-patterns= # Python code to execute, usually for sys.path manipulation such as # pygtk.require(). @@ -165,7 +165,7 @@ contextmanager-decorators=contextlib.contextmanager # List of members which are set dynamically and missed by pylint inference # system, and so shouldn't trigger E1101 when accessed. Python regular # expressions are accepted. -generated-members=*_pb2.py,*.yml,*.md +generated-members=*_pb2.py # Tells whether missing members accessed in mixin class should be ignored. A # mixin class is detected if its name ends with "mixin" (case insensitive). diff --git a/pyproject.toml b/pyproject.toml index bb4e91fcbc..5dbe095fe8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,8 +4,7 @@ exclude = ''' ( /( reference| # original files from DataDog - exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen| - exporter/opentelemetry-exporter-prometheus-remote-write/examples + exporter/opentelemetry-exporter-prometheus-remote-write/src/opentelemetry/exporter/prometheus_remote_write/gen )/ ) ''' From 9c3feca77537609bc7ed423f613f5f3408eebbe9 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:49:53 -0500 Subject: [PATCH 14/19] removing readme.rst and adding explicit file names to ignore --- .pylintrc | 2 +- .../README.rst | 27 ------------------- 2 files changed, 1 insertion(+), 28 deletions(-) delete mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/README.rst diff --git a/.pylintrc b/.pylintrc index 59150fe510..95cab04801 100644 --- a/.pylintrc +++ b/.pylintrc @@ -7,7 +7,7 @@ extension-pkg-whitelist= # Add list of files or directories to be excluded. They should be base names, not # paths. -ignore=CVS,gen,examples +ignore=CVS,gen,examples,Dockerfile,docker-compose.yml,README # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst b/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst deleted file mode 100644 index f204fdb32d..0000000000 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst +++ /dev/null @@ -1,27 +0,0 @@ -OpenTelemetry Prometheus Remote Write Exporter -============================================== - -This library allows exporting metric data to `Prometheus Remote Write Integrated Backends -`_. Latest `types.proto -` and `remote.proto -` Protocol Buffers -used to create WriteRequest objects were taken from Prometheus repository. Development is -currently in progress. - -Installation ------------- - -:: - - pip install opentelemetry-exporter-prometheus-remote-write - - -.. _Prometheus: https://prometheus.io/ -.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/ - - -References ----------- - -* `Prometheus `_ -* `OpenTelemetry Project `_ From 7450d7aba8fe8a7655f7868b73b2320a95d78b8f Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 14:54:03 -0500 Subject: [PATCH 15/19] adding the rest of the files --- .pylintrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pylintrc b/.pylintrc index 95cab04801..e3f4b1fbd2 100644 --- a/.pylintrc +++ b/.pylintrc @@ -7,7 +7,7 @@ extension-pkg-whitelist= # Add list of files or directories to be excluded. They should be base names, not # paths. -ignore=CVS,gen,examples,Dockerfile,docker-compose.yml,README +ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. From 37dadcbd5220e7710ab7a3c796cacab03c7670d5 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 15:05:22 -0500 Subject: [PATCH 16/19] adding readme.rst back --- .../README.rst | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/README.rst diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst b/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst new file mode 100644 index 0000000000..f204fdb32d --- /dev/null +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst @@ -0,0 +1,27 @@ +OpenTelemetry Prometheus Remote Write Exporter +============================================== + +This library allows exporting metric data to `Prometheus Remote Write Integrated Backends +`_. Latest `types.proto +` and `remote.proto +` Protocol Buffers +used to create WriteRequest objects were taken from Prometheus repository. Development is +currently in progress. + +Installation +------------ + +:: + + pip install opentelemetry-exporter-prometheus-remote-write + + +.. _Prometheus: https://prometheus.io/ +.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/ + + +References +---------- + +* `Prometheus `_ +* `OpenTelemetry Project `_ From c21dd66ddc24d29d08fc18af158079ae6ffdcc27 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 15:15:14 -0500 Subject: [PATCH 17/19] adding to ignore glob instead --- .pylintrc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.pylintrc b/.pylintrc index e3f4b1fbd2..f4ee0f81a8 100644 --- a/.pylintrc +++ b/.pylintrc @@ -7,11 +7,11 @@ extension-pkg-whitelist= # Add list of files or directories to be excluded. They should be base names, not # paths. -ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml - +# ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml +ignore=CVS,gen # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. -ignore-patterns= +ignore-patterns=examples/ # Python code to execute, usually for sys.path manipulation such as # pygtk.require(). From 46be9e2c1554779c3fbf0d72855d9bf1f75727a2 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Mon, 30 Nov 2020 15:37:20 -0500 Subject: [PATCH 18/19] reverting back to ignore list --- .pylintrc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.pylintrc b/.pylintrc index f4ee0f81a8..e3f4b1fbd2 100644 --- a/.pylintrc +++ b/.pylintrc @@ -7,11 +7,11 @@ extension-pkg-whitelist= # Add list of files or directories to be excluded. They should be base names, not # paths. -# ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml -ignore=CVS,gen +ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml + # Add files or directories matching the regex patterns to be excluded. The # regex matches against base names, not paths. -ignore-patterns=examples/ +ignore-patterns= # Python code to execute, usually for sys.path manipulation such as # pygtk.require(). From 59b04acf3b9bc20f55fc9d71a1b1b11425a1d0a1 Mon Sep 17 00:00:00 2001 From: Azfaar Qureshi Date: Tue, 1 Dec 2020 18:26:26 -0500 Subject: [PATCH 19/19] converting README.md to README.rst --- .pylintrc | 2 +- .../README.md | 177 ------------- .../README.rst | 239 ++++++++++++++++-- 3 files changed, 225 insertions(+), 193 deletions(-) delete mode 100644 exporter/opentelemetry-exporter-prometheus-remote-write/README.md diff --git a/.pylintrc b/.pylintrc index e3f4b1fbd2..be29414dc3 100644 --- a/.pylintrc +++ b/.pylintrc @@ -192,7 +192,7 @@ ignored-classes=optparse.Values,thread._local,_thread._local # (useful for modules/projects where namespaces are manipulated during runtime # and thus existing member attributes cannot be deduced by static analysis. It # supports qualified module names, as well as Unix pattern matching. -ignored-modules=docker-compose,README,Dockerfile,cortex-config,requirements +ignored-modules= # Show a hint with possible names when a member name was not found. The aspect # of finding the hint is based on edit distance. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md b/exporter/opentelemetry-exporter-prometheus-remote-write/README.md deleted file mode 100644 index 857671c2a9..0000000000 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/README.md +++ /dev/null @@ -1,177 +0,0 @@ -# OpenTelemetry Python SDK Prometheus Remote Write Exporter -This package contains an exporter to send [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md) -metrics from the Python SDK directly to a Prometheus Remote Write integrated -backend (such as Cortex or Thanos) without having to run an instance of the -Prometheus server. The image below shows the two Prometheus exporters in the OpenTelemetry Python SDK. - - -Pipeline 1 illustrates the setup required for a Prometheus "pull" exporter. - - -Pipeline 2 illustrates the setup required for the Prometheus Remote Write exporter. - -![Prometheus SDK pipelines](https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png) - - -The Prometheus Remote Write Exporter is a "push" based exporter and only works with the OpenTelemetry [push controller](https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py). -The controller periodically collects data and passes it to the exporter. This -exporter then converts the data into [`timeseries`](https://prometheus.io/docs/concepts/data_model/) and sends it to the Remote Write integrated backend through HTTP -POST requests. The metrics collection datapath is shown below: - -![controller_datapath_final](https://user-images.githubusercontent.com/20804975/100486582-79d1f380-30d2-11eb-8d17-d3e58e5c34e9.png) - -See the `example` folder for a demo usage of this exporter - -# Table of Contents - * [Summary](#opentelemetry-python-sdk-prometheus-remote-write-exporter) - * [Table of Contents](#table-of-contents) - * [Installation](#installation) - * [Quickstart](#quickstart) - * [Configuring the Exporter](#configuring-the-exporter) - * [Securing the Exporter](#securing-the-exporter) - * [Authentication](#authentication) - * [TLS](#tls) - * [Supported Aggregators](#supported-aggregators) - * [Error Handling](#error-handling) - * [Retry Logic](#retry-logic) - * [Contributing](#contributing) - * [Design Doc](#design-doc) - -## Installation - -* To install from the latest PyPi release, - run `pip install opentelemetry-exporter-prometheus-remote-write` -* To install from the local repository, run - `pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/` in - the project root - -## Quickstart - -```python -from opentelemetry import metrics -from opentelemetry.sdk.metrics import MeterProvider -from opentelemetry.exporter.prometheus_remote_write import ( - PrometheusRemoteWriteMetricsExporter -) - -# Sets the global MeterProvider instance -metrics.set_meter_provider(MeterProvider()) - -# The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here. -meter = metrics.get_meter(__name__) - -exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed - -metrics.get_meter_provider().start_pipeline(meter, exporter, 5) -``` - -## Configuring the Exporter - -The exporter can be configured through parameters passed to the constructor. -Here are all the options: - -* `endpoint`: url where data will be sent **(Required)** -* `basic_auth`: username and password for authentication **(Optional)** -* `headers`: additional headers for remote write request as determined by the remote write backend's API **(Optional)** -* `timeout`: timeout for requests to the remote write endpoint in seconds **(Optional)** -* `proxies`: dict mapping request proxy protocols to proxy urls **(Optional)** -* `tls_config`: configuration for remote write TLS settings **(Optional)** - -Example with all the configuration options: - -```python -exporter = PrometheusRemoteWriteMetricsExporter( - endpoint="http://localhost:9009/api/prom/push", - timeout=30, - basic_auth={ - "username": "user", - "password": "pass123", - }, - headers={ - "X-Scope-Org-ID": "5", - "Authorization": "Bearer mytoken123", - }, - proxies={ - "http": "http://10.10.1.10:3000", - "https": "http://10.10.1.10:1080", - }, - tls_config={ - "cert_file": "path/to/file", - "key_file": "path/to/file", - "ca_file": "path_to_file", - "insecure_skip_verify": true, # for developing purposes - } -) - -``` -## Securing the Exporter - -### Authentication - -The exporter provides two forms of authentication which are shown below. Users -can add their own custom authentication by setting the appropriate values in the `headers` dictionary - -1. Basic Authentication -Basic authentication sets a HTTP Authorization header containing a base64 encoded username/password pair. See [RFC 7617](https://tools.ietf.org/html/rfc7617) for more information. This - -```python -exporter = PrometheusRemoteWriteMetricsExporter( - basic_auth={"username": "base64user", "password": "base64pass"} -) -``` -2. Bearer Token Authentication -This custom configuration can be achieved by passing in a custom `header` to -the constructor. See [RFC 6750](https://tools.ietf.org/html/rfc6750) for more information. - - -```python -header = { - "Authorization": "Bearer mytoken123" -} -``` - -### TLS -Users can add TLS to the exporter's HTTP Client by providing certificate and key files in the `tls_config` parameter. - -## Supported Aggregators - -* Sum -* MinMaxSumCount -* Histogram -* LastValue -* ValueObserver - -## Error Handling -In general, errors are raised by the calling function. The exception is for -failed requests where any error status code is logged as a -warning instead. - -This is because the exporter does not implement any retry logic -as it sends cumulative metrics data. This means that data will be preserved even if some exports fail. - -For example, consider a situation where a user increments a Counter instrument 5 times and an export happens between each increment. If the exports happen like so: -``` -SUCCESS FAIL FAIL SUCCESS SUCCESS -1 2 3 4 5 -``` -Then the recieved data will be: -``` -1 4 5 -``` -The end result is the same since the aggregations are cumulative -## Contributing - -This exporter's datapath is as follows: - -![Exporter datapath](https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png) -*Entites with `*` after their name are not actual classes but rather logical -groupings of functions within the exporter.* - -If you would like to learn more about the exporter's structure and design decisions please view the design document below - -### Design Doc - -[Design Document](TODO: add link) - -This document is stored elsewhere as it contains large images which will -significantly increase the size of this repo. diff --git a/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst b/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst index f204fdb32d..1975ad47d3 100644 --- a/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst +++ b/exporter/opentelemetry-exporter-prometheus-remote-write/README.rst @@ -1,27 +1,236 @@ -OpenTelemetry Prometheus Remote Write Exporter -============================================== +OpenTelemetry Python SDK Prometheus Remote Write Exporter +========================================================= -This library allows exporting metric data to `Prometheus Remote Write Integrated Backends -`_. Latest `types.proto -` and `remote.proto -` Protocol Buffers -used to create WriteRequest objects were taken from Prometheus repository. Development is -currently in progress. +This package contains an exporter to send `OTLP`_ metrics from the +Python SDK directly to a Prometheus Remote Write integrated backend +(such as Cortex or Thanos) without having to run an instance of the +Prometheus server. The image below shows the two Prometheus exporters in +the OpenTelemetry Python SDK. + +Pipeline 1 illustrates the setup required for a Prometheus "pull" +exporter. + +Pipeline 2 illustrates the setup required for the Prometheus Remote +Write exporter. + +|Prometheus SDK pipelines| + +The Prometheus Remote Write Exporter is a "push" based exporter and only +works with the OpenTelemetry `push controller`_. The controller +periodically collects data and passes it to the exporter. This exporter +then converts the data into `timeseries`_ and sends it to the Remote +Write integrated backend through HTTP POST requests. The metrics +collection datapath is shown below: + +|controller_datapath_final| + +See the ``example`` folder for a demo usage of this exporter + +Table of Contents +================= + +- `Summary`_ +- `Table of Contents`_ + + - `Installation`_ + - `Quickstart`_ + - `Configuring the Exporter`_ + - `Securing the Exporter`_ + + - `Authentication`_ + - `TLS`_ + + - `Supported Aggregators`_ + - `Error Handling`_ + - `Retry Logic`_ + - `Contributing`_ + + - `Design Doc`_ Installation ------------ +- To install from the latest PyPi release, run + ``pip install opentelemetry-exporter-prometheus-remote-write`` +- To install from the local repository, run + ``pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/`` + in the project root + +Quickstart +---------- + +.. code:: python + + from opentelemetry import metrics + from opentelemetry.sdk.metrics import MeterProvider + from opentelemetry.exporter.prometheus_remote_write import ( + PrometheusRemoteWriteMetricsExporter + ) + + # Sets the global MeterProvider instance + metrics.set_meter_provider(MeterProvider()) + + # The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here. + meter = metrics.get_meter(__name__) + + exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed + + metrics.get_meter_provider().start_pipeline(meter, exporter, 5) + +Configuring the Exporter +------------------------ + +The exporter can be configured through parameters passed to the +constructor. Here are all the options: + +- ``endpoint``: url where data will be sent **(Required)** +- ``basic_auth``: username and password for authentication + **(Optional)** +- ``headers``: additional headers for remote write request as + determined by the remote write backend's API **(Optional)** +- ``timeout``: timeout for requests to the remote write endpoint in + seconds **(Optional)** +- ``proxies``: dict mapping request proxy protocols to proxy urls + **(Optional)** +- ``tls_config``: configuration for remote write TLS settings + **(Optional)** + +Example with all the configuration options: + +.. code:: python + + exporter = PrometheusRemoteWriteMetricsExporter( + endpoint="http://localhost:9009/api/prom/push", + timeout=30, + basic_auth={ + "username": "user", + "password": "pass123", + }, + headers={ + "X-Scope-Org-ID": "5", + "Authorization": "Bearer mytoken123", + }, + proxies={ + "http": "http://10.10.1.10:3000", + "https": "http://10.10.1.10:1080", + }, + tls_config={ + "cert_file": "path/to/file", + "key_file": "path/to/file", + "ca_file": "path_to_file", + "insecure_skip_verify": true, # for developing purposes + } + ) + +Securing the Exporter +--------------------- + +Authentication +~~~~~~~~~~~~~~ + +The exporter provides two forms of authentication which are shown below. +Users can add their own custom authentication by setting the appropriate +values in the ``headers`` dictionary + +1. Basic Authentication Basic authentication sets a HTTP Authorization + header containing a base64 encoded username/password pair. See `RFC + 7617`_ for more information. This + +.. code:: python + + exporter = PrometheusRemoteWriteMetricsExporter( + basic_auth={"username": "base64user", "password": "base64pass"} + ) + +2. Bearer Token Authentication This custom configuration can be achieved + by passing in a custom ``header`` to the constructor. See `RFC 6750`_ + for more information. + +.. code:: python + + header = { + "Authorization": "Bearer mytoken123" + } + +TLS +~~~ + +Users can add TLS to the exporter's HTTP Client by providing certificate +and key files in the ``tls_config`` parameter. + +Supported Aggregators +--------------------- + +- Sum +- MinMaxSumCount +- Histogram +- LastValue +- ValueObserver + +Error Handling +-------------- + +In general, errors are raised by the calling function. The exception is +for failed requests where any error status code is logged as a warning +instead. + +This is because the exporter does not implement any retry logic as it +sends cumulative metrics data. This means that data will be preserved +even if some exports fail. + +For example, consider a situation where a user increments a Counter +instrument 5 times and an export happens between each increment. If the +exports happen like so: :: - pip install opentelemetry-exporter-prometheus-remote-write + SUCCESS FAIL FAIL SUCCESS SUCCESS + 1 2 3 4 5 +Then the recieved data will be: -.. _Prometheus: https://prometheus.io/ -.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/ +:: + 1 4 5 -References ----------- +The end result is the same since the aggregations are cumulative +Contributing +------------ + +This exporter's datapath is as follows: + +|Exporter datapath| *Entites with ``*`` after their name are not actual +classes but rather logical groupings of functions within the exporter.* + +If you would like to learn more about the exporter's structure and +design decisions please view the design document below + +Design Doc +~~~~~~~~~~ + +`Design Document`_ + +This document is stored elsewhere as it contains large images which will +significantly increase the size of this repo. -* `Prometheus `_ -* `OpenTelemetry Project `_ +.. _Design Document: https://github.com/open-o11y/docs/tree/master/python-prometheus-remote-write +.. |Exporter datapath| image:: https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png +.. _OTLP: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md +.. _push controller: https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py +.. _`timeseries`: https://prometheus.io/docs/concepts/data_model/ +.. _Summary: #opentelemetry-python-sdk-prometheus-remote-write-exporter +.. _Table of Contents: #table-of-contents +.. _Installation: #installation +.. _Quickstart: #quickstart +.. _Configuring the Exporter: #configuring-the-exporter +.. _Securing the Exporter: #securing-the-exporter +.. _Authentication: #authentication +.. _TLS: #tls +.. _Supported Aggregators: #supported-aggregators +.. _Error Handling: #error-handling +.. _Retry Logic: #retry-logic +.. _Contributing: #contributing +.. _Design Doc: #design-doc +.. |Prometheus SDK pipelines| image:: https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png +.. |controller_datapath_final| image:: https://user-images.githubusercontent.com/20804975/100486582-79d1f380-30d2-11eb-8d17-d3e58e5c34e9.png +.. _RFC 7617: https://tools.ietf.org/html/rfc7617 +.. _RFC 6750: https://tools.ietf.org/html/rfc6750