feat(new sink): Initial `influxdb_metrics` sink implementation #1759

bednar · 2020-02-10T07:45:28Z

Closes #562

There is also a work to do, but I hope so that is a good start point to accept PR

TODO:

documentation
integration tests

bednar · 2020-02-11T10:23:07Z

I added a documentation so the pull request is ready to review.

binarylogic · 2020-02-11T16:33:33Z

Thanks! We'll have @a-rodin review this week.

.meta/sinks/influxdb_metrics.toml

.meta/links.toml

src/sinks/influxdb_metrics.rs

docker-compose.yml

src/sinks/influxdb_metrics.rs

.meta/sinks/influxdb_metrics.toml

src/sinks/influxdb_metrics.rs

binarylogic · 2020-02-11T21:42:31Z

@bednar I know Influx has support for ingesting logs, right? Would it make sense to follow this up with an influx_logs sink at some point?

bednar · 2020-02-12T13:50:41Z

@binarylogic I think a preferred way is transform logs to metrics and push these metrics into InfluxDB.

Hoverbear · 2020-02-13T21:43:10Z

src/sinks/influxdb_metrics.rs

+use tower::Service;
+
+pub enum Field {
+    /// string


I feel like these are very unnecessary doc comments.

Comments removed

Hoverbear · 2020-02-13T21:44:55Z

src/sinks/influxdb_metrics.rs

+    /// float
+    Float(f64),
+    /// unsigned integer
+    UnsignedInt(u32),


It seems like these should be 64 bit? https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/#uinteger

It's u32 because is used in MetricValue - https://github.com/timberio/vector/blob/61f9f3356bfdc266d6841d42344dd5dee194b766/src/event/metric.rs#L24

Hoverbear · 2020-02-13T21:45:32Z

src/sinks/influxdb_metrics.rs

+use std::collections::{BTreeMap, HashMap};
+use tower::Service;
+
+pub enum Field {


Seems like this should include the other types listed on https://v2.docs.influxdata.com/v2.0/reference/syntax/line-protocol/#data-types-and-format ?

I think that other types are unnecessary, but Field is useful for extending in future.

Hoverbear · 2020-02-13T22:42:37Z

src/sinks/influxdb_metrics.rs

+        let mut res = client
+            .post("http://localhost:9999/api/v2/query?org=my-org")
+            .json(&body)
+            .header("accept", "application/json")


Suggested change

.header("accept", "application/json")

.header("Accept", "application/json")

src/sinks/influxdb_metrics.rs

ghost · 2020-02-14T13:24:35Z

@bednar Are you planning to add #1759 (comment) as part of this PR? If not, it is fine, we can merge it as it is and then add support for v1.x later if necessary.

bednar · 2020-02-14T13:28:49Z

@a-rodin I am currently works on it. I hope so that today i fix code and at Monday documentation for that.

syepes · 2020-02-14T20:27:22Z

Can wait to see this merged, thanks for the work

ghost

The code looks good to me now. After the docs are updated, this can be merged.

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

…ring in URL, rename host to endpoint, unsigned int serialized as '*u', InfluxDB start with disabled reporting Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

bednar · 2020-02-18T11:15:24Z

Conflict with master is resolved

binarylogic · 2020-02-18T16:22:22Z

@bednar great! Could you give me permission to push to your branch? I'd like to make some documentation changes and then we can merge.

bednar · 2020-02-18T21:02:01Z

@binarylogic Awesome information! Permission granted ;)

binarylogic · 2020-02-19T03:29:06Z

Thanks @bednar! Appreciate the attention to detail and clean commits. Let us know if you'd be interested in working on anything else!

bednar · 2020-02-19T07:23:18Z

Thanks guys for cooperation! @binarylogic @a-rodin

binarylogic · 2020-04-06T13:10:31Z

@bednar I'm curious if you'd be interested in working on other Vector components? Feel free to email me if you are -- ben@timber.io.

bednar requested a review from lukesteensen as a code owner February 10, 2020 07:45

bednar changed the title ~~feat(new sink): Add new InfluxDB sink~~ feat(new sink): Add new InfluxDB 2 sink Feb 10, 2020

binarylogic assigned lukesteensen and ghost and unassigned lukesteensen Feb 10, 2020

binarylogic requested review from a user and removed request for lukesteensen February 10, 2020 16:41

bednar force-pushed the influxdb_metrics branch from ff7f71c to 73a21a5 Compare February 11, 2020 07:44

bednar requested a review from binarylogic as a code owner February 11, 2020 09:51

ghost suggested changes Feb 11, 2020

View reviewed changes

binarylogic changed the title ~~feat(new sink): Add new InfluxDB 2 sink~~ feat(new sink): Initial influx_metrics sink implementation Feb 11, 2020

binarylogic changed the title ~~feat(new sink): Initial influx_metrics sink implementation~~ feat(new sink): Initial influxdb_metrics sink implementation Feb 11, 2020

bednar force-pushed the influxdb_metrics branch from 46c2d53 to 7ef890b Compare February 12, 2020 11:24

binarylogic requested a review from a user February 13, 2020 18:27

Hoverbear reviewed Feb 13, 2020

View reviewed changes

src/sinks/influxdb_metrics.rs Show resolved Hide resolved

ghost approved these changes Feb 16, 2020

View reviewed changes

bednar requested review from bruceg, Jeffail and LucioFranco as code owners February 17, 2020 09:21

bednar added 22 commits February 18, 2020 09:41

feat(influxdb sink): encode timestamp

fa7c0bc

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): encode namespace

9416f4e

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): encode_tags, encode_key

042e021

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): encode_fields

790e853

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): support for Gauge

1fd52a3

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): support for Set

f7f01e4

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): support for AggregatedHistogram

942d835

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): support for AggregatedSummary

774eac0

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): fixed mapping int

04803ce

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): support for Distribution

1f96de0

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): implemented InfluxDBSvc

53650fa

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): added healthchecks integration test

bd75a51

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): added put_data integration test

65f42b5

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): added InfluxDB to test-runtime-deps

35c1fe6

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): added documentation

cb9c187

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): added links to .meta/links.toml, encode query st…

6ee3b85

…ring in URL, rename host to endpoint, unsigned int serialized as '*u', InfluxDB start with disabled reporting Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): optimize encoding to line protocol

8e2d9b9

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): optimize encode_events to line protocol

2cd6e06

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): add support for v1

8fbd69f

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): added documentation about support v1

e2a4669

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): line protocol marked as a code

d3c7ea8

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

feat(influxdb sink): rebased over master

4604033

Signed-off-by: Jakub Bednar <jakub.bednar@gmail.com>

bednar force-pushed the influxdb_metrics branch from 1e8d1b1 to 4604033 Compare February 18, 2020 08:46

binarylogic merged commit b89aded into vectordotdev:master Feb 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(new sink): Initial `influxdb_metrics` sink implementation #1759

feat(new sink): Initial `influxdb_metrics` sink implementation #1759

bednar commented Feb 10, 2020 •

edited

Loading

bednar commented Feb 11, 2020

binarylogic commented Feb 11, 2020

binarylogic commented Feb 11, 2020

bednar commented Feb 12, 2020

Hoverbear Feb 13, 2020

bednar Feb 14, 2020

Hoverbear Feb 13, 2020

bednar Feb 14, 2020

Hoverbear Feb 13, 2020

bednar Feb 14, 2020

Hoverbear Feb 13, 2020

bednar Feb 14, 2020

ghost commented Feb 14, 2020

bednar commented Feb 14, 2020

syepes commented Feb 14, 2020

ghost left a comment

bednar commented Feb 18, 2020

binarylogic commented Feb 18, 2020

bednar commented Feb 18, 2020

binarylogic commented Feb 19, 2020

bednar commented Feb 19, 2020

binarylogic commented Apr 6, 2020

	.header("accept", "application/json")
	.header("Accept", "application/json")

feat(new sink): Initial influxdb_metrics sink implementation #1759

feat(new sink): Initial influxdb_metrics sink implementation #1759

Conversation

bednar commented Feb 10, 2020 • edited Loading

bednar commented Feb 11, 2020

binarylogic commented Feb 11, 2020

binarylogic commented Feb 11, 2020

bednar commented Feb 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Feb 14, 2020

bednar commented Feb 14, 2020

syepes commented Feb 14, 2020

ghost left a comment

Choose a reason for hiding this comment

bednar commented Feb 18, 2020

binarylogic commented Feb 18, 2020

bednar commented Feb 18, 2020

binarylogic commented Feb 19, 2020

bednar commented Feb 19, 2020

binarylogic commented Apr 6, 2020

feat(new sink): Initial `influxdb_metrics` sink implementation #1759

feat(new sink): Initial `influxdb_metrics` sink implementation #1759

bednar commented Feb 10, 2020 •

edited

Loading