Make the plugin compatible with Influxdb 0.9

[contentfree]

I've got a working version in my fork of the repo (which also includes some other fixes and enhancements like the ability to use the event data directly instead of requiring the explicit setting of data_points).

[brupm]

👍

[contentfree]

There's a philosophical change in Influxdb 0.9. The Influxdb team asserts that metadata about the values in the series (things like host, username, IP, etc) should go into Influxdb as tags instead of fields. Values should only be a field if it changes over time.

Logstash also has tags and fields — and thought of in a similar way — however, we'll probably need to decide the method of mapping Logstash fields to Influxdb fields & tags.

I don't think we can just say "just use add_tag and pass through those tags" because things like the grok filter match values into fields and not tags.

Should we use a config setting to provide "value_fields" (an array of fields names that will be used as the field values sent to Influxdb) and everything else would go to Influxdb as tags.

Thoughts @ph, @jsvd?

[GaryRogers]

Yeah, trying to bring up a new InfluxDB instance, fed by Logstash 1.5 and I ran right into this issue.

[gdlx]

👍 but why don't you provide a PR if you did it on your fork ?

[brupm]

@contentfree I like that idea and separation strategy, but it may make sense to have 2 arrays one for value_fields and one for tag_fields or something like that so we can control what data is inserted into the series as values and which data is inserted as tags.

[gdlx]

@contentfree I've tried your fork, which works fine except when it inserts multiple points at the same time (same as this issue: influxdata/influxdb#2984).
Would you open issues on your repo to help testing and reporting problems ?
Anyway, I'll take a look to the code to eventually submit a PR.

[contentfree]

@gauthier-delacroix I've just done so.

I was unable to get logstash -> influxdb working reliably enough to move to production with it, so my immediate needs have waned and I'm fine adding folks as collaborators.

[skynet]

👍

[tgermain]

Few note on making this plugin compatible with influxdb 0.9 : the json protocol will be deprecated at some point in favor of the line protocole, which achieve better performance anyway.

It might be good to adapt how data are formatted before being sent.

see influxdata/influxdb#2696

[rainerhahnekamp]

I was able to successfully setup the fork of @contentfree with approximately 200 data points per second. Currently it runs in production for one week without any issues. Neither performance issues, reboots or crashes of some kind - it just runs fine.

Since that fork is already available since May and InfluxDB in version 9 even longer, I really would like to get an answer or statement from @ph or @jsvd on the status quo. The official plugin is more or less not the very useful.

If some work needs to be done, I can spare time to work on issues, etc.

[rvrignaud]

This is also problematic for me. I'd rather avoid running a fork on production. So a big +1 if this could be merged.

[bcubk]

+1 for merge

[contentfree]

Happy to add folks as collaborators. I stopped pursuing influxdb (& logstash) a while ago, though I might come back around to it again (since graphite is being such a PITA to get installed…)

[jordansissel]

+1 for merge

I'm confused - this is an issue, not a pull request. What would be merged?

[contentfree]

True. I'm happy to create a pull request from my fork – but yes, this is the issue (that references the fork…)

[gnewman7]

Have also used the @contentfree fork of the plugin to successfully ingest data using logstash to influxdb 0.9.3 solution for some months now.

It would be great if Dave Myron could create a pull request.

@jordansissel it would look good on logstash's part to have a updated version of this output plugin that supports influxdb 0.9.x.

[suyograo]

@contentfree please create a PR and we can merge it in. Thanks!

[contentfree]

Created PR #29

[kestel]

Meanwhile InfluxDB 0.10.0 already released...

[suyograo]

Fixed in #30