Telegraf should be resilient to Influx dying and the database going away

[jchauncey]

Right now I am running Influx without persistent storage (it runs on kubernetes and is not saving data to the host filesystem). So that means if the pod dies the database goes away.

Currently, Telegraf will reconnect to Influx but fails to send any data because the database has gone away. I patched my local telegraf copy by removing the following conditional - https://github.com/influxdata/telegraf/blob/master/plugins/outputs/influxdb/influxdb.go#L163

This seemed to fix the problem but now its not reusing the client connection. It would be nice if the database check was performed further down the stack so that way if the database goes away telegraf can just recreate it.

[steverweber]

I agree that telegraf should be resilient to fails!

However I feel telegraf should not create the database if it is missing. perhaps could be a config option ,,, but not default behavior.

Perhaps for your setup just put a cron job on your influxdb server to create database.

[sparrc]

I think we could just clear out the i.conns slice if all writes fail before we return from Write() (which will trigger a call to i.Connect on the next write)

But @jchauncey if you don't care that much about your data, you might want to use the UDP service listener on InfluxDB. The UDP listener has lower overhead on both sides, and it will also create your database for you.