Add support for retrying output writes, using independent threads

[sparrc]

Fixes #285

[sparrc]

@daviesalex this change will add support for retrying failed writes.

Initially I've decided to implement it a bit more simply than a circular shared buffer. This will simply spin off a thread for each "batch" of points, retrying flush_retries times if it fails.

[daviesalex]

@sparrc looks great, we will test this out at scale this week and report back.

[daviesalex]

I'm not 100% sure which issue is best to comment on this; apologies if I have picked the wrong one. But, as promised, we tested this. Summary:

• This works as you would expect
• If you are at scale, you need to set the flush_interval to be the same as jitter_interval
• Even with that, you are at risk of microbursts "on the second"

We deleted all our data, and pushed a change to 1,000 hosts with flush_interval at 60s and jitter_interval 30. See below graph for the moment we brought this config up (before the obvious change old config; after change 1,000 of our several thousand hosts moved to the new config)

These 1,000 hosts have pretty accurate timesync (<25us) and are a small number of cut through (=very fast) switches from the InfluxDB test nodes, so this is an extreme example of the problem, but we saw extreme microbursts (causing drops on Intel 10G NIC cards).

We have not yet done network inspection of capture files, but I strongly suspect that we are going to see a <100us microburst once per second. This isnt affecting InfluxDB too much, but is making network hardware have a bad day in the middle.

We have two further suggestions (as well as making it a fixed sized buffer in #285):

1. We will be adjusting our config to make the two intervals the same, but I suggest that when the snap to a second config is in place you add a non-configurable random sleep of between 0 and 1 second (measured in microseconds). This will prevent users who dont realize that they have microbursts from experiencing the worst of the problems.
2. Any failures should backoff exponentially up to some sane maximum. I've not dug into the code to see what the behaviour in HEAD Is, but when we started InfluxDB up after taking it down with just 1k agents, the microburst of them all retrying their writes was >10G (for a fraction of a second) and sufficient to cause a commercially supported loadbalancer running haproxy to segfault and 10G Intel NICs on appliances rated for 14 million packets per second to drop frames. I dont think we got to 14 million packets in a second, but we probably got over a few million in a few us.

We are going to leave this configuration running for the weekend and will do more testing on Monday. Happy to do any specific testing you can suggest.

Thanks!

-Alex

[daviesalex]

So a further graph of real world usage; this shows Rx traffic for 3 nodes (all writes going to one, the yellow)

Box 1 - old client
Box 2 - new client, flush_interval=60, jitter_interval=30 on 1k clients
Box 3 - new client, flush_interval=60, jitter_interval=60 on 1k clients
Box 4 - InfluxDB fell over with shard errors (Separate issue) and started rejecting a small % of writes

The spikiness in box 3/4 we believe is caused by clients bursting at the same time on a specific second. Our metric data is once per 10 seconds on the server, so we need to do more analysis to dig really into this, which will happen on Monday.

[sparrc]

Thank you @daviesalex, this is very good information. My first reaction was that we must be choosing a random number on 1s resolution, but we're actually choosing it on 1ns resolution. Most likely the problem is that each Telegraf binary is using the same seed from the rand package. I should be able to fix this by setting the random seed to the current time in nanoseconds, this should give you the same behavior as you were getting before, where the flush time was dependent on the Telegraf start time.

From what I can tell, adding another random sleep between 0 and 1 seconds will only result in all Telegraf instances sleeping for the same "random" amount, and the microbursts will remain.

I also see what you mean about the backoff on retries. The current implementation has the batches of points retrying independently. This means that if you have an InfluxDB server down for more than 2 flush intervals, then each Telegraf instance will have 3 batches of points backed up, and will be trying to flush those 3 batches on the same interval. Having the persistent buffer will fix this.

[sparrc]

PS @daviesalex can we send you some InfluxDB swag to say "thanks" for this? If you send your t-shirt size and address to cameron@influxdb.com we can get a package out to you :-)