Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine BatchPoints with the same RoutingTag to one message in amqp output #287

Closed
wants to merge 1 commit into from

Conversation

ekini
Copy link
Contributor

@ekini ekini commented Oct 19, 2015

After some testing, I've found that 80 hosts can generate up to 11k messages per second and send it to rabbitmq, which is quite a lot!
So let's do some batching.

Also, according to https://www.rabbitmq.com/blog/2012/04/25/rabbitmq-performance-measurements-part-2/

Quite a lot of the work done by RabbitMQ is per-message, not per-byte-of-message.

Which seems logical.

So this commit uses []byte to accumulate output. It also honours RoutingTag.

Just a few numbers:
80 hosts, recent telegraf, only system metrics.
Before: 11k msg/s (peaked), LA on influxdb host >3 (4 processors)
After: 3k msg/s, LA on influxdb host ~ 0.15

@ekini ekini force-pushed the amqp branch 2 times, most recently from 7ba0608 to 806cf22 Compare October 19, 2015 18:50
@sparrc
Copy link
Contributor

sparrc commented Oct 20, 2015

@ekini I've made some substantial changes to the accumulator and how output sinks are handled. Could you please rebase or re-write your changes off of HEAD?

sorry about that

@ekini
Copy link
Contributor Author

ekini commented Oct 21, 2015

Ok, updated.

@sparrc
Copy link
Contributor

sparrc commented Oct 21, 2015

great, thanks, will get this merged 👍

@sparrc sparrc closed this in 553208a Oct 21, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants