-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add more outputs than Elasticsearch and Logstash #943
Comments
So... is the support for a rabbitMQ output off the table :( |
@adiworkoholic Our current focus/priority is on implementing Redis and Kafka. Best is to subscribe to the issue https://github.com/elastic/libbeat/issues/313 as we will post an update there in case we add further outputs. |
Kafka support is awesome. How do we see handling security? If I got it right, libbeat supports TLS while Kafka supports SSL and SASL. |
I'm also in favour of a RabbitMQ output. My previous model for this was beaver[0](with a sincedb configured) connecting to rabbitmq, logstash reading off of logstash then outputting to elasticsearch and others. The benefits of this design is that it gives quite a strong ability to deal with various failures or operational necessities. On a future model of Filebeats and friends, I would ideally like to be able to reuse this same design, or a design with its strengths
|
Closing the issue as all the tasks were finished. |
Currently you can configure in the Beats the following as outputs (in chronological order):
Once we added support for Logstash, we deprecated Redis as you can use the redis output plugin to export data to Redis.
For someone that wants to integrate the Elastic Stack into its own system and make use of a queuing system before inserting data to Elasticsearch, the deployment looks like this:
Beats -> Logstash -> Redis/Kafka -> Logstash -> Elasticsearch
We received many requests from the community to add support for more outputs, besides Elasticsearch and Logstash to avoid having an additional Logstash instance to translate the data to the desired queuing system. Here are the links to the GitHub issues:
After internal discussions, we decided to follow the community feedback, so we will un-deprecate the Redis output and add support for Kafka output.
With these improvements, the deployments are simplified:
Beats -> Redis/Kafka -> Logstash -> Elasticsearch
TODOs:
We will continue to be very selective in terms of what outputs we support. For the moment, we are only considering Redis and Kafka based on their popularity.
The text was updated successfully, but these errors were encountered: