Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more outputs than Elasticsearch and Logstash #943

Closed
3 tasks done
monicasarbu opened this issue Feb 8, 2016 · 5 comments
Closed
3 tasks done

Add more outputs than Elasticsearch and Logstash #943

monicasarbu opened this issue Feb 8, 2016 · 5 comments

Comments

@monicasarbu
Copy link
Contributor

monicasarbu commented Feb 8, 2016

Currently you can configure in the Beats the following as outputs (in chronological order):

  • Elasticsearch
  • Redis
  • Logstash

Once we added support for Logstash, we deprecated Redis as you can use the redis output plugin to export data to Redis.
For someone that wants to integrate the Elastic Stack into its own system and make use of a queuing system before inserting data to Elasticsearch, the deployment looks like this:

Beats -> Logstash -> Redis/Kafka -> Logstash -> Elasticsearch

We received many requests from the community to add support for more outputs, besides Elasticsearch and Logstash to avoid having an additional Logstash instance to translate the data to the desired queuing system. Here are the links to the GitHub issues:

After internal discussions, we decided to follow the community feedback, so we will un-deprecate the Redis output and add support for Kafka output.

With these improvements, the deployments are simplified:

Beats -> Redis/Kafka -> Logstash -> Elasticsearch

TODOs:

We will continue to be very selective in terms of what outputs we support. For the moment, we are only considering Redis and Kafka based on their popularity.

@adiworkoholic
Copy link

So... is the support for a rabbitMQ output off the table :(

@ruflin
Copy link
Contributor

ruflin commented Mar 1, 2016

@adiworkoholic Our current focus/priority is on implementing Redis and Kafka. Best is to subscribe to the issue https://github.com/elastic/libbeat/issues/313 as we will post an update there in case we add further outputs.

@brandbintley
Copy link

Kafka support is awesome. How do we see handling security? If I got it right, libbeat supports TLS while Kafka supports SSL and SASL.

@froztbyte
Copy link

I'm also in favour of a RabbitMQ output.

My previous model for this was beaver[0](with a sincedb configured) connecting to rabbitmq, logstash reading off of logstash then outputting to elasticsearch and others. The benefits of this design is that it gives quite a strong ability to deal with various failures or operational necessities. On a future model of Filebeats and friends, I would ideally like to be able to reuse this same design, or a design with its strengths

  • [0] The one downside with beaver was that it was synchronous by default. That wasn't great.

@monicasarbu
Copy link
Contributor Author

Closing the issue as all the tasks were finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants