Skip to content

first release
Compare
Choose a tag to compare
@yazgoo yazgoo released this 14 Nov 09:53
· 394 commits to master since this release
v0.1.3
5238092

This is the first official release of fuse_kafka, a fuse-based logging agent.
It is written for collecting logs from servers running all kind of software,
as a generic way to collect logs without needing to know about each logger.

Home:
https://github.com/yazgoo/fuse_kafka

Here are some functionnalities:

  • sends all writes to given directories to kafka
  • passes through FS syscalls to underlying directory
  • captures the pid, gid, uid, user, group, command line doing the write
  • you can add metadata to identify from where the message comes from (e.g. ip-address, ...)
  • you can configure kafka destination cluster either by giving a broker list or a zookeeper list
  • you can specify a bandwidth quota: fuse_kafka won't send data if a file is written more than a given size per second (useful for preventing floods caused by core files dumped or log rotations in directories watched by fuse_kafka)

It is based on:

  • FUSE (filesystem in userspace), to capture writes done under a given directory
  • kafka (messaging queue), as the event transport system
  • logstash: events are written to kafka in logstash format (except messages and commands which are stored in base64)

Quickstart:
http://playterm.org/r/fusekafka-quickstart-1416935569

Packages are provided for various distros, see installing section in README.md.
FUSE adds an overhead, so it should not be used on filesystems where high throughput is necessary.
Here are benchmarks:
http://htmlpreview.github.io/?https://raw.githubusercontent.com/yazgoo/fuse_kafka/master/benchs/benchmarks.html

Assets 2
Loading