Auto-discover docker containers, allow config, auto update when containers are created/destroyed

[taxilian]

Hi,

You probably won't actually consider this ready to merge, but I thought I'd pass it along in case you're interested; it is 100% working and we're using it in production.

I use the following script to start it:

#!/usr/bin/env bash
DOCKER_IMG=logio
CNAME=logio_harvester
docker pull $DOCKER_IMG
docker stop $CNAME
docker rm $CNAME
docker run -d \
    -v /var/run/docker.sock:/tmp/docker.sock:ro \
    -v /var/lib/docker/containers:/var/lib/docker/containers \
    -e LOGIO_HARVESTER_NODENAME=`hostname -f` \
    -e LOGIO_HARVESTER_MASTER_HOST=172.16.0.50 \
    --user root \
    --restart=always \
    --name $CNAME \
    $DOCKER_IMG harvester

When it starts it queries docker.sock to get a list of all running containers and their log file; it allows the container to override the stream name by setting the "LOGIO_STREAM" variable on the container if desired. There are some other options in there as well.

If a container is started or stopped it will automatically update the config file and restart the harvester.

[blacklabelops]

I really like your implementation, as it represents what the logio image should be from the beginning. It's awesome!

As a matter of fact it would be hard for me to support this image with this completely rewrite including multiple technologies I do not master and therefore ask you to either become a main contributor to this repository.

[taxilian]

Sure, I can help keep things running. You'll notice I did have to change the base image somewhat in order to install everything I needed; you might know a better way to do it, just need node.js 6.x or later for my code to work. Anyway, just let me know how I can help and if you have any requests for changes. it's mostly the docs that I'm bad at writing =]

[blacklabelops]

The old behavior has been removed here. Is this image fully downwards compatible (same behavior and envs) or do we need some kind of trigger for the old static one and the new dynamic one?

[taxilian]

It is not totally backwards compatible, no; mostly this is because I don't think (and feel free to correct me if you feel I've overlooked something, it's quite possible) that all of the old variables are still useful or needed.

Harvester container supported ENV vars are:

• LOGIO_REQUIRE_EXPLICIT - set to 1 to only include containers which have an env variable LOGIO_INCLUDE=1 (not tested)
• LOGIO_HARVESTER_NODENAME - sets the node name in the harvester config
• LOGIO_HARVESTER_PREFIX - if set, adds this prefix to all stream names
• LOGIO_HARVESTER_MASTER_HOST - the address of the logio server
• LOGIO_HARVESTER_MASTER_PORT - the port of the logio server
• DOCKER_SOCKET - the path to the docker.sock file, defaults to /tmp/docker.sock
• LOGIO_HARVESTER_CONFIG - the path for the harvester conf file, defaults to /root/.log.io/harvester.conf

Supported Container ENV -- these if set on an arbitrary docker container modify the way that the docker container is treated by the harvester:

• LOGIO_INCLUDE - When LOGIO_REQUIRE_EXPLICIT=1 on the harvester container, LOGIO_INCLUDE=1 must be set on each container for which logs should be harvested (not tested yet)
• LOGIO_EXCLUDE - if LOGIO_EXCLUDE=1 logs will not be harvested for this container
• LOGIO_STREAM - if set, the containers logs will show up under this name instead of the container name; if multiple containers have the same LOGIO_STREAM their logs files will all be added to the same stream

Anything else you feel needs to be brought forward? I admit I only looked at the original for a bit before I decided I needed something which required less configuration. (I'm lazy that way)

[blacklabelops]

I see, so it's fixed for it's purpose as a docker logs client.

Then the old behavior must be conserved.

Tha old harvester functioned also outside the docker context. The main task of the old harvester had been to attach to specified logfiles with specific fileendings in specific folders.

E.g. to crawl logfiles by webservers that are not forwarded to stdout.