fluent-plugin-ecs-metadata-filter

Filter plugin to add AWS ECS metadata to fluentd events. Based on fabric8io/fluent-plugin-kubernetes_metadata_filter.

Requirements

fluent-plugin-ecs-metadata-filter	fluentd	ruby
>= 0.2.0	>= v0.14.0	>= 2.1
< 0.2.0	>= v0.12.0	>= 1.9

Installation

gem install fluent-plugin-ecs-metadata-filter

Configuration

Configuration options for fluent.conf are:

• cache_size - Size of the cache of ECS container metadata which reduces requests to the API server - default: 1000
• cache_ttl - TTL in seconds for each cached element. Set to negative value to disable TTL eviction - default: 3600 (1 hour)
• ecs_agent_uri - Base address used to contact the ecs-agent - default: http://localhost:51678/v1
• fields_key - Key in the final record holding the metadata fields. Set to "" to set fields in the record itself - default: ecs
• fields - Array of metadata fields that should be added to a log record - default: docker_name, family, cluster, name - Available options:
  • cluster
  • container_instance_arn
  • container_instance_version
  • desired_status
  • docker_id
  • docker_name - Name of the docker container
  • family
  • known_status
  • name - Name as specified in the task definition
  • task_arn
  • version
• merge_json_log - Merge in JSON format as top level keys in the log, if the log is JSON - default: true
• tag_regexp - Regular expression used to extract the docker_id from the fluentd tag - default: var\.lib\.docker\.containers\.(?<docker_id>[a-z0-9]{64})\.[a-z0-9]{64}-json.log$ (note the full 64-character id is needed - a short id will not work)

Reading from the docker container

<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S
  tag ecs.*
  format json
  read_from_head true
</source>

<filter ecs.var.lib.docker.containers.*.*-json.log>
  type ecs_metadata
</filter>

<match **>
  type stdout
</match>