Docker configuration

[philippkahr]

Hi,

I am having a hard time working out how to use it in conjunction with docker. I have a Java application that logs everything in JSON to a file inside the docker container. I have created a ln -sf /dev/stdout /opt/application.json symlink, so everything will be output to stdout. Of course, looking into the -json.log file created by Docker in the /var/lib/docker/container/containerid/*-json.log it looks something like this:

{"log":"{\"@timestamp\":\"2019-11-03T12:33:45.658Z\", \"log.level\": \"INFO\", \"message\":\"Login pkahr (philipp.kahr)\", \"service.name\":\"myservice\",\"process.thread.name\":\"http-bio-8080-exec-8\",\"log.logger\":\"com.myservice.server.core.utils.user.ThisDirectoryServiceImpl\",\"log.origin\":{\"file.name\":\"ThisDirectoryServiceImpl.java\",\"function\":\"loadUserByUsername\",\"file.line\":622}}\n","stream":"stdout","time":"2019-11-03T12:33:45.658546603Z"}

My filebeat config looks like this:

filebeat.inputs:
- type: container
  paths: 
    - '/var/lib/docker/containers/*/*.log'
  json.keys_under_root: true
  json.overwrite_keys: true
  json.message_key: log
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after

I am using the message decoder like suggested in the README

  - decode_json_fields:
      fields: message
      target: ""
      overwrite_keys: true
  # flattens the array to a single string
  - script:
      when:
        has_fields: ['error.stack_trace']
      lang: javascript
      id: my_filter
      source: >
        function process(event) {
            event.Put("error.stack_trace", event.Get("error.stack_trace").join("\n"));
        }

however, I can see a couple of errors in journalctl -u filebeat -f

filebeat[29865]: 2019-11-03T13:56:09.252+0100        ERROR        readjson/json.go:52        Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
filebeat[29865]: 2019-11-03T13:56:21.269+0100        ERROR        readjson/json.go:52        Error decoding JSON: invalid character '-' in numeric literal
filebeat[29865]: 2019-11-03T13:56:21.269+0100        ERROR        readjson/json.go:52        Error decoding JSON: invalid character 'E' looking for beginning of value
filebeat[29865]: 2019-11-03T13:56:21.268+0100        ERROR        readjson/json.go:52        Error decoding JSON: unexpected EOF

filebeat[29865]: 2019-11-03T13:56:17.267+0100        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x120f69c0, ext:63708382571, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"f3bb8324-dda5-4dbe-9f1a-6fc86820c37d", "hostname":"CentOS-76-64-minimal", "id":"d8cb4370-3918-40a8-88c5-80a992868234", "type":"filebeat", "version":"7.4.2"}, "ecs":common.MapStr{"version":"1.1.0"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"CentOS-76-64-minimal", "id":"7d91e31c18374a518bb44ad58646dfeb", "name":"CentOS-76-64-minimal", "os":common.MapStr{"codename":"Core", "family":"redhat", "kernel":"3.10.0-1062.4.1.el7.x86_64", "name":"CentOS Linux", "platform":"centos", "version":"7 (Core)"}}, "input":common.MapStr{"type":"container"}, "log":"", "log.level":"INFO", "log.logger":"com.myservice.server.core.utils.user.ServiceDirectoryServiceImpl", "log.origin":common.MapStr{"file.line":622, "file.name":"ServiceDirectoryServiceImpl.java", "function":"loadUserByUsername"}, "message":"Login pkahr (philipp.kahr)", "process.thread.name":"http-bio-8080-exec-2", "service.name":"myservice", "stream":"stdout", "tags":[]string{"testing"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00022cc30), Source:"/var/lib/docker/containers/b35f79cc3aa44c98709c3a05dfee5fcc13c92d18b0cff35d821af9820e52c398/b35f79cc3aa44c98709c3a05dfee5fcc13c92d18b0cff35d821af9820e52c398-json.log", Offset:656232, Timestamp:time.Time{wall:0xbf67d03f87d679e9, ext:12500845494, loc:(*time.Location)(0x4de6580)}, TTL:-1, Type:"container", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x18802e6, Device:0x902}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [log] tried to parse field [log] as object, but found a concrete value"}

Any idea on how to use the ECS logging in combination with docker? I would love to have it, so I can jump from the APM to the container logs.

[clarkritchie]

@philippkahr Did you ever make progress on this? I think this is my issue at the moment as well.

[philippkahr]

@clarkritchie sadly, no I never did any good progress on this. I think I solved it by doing some Docker filesystem mapping and picking it up with filebeat.
I talked to @xeraa about that a while ago, that's what he has told me.

You can mount files around for Docker logging, but it's probably not the most elegant solution. System.Out is nicer, since you can then just pick up the default docker logs and also enrich them with metadata. https://github.com/xeraa/java-logging shows the different approaches in detail more detail and you can use those. And you will want to pick up the ECS output with https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html. But I'll try to get a complete example out in the next couple of days anyway.

[xeraa]

@philippkahr sorry, this got burried under too much other stuff. I just worked with something similar today and also ran into Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}.

You are collecting the log files from all the containers ('/var/lib/docker/containers/*/*.log') and try to convert them into JSON (json.message_key: log). Are you running Filebeat in a container as well? Because Filebeat itself doesn't log JSON by default, so trying to convert those logs into JSON will fail. The two workaround could be:

• Write the Filebeat logs to a file (logging.to_files: true), so Filebeat won't try to collect its own (non-JSON) logs. Also you would avoid a logging loop when an error in Filebeat triggers a log statement that it then tries to collect itself,...
• Annotate on the logged container how it's data should be processed with hints on autodiscovery. The more containers you have running, the more different log formats you will have. Building rules around them will get complicated and unpredictable, so annotating the container itself with its rules should be more straight forward.

[SylvainJuge]

Hi, this issue hasn't had any update in a long time !

Regarding integration of ECS logs and APM integration we now have a few "ready to use" examples in our contrib repo: https://github.com/elastic/apm-contrib/tree/main/apm-agent-java/log-ingest.

In particular here, the example 02 uses ECS logging library (the one in this repo) with a filebeat integration.
Those examples remain file-based and do not use auto-discovery, but it should provide you a good start.

[C0rn3j]

You are running into this issue elastic/beats#20053 (comment)

You are JSON decoding your log three times.

First time by adding the container input.

Second time by adding the json keys to the container input, against log, which does not exist anymore after the first decode.

Third time in decode_json_fields processor.

The documentation is confusing in this regard to be fair, I spent quite some time on it.

If anyone is running into this, start simple by removing the processors and parsers and only keep the input type and path, with output to console, look at it, make sure you understand the output and move on.

output.console:
  pretty: true

This can be closed I suppose, just like the issue I linked in the beginning, but the docs really need some quality examples added.