filebeat container multiline error Error decoding JSON: invalid character

[xilu0]

• Version: filebeat 7.8
• Operating System: ubuntu
• Discuss Forum URL:
• Steps to Reproduce:

full error:

ERROR [reader_json] readjson/json.go:57 Error decoding JSON: invalid character 'a' looking for beginning of value.

filebeat config:

- type: container
  multiline.pattern: '^2020-07-15'
  multiline.negate: true
  multiline.match: after
  json.message_key: log
 # json.keys_under_root: false
  overwrite_keys: true
  paths:
    - /var/log/err.log

raw log in java from docker log:

{"log":"2020-07-15 03:15:35 [INFO ] [main] org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'shiroProperyConfig' of type [com.wwtx.chinesemedicine.service.config.propery.ShiroProperyConfig$$EnhancerBySpringCGLIB$$b7434c17] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)\n","stream":"stdout","time":"2020-07-15T03:15:35.819881438Z"}
{"log":"Logging initialized using 'class org.apache.ibatis.logging.stdout.StdOutImpl' adapter.\n","stream":"stdout","time":"2020-07-15T03:15:35.968119304Z"}
{"log":"2020-07-15 03:15:36 [INFO ] [main] org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker' of type [org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)\n","stream":"stdout","time":"2020-07-15T03:15:36.428202427Z"}
{"log":"Registered plugin: 'AbstractSqlParserHandler(sqlParserList=null, sqlParserFilter=null)'\n","stream":"stdout","time":"2020-07-15T03:15:36.51131214Z"}
{"log":"Parsed mapper file: 'class path resource [mapper/ArticlesHitsMapper.xml]'\n","stream":"stdout","time":"2020-07-15T03:15:36.877433778Z"}
{"log":"[com.wwtx.chinesemedicine.service.dao.ArticlesDao.selectById] Has been loaded by XML or SqlProvider or Mybatis's Annotation, so ignoring this injection for [class com.baomidou.mybatisplus.core.injector.methods.SelectById]\n","stream":"stdout","time":"2020-07-15T03:15:36.963122274Z"}
{"log":"Parsed mapper file: 'class path resource [mapper/ArticlesMapper.xml]'\n","stream":"stdout","time":"2020-07-15T03:15:36.99978738Z"}

I can't debug it, who can help me?

[humberto-garza]

Hi @andresrc ,

did you ever find a solution to your problem?
I believe I am experiencing something similar:

https://discuss.elastic.co/t/filebeat-on-ubuntu-hosts-getting-sporadic-error-error-decoding-json/258783

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[urso]

How is logging and rotation configured in general? The original report looks like Docker or kubernetes. Some environments use copytruncate, which at times can "trick" the reader to continue reading from the wrong location. In that case an "invalid" line is read, but the input should mostly recover.

Also note, multiline JSON is not supported as well.

Although we see parsing errors, the cases seem to be different, as the log source we read from differs.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humberto-garza]

We are still running into this issue.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[jlind23]

@humberto-garza on which filebeat version are you testing it?

[humberto-garza]

@jlind23, we are using filebeat-7.10.0

Originally we were using 7.8 (as mentioned in the description), but even after upgrading, we have the same issue.

[jlind23]

@humberto-garza is there any chance to test it with the latest filebeat release available?

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

[jon-eicher]

Still seeing the issue too.

Feb 1 20:55:48 ip-x-x-x-x filebeat[471]: {"log.level":"error","@timestamp":"2023-02-01T20:55:47.995Z","log.logger":"reader_json","log.origin":{"file.name":"readjson/json.go","file.line":75},"message":"Error decoding JSON: invalid character '.' looking for beginning of value","service.name":"filebeat","ecs.version":"1.6.0"}

[lysenkojito]

https://discuss.elastic.co/t/filebeat-on-ubuntu-hosts-getting-sporadic-error-error-decoding-json/258783

have you solved it?

[C0rn3j]

Final edit and a complete rewrite:

Well, this turns out to be a PEBKAC issue, and since I wasted many hours on debugging it, I might as well describe it in detail.

The container input backend, or the container parser in filestream input, expect a message like this from docker:

{"log":"This is a log\n","stream":"stdout","time":"2024-01-24T19:09:55.568450771Z"}

That's the log message with a newline at the end of string, and some other parameters. If it doesn't have that newline, it's ignored.

The following filebeat config will (somewhat, if we ignore the newline char) correctly parse that.

- type: filestream
  id: docker-logs
  paths:
    - /var/lib/docker/containers/*/*-json.log
  parsers:
    - container: ~

Resulting in an object like this:

{
...
"message": "This is a log\n"
...
}

Unfortunately, me, OP and the rest of the village folk have seen the helpful example config for nested JSON objects, and didn't realize its purpose, the fact that the documentation is very confusing does not help.

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html#filebeat-input-container-config-json

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-ndjson

"This can be helpful in situations where the application logs are wrapped in JSON objects, like when using Docker."...
Unless you already used the container parser....

So if your config looks like this instead for the same message (or equivalent config with json in container input like OP's example):

- type: filestream
  id: docker-logs
  paths:
    - /var/lib/docker/containers/*/*-json.log
  parsers:
    - container: ~
    - ndjson:
        target: ""
        add_error_key: true

It's going to violently break itself with unhelpful error messages.

So unless you have a log likes this

{"log":"{\"message\":\"Started synchronization.\",\"request_id\":null}}\n","stream":"stdout","time":"2024-01-24T16:42:51.200033705Z"}

The above config makes no sense and you should not parse it twice with ndjson.

If you happen to have both formats and need to process both, config like this to avoid running the parser against the simple log form will do: Actually that's wrong as that just drops the non-nested ones... well, will edit this later

# This will drop non-nested messages, if you want to log both, this is not what you want
- type: filestream
  id: docker-logs
  paths:
    - /var/lib/docker/containers/*/*-json.log
  parsers:
    - container: ~
    - include_message.patterns: ['^\{']
    - ndjson:
        target: ""
        add_error_key: true

This issue can be closed, unless someone wants to improve the documentation with clear examples.

[Diluka]

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/lib/docker/containers/${data.container.id}/*.log
        parsers:
          - ndjson:
              message_key: "log"
        multiline.type: pattern
        multiline.pattern: ' \d+ --- \['
        multiline.negate: true
        multiline.match: after

this is working with 8.12.0