Error message shown up while adding Kafka integration (v0.3.6 on 7.12 BC2)

[dikshachauhan-qasource]

Kibana version:
7.12 BC2 Kibana Cloud environment
BUILD 39000
COMMIT 4f65a5a1268fa78f1af9117d12312e1cee433376

Steps to reproduce:

1. Create a new policy in Fleet (no Agents assigned to this policy yet)
2. Add Integration 'Kafka' to this new policy.
3. Observe an error message is shown up. Screenshot attached for reference.

Expected Result:
No error message should be displayed while adding kafka integration to any Policy.

Screenshots:

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[dikshachauhan-qasource]

@manishgupta-qasource Please review

[manishgupta-qasource]

Reviewed & assigned to @EricDavisX

[EricDavisX]

I verified the issue and confirmed it is seen on v0.3.6 of the Kafka package, as the latest seen with a 7.12 stack currently.

I don't see any other Kafka packages going through the package-storage stages - @masci let me know if you need any help seeing this or discussing testing. The team doesn't usually dive deep on any integrations other than System, Linux, Windows, and Endpoint - but we do exploratory tests and are happy to log bugs when we see them.

for everyone, including @dikshachauhan-qasource foremost, we acknowledge prior discussion that some integrations Will Not function properly if there is no corresponding data pushing it, but since this is not even deployed to any Agent and fails at this stage it seems a clear bug. And I assume in the package, if it is believed on Fleet side (or other) please do push it back with comment, etc. Thanks.

[ycombinator]

The team doesn't usually dive deep on any integrations other than System, Linux, Windows, and Endpoint - but we do exploratory tests and are happy to log bugs when we see them.

Thanks @EricDavisX and @dikshachauhan-qasource for catching this.

Probably worth noting that currently there are no system tests defined for the kafka package's log data stream, which appears to be the problematic one according the screenshot. I think adding a system test would've caught this error automatically as part of the integrations repo CI — although, looking at this particular error, it doesn't appear to be specific to the kafka package but probably in Fleet UI code.

[ycombinator]

I'm able to repro this error with Kibana 8.0.0-SNAPSHOT as well. Grabbed the full stack trace for the error from the Kibana server log:

YAMLException: bad indentation of a sequence entry at line 2, column 17:
     - \"/opt/kafka*\"\"/logs/controller.log*\"
                    ^
    at generateError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
    at throwError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:173:9)
    at readBlockSequence (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:962:7)
    at composeNode (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1358:12)
    at readBlockMapping (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1089:11)
    at composeNode (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1359:12)
    at readDocument (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1525:3)
    at loadDocuments (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1588:5)
    at load (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1614:19)
    at safeLoad (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1637:10)
    at compileTemplate (/usr/share/kibana/x-pack/plugins/fleet/server/services/epm/agent/agent.js:37:57)
    at _compilePackageStream (/usr/share/kibana/x-pack/plugins/fleet/server/services/package_policy.js:534:43)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Promise.all (index 0)
    at _compilePackageStreams (/usr/share/kibana/x-pack/plugins/fleet/server/services/package_policy.js:495:10) {
  reason: 'bad indentation of a sequence entry',
  mark: Mark {
    name: null,
    buffer: 'paths:\
' +
      ' - \"/opt/kafka*\"\"/logs/controller.log*\"\
' +
      ' - \"/opt/kafka*\"\"/logs/server.log*\"\
' +
      ' - \"/opt/kafka*\"\"/logs/state-change.log*\"\
' +
      ' - \"/opt/kafka*\"\"/logs/kafka-*.log*\"\
' +
      'exclude_files: [\".gz$\"]\
' +
      'multiline:\
' +
      \"  pattern: '^\\\\['\
\" +
      '  negate: true\
' +
      '  match: after\
' +
      'processors:\
' +
      '- add_locale: ~\
' +
      '\\x00',
    position: 23,
    line: 1,
    column: 16
  }
}

[ycombinator]

Still not sure if this is a Fleet UI bug or an issue in the kafka package itself. But the part of the package that's being reported in the error is this:

integrations/packages/kafka/data_stream/log/agent/stream/log.yml.hbs

Line 3 in 06284f6

- {{../kafka_home}}{{path}}

[jsoriano]

Still not sure if this is a Fleet UI bug or an issue in the kafka package itself. But the part of the package that's being reported in the error is this:

integrations/packages/kafka/data_stream/log/agent/stream/log.yml.hbs

Line 3 in 06284f6

- {{../kafka_home}}{{path}}

Yep, this configuration concatenates two variables this way. This works in Filebeat and used to work in Fleet (at least till 7.11).

It seems that now double-quoutes are being added to each placeholder, generating bad yaml:

- "/opt/kafka*""/logs/controller.log*"

This looks like some change in Fleet.

[exekias]

Can someone from the @elastic/fleet team confirm? I wonder if this new behavior is expected or this can be considered a bug in the UI

[mtojek]

I suspect two issues:

1. Kafka package:
   Double dots and slash look suspicious here: https://github.com/elastic/integrations/blob/master/packages/kafka/data_stream/log/agent/stream/log.yml.hbs#L3

2. Kibana implementation:
   Kibana doesn't play well in terms of vars concatenation: https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/agent/agent.ts#L68

For example:

paths:
{{#each paths as |path i|}}
- {{kafka_home}}{{path}}
{{/each}}

results in: "/opt/kafka*""/logs/controller.log*"

@skh Could you please confirm the issue no. 2?

EDIT:

I understand that it may be hard to introduce a bugfix for concatenating unsafe fields, so maybe a special concat helper?

[skh]

Maybe I was overeager in the fix for elastic/kibana#91401 .

What I was trying to do is enclose strings in double quotes that contained characters with special meaning in yaml, because we had parsing errors on * which is used as a wildcard in strings.

From this list of characters, which would you recommend to take out of this special handling? { and } obviously from the bug you're seeing, but are there more?

  // List of special chars that may lead to YAML parsing errors when not quoted.
  // See YAML specification section 5.3 Indicator characters
  // https://yaml.org/spec/1.2/spec.html#id2772075
  const yamlSpecialCharsRegex = /[{}\[\],&*?|\-<>=!%@:]/;

(from https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/agent/agent.ts#L60-L71 )

[mtojek]

Actually I would suggest to rethink the idea of escaping here as it will always harm concatenation. Maybe we should consider the package as the only source of truth? Whether a variable should be escaped or not.

EDIT:

there is always a nasty hack to silently remove all "" (double quotes) which are results of such concatenation :)

[skh]

The * in question was a "*" in the package, but something already stripped the double quotes which lead to the original bug.

[skh]

I spoke too soon. The issues are the wildcards in the values /opt/kafka* and e.g. /logs/controller.log*. I'll spare you the details, and I'm preparing a fix for Fleet/Kibana.

[EricDavisX]

transferring to Kibana repo.

[skh]

Fixed in elastic/kibana#93585