Tests failing with constant keywords in transforms with LogsDB

[jsoriano]

Some packages are failing when logsdb is enabled with:

test case failed: errors found in documents of preview for transform logs-ti_cybersixgill.latest_ioc-default-0.3.0 for data stream logs-ti_cybersixgill.threat-34090: [0] parsing field value failed: field "labels.is_ioc_transform_source"'s value "true" does not match the declared constant_keyword value "false"

For testing we use the preview transform API, where the value actually doesn't match with the definition, but this doesn't seem to be an issue in a real deployment. Confirm if this is actually not an issue, and adjust the tests so they don't fail on this case.

[jsoriano]

@kpollich this would be kind of urgent because there are many tests failing in the daily builds because of this.

[kpollich]

Thanks @jsoriano I added this to our current sprint.

[jsoriano]

@chrisberkhout I was working on this issue, using the ti_anomali package, and I have found that the transform is actually failing:

java.lang.IllegalArgumentException: [constant_keyword] field [labels.is_ioc_transform_source] only accepts values that are equal to the value defined in the mappings [false], but got [true]

This can be reproduced with:

elastic-package test system --setup -v --config-file ./data_stream/threatstream/_dev/test/system/test-integrator-http-config.yml

So there seems to be an actual issue.

We would need to work on two things:

• Ensuring that issues in transforms are detected by elastic-package test.
• Fixing these packages. For this maybe we need to add ingest pipelines to remove or modify these labels.is_ioc_transform_source fields in transform destination indexes.

[jsoriano]

Fixing these packages. For this maybe we need to add ingest pipelines to remove or modify these labels.is_ioc_transform_source fields in transform destination indexes.

Tried to add a pipeline but I receive the same failure, even if logsdb is not enabled 🤔 elastic/integrations#11750

[jsoriano]

Fixing these packages. For this maybe we need to add ingest pipelines to remove or modify these labels.is_ioc_transform_source fields in transform destination indexes.

Tried to add a pipeline but I receive the same failure, even if logsdb is not enabled 🤔 elastic/integrations#11750

Ah, I think the problem was on the type, I was setting it to boolean false, instead of to string "false".

[jsoriano]

There is already an open issue about testing transforms: #859
I have opened another one about supporting ingest pipelines in transforms: elastic/package-spec#833