[Cloud Security] Update manifest format_version to 3.0.0 for cloud_security_posture package

[maxcold]

Proposed commit message

making necessary changes to update cloud_security_posture package format_version to 3.0.0 to opt in it for serverless:

• updated type: object field mappings either to more suitable types or added object_type as it's required now
• added DLM data_retention of 180d for findings index to enable data retention in serverless as ILM is not supported in serverless due to the issue will be handled separately as a part of https://github.com/elastic/security-team/issues/7642
• added security capability so the package shows up in serverless

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

• Install elastic-package and run elastic-package check inside cloud_security_posture package to validate
• Follow the steps in https://docs.google.com/document/d/1VbPZp_JgLuMpKixSGrQrZejARAJYje5zjoS3vmENF-s/edit#heading=h.7ymomq669pb4 to run stack and test the package

Related issues

• Fixes https://github.com/elastic/security-team/issues/7624

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-12T13:50:36.790+0000

• Duration: 15 min 7 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4
Skipped	0
Total	4

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚
Classes	100.0% (0/0)	💚
Methods	25.0% (2/8)	👎 -70.506
Lines	100.0% (0/0)	💚 2.045
Conditionals	100.0% (0/0)	💚

[maxcold]

@orestisfl I found an issue with specifying data_retention for our data stream. When I try to install the package I get an error

"Internal Server Error","message":"illegal_argument_exception\n\tCaused by:\n\t\tillegal_argument_exception: index template [logs-cloud_security_posture.findings_latest] specifies lifecycle configuration that can only be used in combination with a data stream\n\tRoot causes:\n\t\tillegal_argument_exception: updating component template [logs-cloud_security_posture.findings@package] results in invalid composable template [logs-cloud_security_posture.findings_latest] after templates are merged"

I think this is because we have logs-cloud_security_posture.findings which is a data stream and logs-cloud_security_posture.findings_latest which is an index and they share the same index template as it seems. I think it needs more investigation on how to solve it properly.

To not block the format_version update I will remove the lifecycle addition from this PR and summarise the problem in https://github.com/elastic/security-team/issues/7642 to do that as a follow-up. Adding data_retention for our data stream is not a must for serverless, we should be able to do it independent of the version update

[maxcold]

I tested the package by building it and installing it on the stack created via elastic-package. After that, I added CNVM and CSPM integrations to check if the data is ingested properly. Everything looks fine to me

[orestisfl]

In your tests, have you verified that these fields work as expected with the new type?

[maxcold]

Yes, I didn't find anything weird about them. They show up in the document, and flyout Table/JSON view works, but searching doesn't work with these fields as it doesn't work without the change either, I guess because they are not being indexed

[opauloh]

LGTM, just checked integrity using elastic-package check and tested Kibana works properly using elastic-package stack

[elasticmachine]

Package cloud_security_posture - 1.6.2 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture