add rule templates sync workflow #2012

orouz · 2024-03-10T11:12:49Z

Summary of your changes

adds a new workflow that triggers on:

every push to main
only when some security-policies/bundle/compliance/**/rules/**/data.yaml file changes/added.

when ran, the script will:

generate the CIS rules templates
open/update a PR in the integrations repo
bump integrations version

Screenshot/Data

Related Issues

closes Automate rule templates creation in integrations repo #1996

mergify · 2024-03-10T11:13:24Z

This pull request does not have a backport label. Could you fix it @orouz? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

github-actions · 2024-03-10T11:13:51Z

📊 Allure Report - 💚 No failures were reported.

Result	Count
🟥 Failed	0
🟩 Passed	162
⬜ Skipped	0

orouz · 2024-03-13T17:32:42Z

scripts/common.sh

added new utility functions to bump an integration version. the version bump dispatch workflow also does this and later on it will use these functions

this file is currently excluded from shellcheck and introduces 2 new errors:

In scripts/common.sh line 93: local second_version="$(echo "${major2}.${minor2}.x" | xargs)" ^------------^ SC2155 (warning): Declare and assign separately to avoid masking return values. In scripts/common.sh line 124: sed -i '' -e '3i\'$'\n'"$next_entry" "$changelog_path" ^-- SC1003 (info): Want to escape a single quote? echo 'This is how it'\''s done'.

it works without resolving them and they don't seem critical. someday later on we'll remove this file from the excluded paths and fix all errors.

orouz · 2024-03-13T17:38:15Z

scripts/common.sh

+# bumps existing preview version: 1.0.0-preview01 -> 1.0.0-preview02, or
+# creates a new preview version: 1.0.0 -> 1.1.0-preview01, and
+# updates the manifest and changelog files
+bump_integration_version() {


a PR to an integration must include a changelog and manifest update. depending on when the PR is made, the current integration version may be one of the following:

some preview version, like 1.0.0-preview01, in which case the preview suffix will be incremented. see example PR for actual changes made

some version, like 1.0.0, in which case the changes will result in this example diff:

+++ b/packages/cloud_security_posture/changelog.yml @@ -1,5 +1,6 @@ # newer versions go on top # version map: +# 1.10.x - 8.15.x # 1.9.x - 8.14.x # 1.8.x - 8.13.x # 1.7.x - 8.12.x @@ -8,6 +9,11 @@ # 1.4.x - 8.9.x # 1.3.x - 8.8.x # 1.2.x - 8.7.x +- version: 1.10.0-preview01 + changes: + - description: Bump version + type: enhancement + link: https://github.com/elastic/integrations/pull/9328 - version: "1.9.0" changes: - description: Convert fields to secrets diff --git a/packages/cloud_security_posture/manifest.yml b/packages/cloud_security_posture/manifest.yml index bda29ce3e..aaa4fcca8 100644 --- a/packages/cloud_security_posture/manifest.yml +++ b/packages/cloud_security_posture/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.0.0 name: cloud_security_posture title: "Security Posture Management" -version: "1.9.0" +version: "1.10.0-preview01" source: license: "Elastic-2.0" description: "Identify & remediate configuration risks in your Cloud infrastructure" @@ -11,7 +11,7 @@ categories: - cloudsecurity_cdr conditions: kibana: - version: "^8.14.0" + version: "^8.15.0" elastic: subscription: basic capabilities:

orouz · 2024-03-13T17:48:39Z

scripts/sync_rule_templates.sh

this script does the following:

checkout a new branch from main or the existing branch, with hard reset to main

reset is done to avoid conflicts (in manifest/changelog), or outdated branch if a previous PR was closed and not merged, for some reason

checking out existing branch instead of deleting to avoid closing an existing PR

generate the rule templates from cloudbeat's main

this means that if for example, PR-1 merged a rule and triggered a workflow to open a PR to integrations, then PR-2 was merged to cloudbeat's main with another rule, the generation will include both rules

commit new rule templates (PR will always have only 2 commits: adding templates and bumping version)

if a PR is not opened, open it and assign labels.

bump integration version

edit the PR body with a nice markdown table with links to added rule templates

gh api is used instead of gh pr because the latter is buggy

orouz · 2024-03-13T17:51:13Z

scripts/sync_rule_templates.sh

+        -f "labels[]=Team:Cloud Security" -f "labels[]=enhancement"
+fi
+
+pr_url=$(gh api $repo/pulls -q ".[] | select(.head.ref == \"$branch_name\" and .state == \"open\") | .html_url")


we need to open a PR and update it so we can use the PR link for the changelog.yml entry and the PR body table links

orouz · 2024-03-13T17:59:45Z

scripts/sync_rule_templates.sh

+git push origin "$branch_name"
+
+# create PR body
+rows="$(git diff --name-only origin/main -- "$templates_path" | while read -r file; do jq --arg a "$pr_url/files#diff-$(echo -n "$file" | openssl dgst -sha256 | awk '{print $2}')" -r '.attributes.metadata.benchmark | "\(.id): \(.rule_number): \($a)"' "$file"; done | awk '{split($0, a, ": "); b[a[1]] = (b[a[1]] == "" ? "" : b[a[1]] ", ") "["a[2]"]""("a[3]")"} END {for (i in b) printf("| %s | %s |\n", i, b[i])}')"


this abomination creates rows with benchmark.id in the 1st column and all of its rules' benchmark.rule_number in the 2nd column. table does look cool though..

oren-zohar

LGTM. I think sync_rule_templates.sh became too complex and would be hard to maintain :(

oren-zohar · 2024-03-25T13:20:17Z

.github/workflows/sync-rule-templates.yml

+      - name: Checkout Integrations repo
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.CLOUDSEC_MACHINE_TOKEN }}


why do we need a token for checkout?

because we need access to push/open PR to the integrations too

oren-zohar · 2024-03-25T13:26:20Z

.github/workflows/sync-rule-templates.yml

+      - name: Install Poetry
+        working-directory: cloudbeat
+        run: |
+          curl -sSL https://install.python-poetry.org | python3 -


We usually install it using a package manager like pip/pipx in other workflows, any reason to install it directly?

Suggested change

curl -sSL https://install.python-poetry.org | python3 -

pipx install poetry

looks like most workflows install in via curl. hardly an issue IMO

oren-zohar · 2024-03-25T13:33:34Z

scripts/sync_rule_templates.sh

+git push origin "$branch_name"
+
+# create PR body
+rows="$(git diff --name-only origin/main -- "$templates_path" | while read -r file; do jq --arg a "$pr_url/files#diff-$(echo -n "$file" | openssl dgst -sha256 | awk '{print $2}')" -r '.attributes.metadata.benchmark | "\(.id): \(.rule_number): \($a)"' "$file"; done | awk '{split($0, a, ": "); b[a[1]] = (b[a[1]] == "" ? "" : b[a[1]] ", ") "["a[2]"]""("a[3]")"} END {for (i in b) printf("| %s | %s |\n", i, b[i])}')"


mergify bot assigned orouz Mar 10, 2024

mergify bot added the backport-skip label Mar 10, 2024

mergify bot mentioned this pull request Mar 10, 2024

add rule templates sync workflow #2011

Closed

orouz force-pushed the rule_templates_auto branch 2 times, most recently from 5b9fb1e to d856e33 Compare March 13, 2024 17:26

orouz commented Mar 13, 2024

View reviewed changes

orouz force-pushed the rule_templates_auto branch from d856e33 to 0afbdf4 Compare March 14, 2024 09:25

add sync integration rules workflow

030059e

orouz force-pushed the rule_templates_auto branch from bfea20a to 030059e Compare March 14, 2024 10:54

orouz marked this pull request as ready for review March 14, 2024 11:13

orouz requested a review from a team as a code owner March 14, 2024 11:13

orouz mentioned this pull request Mar 14, 2024

minor improvements to version bump workflow #1964

Merged

oren-zohar approved these changes Mar 25, 2024

View reviewed changes

Merge branch 'main' into rule_templates_auto

9573600

orouz merged commit 6db9c91 into main Mar 26, 2024
27 checks passed

orouz deleted the rule_templates_auto branch March 26, 2024 10:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add rule templates sync workflow #2012

add rule templates sync workflow #2012

orouz commented Mar 10, 2024 •

edited

Loading

mergify bot commented Mar 10, 2024

github-actions bot commented Mar 10, 2024 •

edited

Loading

orouz Mar 13, 2024

orouz Mar 13, 2024

orouz Mar 13, 2024

orouz Mar 13, 2024 •

edited

Loading

orouz Mar 13, 2024

orouz Mar 13, 2024

orouz Mar 13, 2024

oren-zohar Mar 25, 2024

oren-zohar left a comment

oren-zohar Mar 25, 2024

orouz Mar 26, 2024

oren-zohar Mar 25, 2024

orouz Mar 26, 2024

oren-zohar Mar 25, 2024

	curl -sSL https://install.python-poetry.org \| python3 -
	pipx install poetry

add rule templates sync workflow #2012

add rule templates sync workflow #2012

Conversation

orouz commented Mar 10, 2024 • edited Loading

Summary of your changes

Screenshot/Data

Related Issues

mergify bot commented Mar 10, 2024

github-actions bot commented Mar 10, 2024 • edited Loading

📊 Allure Report - 💚 No failures were reported.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orouz Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oren-zohar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

orouz commented Mar 10, 2024 •

edited

Loading

github-actions bot commented Mar 10, 2024 •

edited

Loading

orouz Mar 13, 2024 •

edited

Loading