Move to static BK pipeline for metricbeat #38827

dliappis · 2024-04-10T12:56:14Z

Proposed commit message

This commit refactors the metricbeat pipeline to the pipeline selector
approach as laid out in #38783

Related issues

Relates https://github.com/elastic/ingest-dev/issues/3072#issuecomment-2046970938

Tests

Without any labels: https://buildkite.com/elastic/beats-metricbeat/builds/4684
Labels: Using windows-2019 to trigger the corresponding extended test: https://buildkite.com/elastic/beats-metricbeat/builds/4691
Labels: Using windows-10/11/2019 to trigger ALL extended windows tests: https://buildkite.com/elastic/beats-metricbeat/builds/4694
Labels: Using macos: https://buildkite.com/elastic/beats-metricbeat/builds/4721#018ecc0d-3559-4852-9fcc-0caf7980f66c
Complete run with all labels and latest commit: https://buildkite.com/elastic/beats-metricbeat/builds/4751 / https://buildkite.com/elastic/beats-metricbeat/builds/4764

GH status screenshots

Mandatory

Extended Windows

Packaging

macOS (x86_64)

This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in elastic#38783 Relates elastic/ingest-dev#3072 (comment)

This reverts commit 4165994.

After the introduction of a centralized beats pipeline and statically defined changeset rules in elastic#38783, we still need a way to grab changes for modules for IT tests. Currently this is done in common.sh#defineModuleFromTheChangeSet(). This commit adds a new script that has only the changeset related functionality, required to build the `MODULE` env var. The functions themselves, are copied from common.sh (with minor enhancements in documentation), paving the way to sunset common.sh in the future.

…c-pipeline

and remove generator

mergify · 2024-04-10T12:57:38Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b metricbeat-static-pipeline upstream/metricbeat-static-pipeline
git merge upstream/main
git push upstream metricbeat-static-pipeline

mergify · 2024-04-10T12:57:38Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @dliappis? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-04-10T13:14:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 13 min 3 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Install kind via asdf, ensure goIntegTests work correctly and small fixes in group logic.

dliappis · 2024-04-11T10:36:10Z

/test

dliappis · 2024-04-11T11:14:56Z

.buildkite/metricbeat/pipeline.yml

+
+  - group: "Metricbeat Extended Windows Tests"
+    key: "metricbeat-extended-win-tests"
+    if: build.env("BUILDKITE_PULL_REQUEST") == "false" || build.env("GITHUB_PR_LABELS") =~ /.*windows.*/


todo: make this more permissive

dliappis · 2024-04-11T11:16:54Z

.buildkite/metricbeat/pipeline.yml

+  K8S_VERSION: "v1.29.0"
+  ASDF_KIND_VERSION: "0.20.0"
+
+  # Packaging
  PACKAGING_ARM_PLATFORMS: "linux/arm64"


move to packaging

removed in 9185c7d / 5c1a493

dliappis · 2024-04-11T11:44:46Z

/test

.buildkite/pipeline.yml

Co-authored-by: Alexandros Sapranidis <alexsapran@gmail.com>

This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in #38783 Relates: https://github.com/elastic/ingest-dev/issues/3072#issuecomment-2046970938 (cherry picked from commit 60b3774) # Conflicts: # .buildkite/metricbeat/pipeline.yml # .buildkite/pipeline.yml

This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in #38783 Relates: https://github.com/elastic/ingest-dev/issues/3072#issuecomment-2046970938 (cherry picked from commit 60b3774) # Conflicts: # .buildkite/pipeline.yml

…8859) This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in #38783 Relates: https://github.com/elastic/ingest-dev/issues/3072#issuecomment-2046970938 (cherry picked from commit 60b3774) --------- Signed-off-by: Alexandros Sapranidis <alexandros@elastic.co> Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co>

…8858) This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in #38783 Relates: https://github.com/elastic/ingest-dev/issues/3072#issuecomment-2046970938 (cherry picked from commit 60b3774) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

…eat (elastic#38858) This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in elastic#38783 Relates: elastic/ingest-dev#3072 (comment) (cherry picked from commit 60b3774) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

dliappis added 9 commits April 10, 2024 12:21

Move to static BK pipeline for metricbeat

8f3ef35

This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in elastic#38783 Relates elastic/ingest-dev#3072 (comment)

test changeset

4165994

Revert "test changeset"

6502b2e

This reverts commit 4165994.

Merge branch 'bk-dedicated-changeset-functions' into metricbeat-stati…

9dd981a

…c-pipeline

Remove beats-metricbeat from pull-requests.json

cf560da

add metricbeat to main pipeline + trigger

7bfaba3

remove metricbeat references from scripts

453bea7

and remove generator

update static pipeline

186183b

dliappis added ci Team:Ingest-EngProd labels Apr 10, 2024

dliappis self-assigned this Apr 10, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 10, 2024

Merge branch 'main' into metricbeat-static-pipeline

e6f7722

dliappis added 6 commits April 10, 2024 16:25

typo

d925596

fix

f25c419

another fix

cf08282

more fixes

fbc37e3

fix bug

b420de4

Merge branch 'main' into metricbeat-static-pipeline

688749f

dliappis added windows-2016 Enable builds in the CI for windows-2016 windows-10 Enable builds in the CI for windows-10 windows-2019 windows-2022 windows-11 Enable builds in the CI for windows-10 labels Apr 10, 2024

Various bug fixes and cleanups

8c10ddd

Install kind via asdf, ensure goIntegTests work correctly and small fixes in group logic.

dliappis commented Apr 11, 2024

View reviewed changes

PR comment move platform env vars to packaging

5c1a493

dliappis added :Windows and removed windows-2016 Enable builds in the CI for windows-2016 windows-10 Enable builds in the CI for windows-10 windows-2019 windows-2022 windows-11 Enable builds in the CI for windows-10 labels Apr 11, 2024

dliappis added 2 commits April 11, 2024 14:34

pr comment

5e563f4

remove packaging

9185c7d

dliappis added 2 commits April 11, 2024 14:52

fix bug with simplified ext-windows conditionals

5aaf2a9

Merge branch 'main' into metricbeat-static-pipeline

80d61f3

dliappis enabled auto-merge (squash) April 11, 2024 12:33

alexsapran reviewed Apr 11, 2024

View reviewed changes

.buildkite/pipeline.yml Outdated Show resolved Hide resolved

Update .buildkite/pipeline.yml

4d4a6ca

Co-authored-by: Alexandros Sapranidis <alexsapran@gmail.com>

dliappis requested a review from alexsapran April 11, 2024 12:45

alexsapran approved these changes Apr 11, 2024

View reviewed changes

dliappis added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.13.0 Automated backport with mergify labels Apr 11, 2024

dliappis merged commit 60b3774 into elastic:main Apr 11, 2024
99 of 128 checks passed

This was referenced Apr 11, 2024

[7.17](backport #38827) Move to static BK pipeline for metricbeat #38858

Merged

[8.13](backport #38827) Move to static BK pipeline for metricbeat #38859

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move to static BK pipeline for metricbeat #38827

Move to static BK pipeline for metricbeat #38827

dliappis commented Apr 10, 2024 •

edited

Loading

mergify bot commented Apr 10, 2024

mergify bot commented Apr 10, 2024

elasticmachine commented Apr 10, 2024 •

edited

Loading

Build stats

dliappis commented Apr 11, 2024

dliappis Apr 11, 2024

dliappis Apr 11, 2024

dliappis Apr 11, 2024

dliappis commented Apr 11, 2024

Move to static BK pipeline for metricbeat #38827

Move to static BK pipeline for metricbeat #38827

Conversation

dliappis commented Apr 10, 2024 • edited Loading

Proposed commit message

Related issues

Tests

GH status screenshots

Mandatory

Extended Windows

Packaging

macOS (x86_64)

mergify bot commented Apr 10, 2024

mergify bot commented Apr 10, 2024

elasticmachine commented Apr 10, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

dliappis commented Apr 11, 2024

dliappis Apr 11, 2024

Choose a reason for hiding this comment

dliappis Apr 11, 2024

Choose a reason for hiding this comment

dliappis Apr 11, 2024

Choose a reason for hiding this comment

dliappis commented Apr 11, 2024

dliappis commented Apr 10, 2024 •

edited

Loading

elasticmachine commented Apr 10, 2024 •

edited

Loading