Single pipeline entrypoint #38783

pazone · 2024-04-09T11:33:19Z

Proposed commit message

A single pipeline entry point that triggers other pipelines according to the changeset.
The Main pipeline triggers corresponding pipelines: Auditbeat and Heartbeat.

the statuses are reported back to GH (Simply auditbeat and heartbeat) :

Labels and other critical env vars are passed to the trigger step

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

botelastic · 2024-04-09T11:33:24Z

This pull request doesn't have a Team:<team> label.

mergify · 2024-04-09T11:34:09Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @pazone? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-04-09T11:44:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 12 min 14 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

pazone · 2024-04-09T15:00:44Z

/test

pazone · 2024-04-09T15:04:37Z

The triggered pipeline by the comment

alexsapran · 2024-04-09T15:31:10Z

.buildkite/hooks/pre-command

@@ -65,6 +65,7 @@ if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats" || "$BUILDKITE_PIPELINE_SLUG" == "fi
 fi

 ENABLED_BEATS_PIPELINES_SLUGS=(
+  "beats"


Do we need any of those env vars or secrets since the job triggers the other pipelines? (ie delete this line)

same comment from me, let's remove this because it's not useful in anyway for the execution of the mainpipeline, via the monorepo plugin.

For the pre-exit hook. It uses util.sh and it needs BIN var

dliappis

Basically LGTM -- once we've addressed the comment about the unnecessary change in pre-command let's ship this.

dliappis · 2024-04-09T15:37:02Z

.buildkite/hooks/pre-command

@@ -65,6 +65,7 @@ if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats" || "$BUILDKITE_PIPELINE_SLUG" == "fi
 fi

 ENABLED_BEATS_PIPELINES_SLUGS=(
+  "beats"


same comment from me, let's remove this because it's not useful in anyway for the execution of the mainpipeline, via the monorepo plugin.

dliappis · 2024-04-09T15:54:52Z

@pazone note that for backports we should only backport to 8.13 and 7.17

dliappis

Lgtm

alexsapran

lgtm, but needs further improvement that can happen in follow up work

* Single pipeline entrypoint * Fix indentations * Notifications (cherry picked from commit d0a3b45) # Conflicts: # .buildkite/auditbeat/auditbeat-pipeline.yml # .buildkite/heartbeat/heartbeat-pipeline.yml # .buildkite/pipeline.yml # .buildkite/scripts/common.sh

* Single pipeline entrypoint * Fix indentations * Notifications (cherry picked from commit d0a3b45) # Conflicts: # .buildkite/heartbeat/heartbeat-pipeline.yml # .buildkite/pipeline.yml # .buildkite/scripts/generate_auditbeat_pipeline.sh

* This commit refactors the x-pack/osquerybeat pipeline to the pipeline selector approach, as laid out in #38783. Relates: elastic/ingest-dev#3072 * remove pull-requests.json entry * fix typo in pipeline name * fix copy/pasta for aws label from metricbeat --------- Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> Co-authored-by: Julien Lind <julien.lind@elastic.co>

* This commit refactors the x-pack/osquerybeat pipeline to the pipeline selector approach, as laid out in #38783. Relates: https://github.com/elastic/ingest-dev/issues/3072 * remove pull-requests.json entry * fix typo in pipeline name * fix copy/pasta for aws label from metricbeat --------- Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> Co-authored-by: Julien Lind <julien.lind@elastic.co> (cherry picked from commit 6e3be8b)

* This commit refactors the x-pack/osquerybeat pipeline to the pipeline selector approach, as laid out in #38783. Relates: https://github.com/elastic/ingest-dev/issues/3072 * remove pull-requests.json entry * fix typo in pipeline name * fix copy/pasta for aws label from metricbeat --------- Co-authored-by: Alexandros Sapranidis <alexandros@elastic.co> Co-authored-by: Julien Lind <julien.lind@elastic.co> (cherry picked from commit 6e3be8b) # Conflicts: # .buildkite/x-pack/pipeline.xpack.osquerybeat.yml

This commit refactors the x-pack/osquerybeat pipeline to the pipeline selector approach, as laid out in #38783. Relates: https://github.com/elastic/ingest-dev/issues/3072 --------- (cherry picked from commit 6e3be8b)

…#38974) This commit refactors the x-pack/osquerybeat pipeline to the pipeline selector approach, as laid out in #38783. Relates: https://github.com/elastic/ingest-dev/issues/3072 (cherry picked from commit 6e3be8b)

elastic#38821) After the introduction of a centralized beats pipeline and statically defined changeset rules in elastic#38783, we still need a way to grab changes for modules for IT tests. Currently this is done in common.sh#defineModuleFromTheChangeSet(). This commit adds a new script that has only the changeset related functionality, required to build the `MODULE` env var. The functions themselves, are copied from common.sh (with minor enhancements in documentation), paving the way to sunset common.sh in the future. Relates: elastic/ingest-dev#3072 (cherry picked from commit d0dda41) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

) * Single pipeline entrypoint (elastic#38783) * Single pipeline entrypoint * Fix indentations * Notifications (cherry picked from commit d0a3b45) # Conflicts: # .buildkite/auditbeat/auditbeat-pipeline.yml # .buildkite/heartbeat/heartbeat-pipeline.yml # .buildkite/pipeline.yml # .buildkite/scripts/common.sh * Resolved conflicts --------- Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co>

…eat (elastic#38858) This commit refactors the metricbeat pipeline to the pipeline selector approach as laid out in elastic#38783 Relates: elastic/ingest-dev#3072 (comment) (cherry picked from commit 60b3774) Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>

…metricbeat (elastic#38931) This commit refactors the x-pack/metricbeat pipeline to the pipeline selector approach, as laid out in elastic#38783 Relates: elastic/ingest-dev#3072 (cherry picked from commit 4f4fad6)

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in elastic#38783.

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates elastic/ingest-dev#3072

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates https://github.com/elastic/ingest-dev/issues/3072 (cherry picked from commit 8f8f313) # Conflicts: # .buildkite/packetbeat/pipeline.packetbeat.yml # .buildkite/pipeline.yml # .buildkite/scripts/common.sh

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates https://github.com/elastic/ingest-dev/issues/3072 (cherry picked from commit 8f8f313) # Conflicts: # .buildkite/pipeline.yml # .buildkite/scripts/common.sh

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in elastic#38783. Relates elastic/ingest-dev#3072

…ne (#39108) This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates elastic/ingest-dev#3072 Cherry-picked from 8f8f313

…ne (#39109) This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates elastic/ingest-dev#3072 Cherry-picked from 8f8f313

…ne (#39110) This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates elastic/ingest-dev#3072 Cherry-picked from 8f8f313

Single pipeline entrypoint

0bf638b

pazone added the enhancement label Apr 9, 2024

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 9, 2024

mergify bot assigned pazone Apr 9, 2024

Fix indentations

3cf9dbd

Notifications

c8fbba9

oakrizan approved these changes Apr 9, 2024

View reviewed changes

dliappis marked this pull request as ready for review April 9, 2024 15:13

dliappis requested a review from a team as a code owner April 9, 2024 15:13

pazone requested review from alexsapran, dliappis and sharbuz April 9, 2024 15:17

alexsapran requested changes Apr 9, 2024

View reviewed changes

dliappis requested changes Apr 9, 2024

View reviewed changes

pazone added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.13.0 Automated backport with mergify labels Apr 9, 2024

dliappis approved these changes Apr 9, 2024

View reviewed changes

alexsapran approved these changes Apr 9, 2024

View reviewed changes

pazone merged commit d0a3b45 into main Apr 9, 2024
91 of 97 checks passed

pazone deleted the single_pipeline_entrypoint branch April 9, 2024 19:43

mergify bot mentioned this pull request Apr 9, 2024

[7.17](backport #38783) Single pipeline entrypoint #38793

Merged

6 tasks

mergify bot mentioned this pull request Apr 9, 2024

[8.13](backport #38783) Single pipeline entrypoint #38794

Merged

6 tasks

dliappis mentioned this pull request Apr 15, 2024

Migrate x-pack/osquerybeat to static pipeline #38939

Merged

This was referenced Apr 16, 2024

[7.17](backport #38939) Migrate x-pack/osquerybeat to static pipeline #38974

Merged

[8.13](backport #38939) Migrate x-pack/osquerybeat to static pipeline #38975

Merged

dliappis added a commit to dliappis/beats that referenced this pull request Apr 19, 2024

Migrate packetbeat to static Buildkite pipeline

22bdc73

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in elastic#38783.

dliappis mentioned this pull request Apr 19, 2024

Migrate packetbeat to static Buildkite pipeline #39087

Merged

dliappis added a commit that referenced this pull request Apr 22, 2024

Migrate packetbeat to static Buildkite pipeline (#39087)

8f8f313

This commit refactors the packetbeat pipeline to the pipeline selector approach as laid out in #38783. Relates elastic/ingest-dev#3072

This was referenced Apr 22, 2024

[7.17](backport #39087) Migrate packetbeat to static Buildkite pipeline #39104

Closed

[8.13](backport #39087) Migrate packetbeat to static Buildkite pipeline #39105

Closed

[8.14](backport #39087) Migrate packetbeat to static Buildkite pipeline #39106

Closed

dliappis mentioned this pull request Apr 22, 2024

[8.14](backport #39087) Migrate packetbeat to static Buildkite pipeline #39108

Merged

dliappis mentioned this pull request Apr 22, 2024

[8.13](backport #39087) Migrate packetbeat to static Buildkite pipeline #39109

Merged

dliappis mentioned this pull request Apr 22, 2024

[7.17](backport #39087) Migrate packetbeat to static Buildkite pipeline #39110

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Single pipeline entrypoint #38783

Single pipeline entrypoint #38783

pazone commented Apr 9, 2024 •

edited

Loading

botelastic bot commented Apr 9, 2024

mergify bot commented Apr 9, 2024

elasticmachine commented Apr 9, 2024 •

edited

Loading

Build stats

pazone commented Apr 9, 2024

pazone commented Apr 9, 2024

alexsapran Apr 9, 2024 •

edited

Loading

dliappis Apr 9, 2024

pazone Apr 9, 2024

dliappis left a comment

dliappis Apr 9, 2024

dliappis commented Apr 9, 2024

dliappis left a comment

alexsapran left a comment

Single pipeline entrypoint #38783

Single pipeline entrypoint #38783

Conversation

pazone commented Apr 9, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

botelastic bot commented Apr 9, 2024

mergify bot commented Apr 9, 2024

elasticmachine commented Apr 9, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

pazone commented Apr 9, 2024

pazone commented Apr 9, 2024

alexsapran Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

dliappis Apr 9, 2024

Choose a reason for hiding this comment

pazone Apr 9, 2024

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

dliappis Apr 9, 2024

Choose a reason for hiding this comment

dliappis commented Apr 9, 2024

dliappis left a comment

Choose a reason for hiding this comment

alexsapran left a comment

Choose a reason for hiding this comment

pazone commented Apr 9, 2024 •

edited

Loading

elasticmachine commented Apr 9, 2024 •

edited

Loading

alexsapran Apr 9, 2024 •

edited

Loading