Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301

gizas · 2024-07-19T12:12:15Z

Enhancement

~~NOTE: Dont merge until elastic/elastic-agent-autodiscover#98 is also merged~~

Proposed commit message

WHAT: Provide a configuration option to disable /api/v1/namespaces/kube-system/configmaps/kubeadm-config for filebeat autodiscovery and add_kubernetes_metadata processor. In more details user will disable the api requests by using configuration use_kubeadm: false

In this PR we added the option in order to support both filbeat.input and filebeat.autodiscover blocks of configuration.

WHY: https://github.com/elastic/enhancements/issues/21172

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Follow same steps as described [Kubernetes - Metricbeat] Providing config option to disable Kubeadm config api requests #40086
The filebeat configuration should contain only autodiscover config: https://github.com/elastic/beats/blob/main/deploy/kubernetes/filebeat-kubernetes.yaml#L133C1-L150C42

Related issues

Screenshots

LOG processing continues successfully:

For the filebeat.input block:

LOG processing continues successfully:

mergify · 2024-07-19T12:12:54Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gizas? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

cla-checker-service · 2024-09-10T08:06:18Z

💚 CLA has been signed

dev-tools/kubernetes/filebeat/Dockerfile.debug

gizas · 2024-09-10T10:29:03Z

libbeat/autodiscover/providers/kubernetes/pod.go

@@ -97,6 +97,15 @@ func NewPodEventer(uuid uuid.UUID, cfg *conf.C, client k8s.Interface, publish fu
 	}

 	metaConf := config.AddResourceMetadata
+	// We initialise the use_kubeadm variable based on modules KubeAdm base configuration
+	err = metaConf.Namespace.SetBool("use_kubeadm", -1, config.KubeAdm)
+	if err != nil {


Added due to go-lint errors

dev-tools/kubernetes/filebeat/Dockerfile.debug

filebeat/_meta/test/docs/01_playground/filebeat.yaml

libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

…_metadata.asciidoc Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

libbeat/processors/add_kubernetes_metadata/kubernetes.go

mergify · 2024-09-11T11:52:51Z

backport-8.x has been added to help with the transition to the new branch 8.x.

filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl

Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

…over) (#40301) * disable kubeadm for filebeat autodiscover * Update filebeat/_meta/test/docs/01_playground/filebeat.yaml Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * Update libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * Update dev-tools/kubernetes/filebeat/Dockerfile.debug Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * fixing docs and adding mericbeat references * fixing docs and adding mericbeat references * Update filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> (cherry picked from commit 89cd840) # Conflicts: # metricbeat/module/kubernetes/util/kubernetes.go

…ut and filbeat.autodiscover) (#40889) * Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) (#40301) * disable kubeadm for filebeat autodiscover * Update filebeat/_meta/test/docs/01_playground/filebeat.yaml Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * Update libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes_metadata.asciidoc Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * Update dev-tools/kubernetes/filebeat/Dockerfile.debug Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> * fixing docs and adding mericbeat references * fixing docs and adding mericbeat references * Update filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> --------- Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co> Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> (cherry picked from commit 89cd840) # Conflicts: # metricbeat/module/kubernetes/util/kubernetes.go * fixing worng backport comments --------- Co-authored-by: Andrew Gizas <andreas.gkizas@elastic.co>

first effort to disable kubeadm for filebeat autodiscover

7ec7a1b

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 19, 2024

gizas added the Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team label Jul 19, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 19, 2024

mergify bot assigned gizas Jul 19, 2024

gizas mentioned this pull request Jul 19, 2024

[META Issue]Enhancement for being able to toggle the kubeadm-config api calls #40276

Open

4 tasks

gizas added 6 commits August 28, 2024 17:01

merging with main

e040782

merging with main util

634c37f

merging with main test file

badda64

adding kubeadm changes and merging to main

2657b9a

adding filebeat

9cb2e68

adding filebeat manifest

9f127ca

mergify bot added the backport-v8.x label Sep 10, 2024

gizas added 3 commits September 10, 2024 11:06

adding go.sum

ed5e62f

removing printf

72e8ec2

Merge branch 'main' into kubeadm_filebeat

4301d26

gizas commented Sep 10, 2024

View reviewed changes

dev-tools/kubernetes/filebeat/Dockerfile.debug Outdated Show resolved Hide resolved

gizas added 6 commits September 10, 2024 11:44

fixing conflicts

df80e6f

fixing conflicts

c57c9a3

adding Changelog

c830308

adding description on relevant refenrence yaml files

f048bb0

adding description on relevant refenrence yaml files

d05c2ac

adding description on relevant refenrence yaml files

ea37261

gizas marked this pull request as ready for review September 10, 2024 09:18

gizas requested review from a team as code owners September 10, 2024 09:18

gizas requested review from belimawr and leehinman September 10, 2024 09:18

gizas commented Sep 10, 2024

View reviewed changes

gizas added 3 commits September 10, 2024 14:23

adding processor fixes

3534ecf

adding processor docs

f5aff00

adding processor docs

cef81a6

gizas changed the title ~~Disable kubeadm for filebeat autodiscover~~ Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) Sep 10, 2024

belimawr reviewed Sep 10, 2024

View reviewed changes

gizas and others added 3 commits September 10, 2024 16:23

Update filebeat/_meta/test/docs/01_playground/filebeat.yaml

831a647

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

Update libbeat/processors/add_kubernetes_metadata/docs/add_kubernetes…

3adaadf

…_metadata.asciidoc Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

Update dev-tools/kubernetes/filebeat/Dockerfile.debug

41ffe1c

Co-authored-by: Tiago Queiroz <tiago.queiroz@elastic.co>

MichaelKatsoulis reviewed Sep 10, 2024

View reviewed changes

libbeat/processors/add_kubernetes_metadata/kubernetes.go Outdated Show resolved Hide resolved

MichaelKatsoulis approved these changes Sep 10, 2024

View reviewed changes

gizas added 2 commits September 10, 2024 17:27

fixing docs and adding mericbeat references

47e82d3

fixing docs and adding mericbeat references

f079b5a

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 11, 2024

v1v removed the backport-v8.x label Sep 11, 2024

Merge branch 'main' into kubeadm_filebeat

e3613ac

leehinman approved these changes Sep 16, 2024

View reviewed changes

tetianakravchenko approved these changes Sep 18, 2024

View reviewed changes

filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl Show resolved Hide resolved

gizas and others added 5 commits September 18, 2024 13:50

Update filebeat/_meta/config/filebeat.autodiscover.reference.yml.tmpl

69862e3

Co-authored-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Merge branch 'main' of github.com:elastic/beats into kubeadm_filebeat

4489901

updating docs

b50a0cb

Merge branch 'main' of github.com:elastic/beats into kubeadm_filebeat

a419fcb

updating docs

8213010

gizas enabled auto-merge (squash) September 18, 2024 13:53

gizas merged commit 89cd840 into main Sep 18, 2024
128 of 131 checks passed

gizas deleted the kubeadm_filebeat branch September 18, 2024 15:18

mergify bot mentioned this pull request Sep 18, 2024

[8.x](backport #40301) Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40889

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301

Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301

gizas commented Jul 19, 2024 •

edited

Loading

mergify bot commented Jul 19, 2024

cla-checker-service bot commented Sep 10, 2024 •

edited

Loading

gizas Sep 10, 2024

mergify bot commented Sep 11, 2024

Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301

Disable kubeadm for filebeat (both filbeat.input and filbeat.autodiscover) #40301

Conversation

gizas commented Jul 19, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

mergify bot commented Jul 19, 2024

cla-checker-service bot commented Sep 10, 2024 • edited Loading

gizas Sep 10, 2024

Choose a reason for hiding this comment

mergify bot commented Sep 11, 2024

gizas commented Jul 19, 2024 •

edited

Loading

cla-checker-service bot commented Sep 10, 2024 •

edited

Loading