[CWS] make use of the remote workloadmeta in the system-probe #28070

paulcacheux · 2024-07-30T16:57:59Z

What does this PR do?

This PR lays the foundation of being able to use the remote workloadmeta store in the system probe. This PR only allows the remote workloadmeta store to be used, since in most deployments the system probe cannot really maintain a working local store (cannot access the containerd socket, some configuration flags are only applied to the core agent etc).

Most of the changes from this PR are related to the transition from an optional workloadmeta component to a non-optional one.

The main motivations behind this change are:

being able to use the tagger component (instead of the legacy constructor for which we are the last users)
being able to drive the containers_running CWS telemetry in the system-probe instead of the security agent

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

cit-pr-commenter · 2024-07-30T17:04:20Z

Go Package Import Differences

Baseline: 33facf6
Comparison: e06bd03

binary

os

arch

change

system-probe

linux

amd64

+7, -0

+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/catalog
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/internal/remote
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/internal/remote/workloadmeta
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/fx
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/impl
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/proto
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/telemetry

system-probe

linux

arm64

+7, -0

+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/catalog
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/internal/remote
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/internal/remote/workloadmeta
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/fx
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/impl
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/proto
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/telemetry

system-probe

windows

amd64

+12, -0

+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/catalog
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/internal/remote
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/collectors/internal/remote/workloadmeta
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/fx
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/impl
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/proto
+github.com/DataDog/datadog-agent/comp/core/workloadmeta/telemetry
+github.com/DataDog/datadog-agent/pkg/util/flavor
+github.com/samber/lo
+github.com/samber/lo/internal/constraints
+github.com/samber/lo/internal/rand
+math/rand/v2

pr-commenter · 2024-07-30T18:54:33Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=43006925 --os-family=ubuntu

Note: This applies to commit e06bd03

pr-commenter · 2024-07-30T20:06:08Z

Regression Detector

Regression Detector Results

Run ID: 1bc095e7-64a4-43fc-a350-69f29116af9a Metrics dashboard Target profiles

Baseline: 33facf6
Comparison: e06bd03

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

Significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

perf	experiment	goal	Δ mean %	Δ mean % CI	links
❌	tcp_syslog_to_blackhole	ingress throughput	-12.91	[-24.87, -0.95]	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+1.45	[+0.49, +2.42]	Logs
➖	pycheck_lots_of_tags	% cpu utilization	+1.13	[-1.43, +3.70]	Logs
➖	file_tree	memory utilization	+1.01	[+0.95, +1.08]	Logs
➖	idle	memory utilization	+0.51	[+0.47, +0.54]	Logs
➖	basic_py_check	% cpu utilization	+0.49	[-2.18, +3.16]	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.00, +0.00]	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	Logs
➖	otel_to_otel_logs	ingress throughput	-0.91	[-1.71, -0.10]	Logs
❌	tcp_syslog_to_blackhole	ingress throughput	-12.91	[-24.87, -0.95]	Logs

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
✅	idle	memory_usage	10/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

tasks/security_agent.py

brycekahle · 2024-08-05T19:11:12Z

Does this mean the workloadmeta component will be created, even if not used? Is it only CWS that requires this component?

paulcacheux · 2024-08-06T18:17:51Z

@brycekahle yes.. sadly it means that it will always be there. But since it's a dependency of the event monitor (not just CWS), I think it might be fine since it's enabled by default for other major system probe products. I would be happy to hear a solution to load it only if need honestly, I just didn't found one.

My testing on system-probe, but also on the security agent side where it's already in use shows that the impact at runtime is really low.

L3n41c

👍🏻 LGTM for files owned by @DataDog/container-integrations

stanistan

LGTM for @DataDog/apm-onboarding

paulcacheux · 2024-08-28T08:01:16Z

/merge

dd-devflow · 2024-08-28T08:01:24Z

🚂 MergeQueue: pull request added to the queue

The median merge time in main is 21m.

Use /merge -c to cancel this operation!

dd-devflow · 2024-08-28T08:11:14Z

❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 25b8f70:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

If you need support, contact us on Slack #devflow with those details!

paulcacheux · 2024-08-28T09:19:36Z

/merge

dd-devflow · 2024-08-28T09:19:46Z

🚂 MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

dd-devflow · 2024-08-28T09:39:15Z

🚂 MergeQueue: pull request added to the queue

The median merge time in main is 21m.

Use /merge -c to cancel this operation!

github-actions bot added the component/system-probe label Jul 30, 2024

paulcacheux changed the title ~~Paulcacheux/wm sysprobe~~ [CWS] make use of the remote workloadmeta in the system-probe Jul 30, 2024

github-actions bot added the team/ebpf-platform label Jul 30, 2024

paulcacheux added the changelog/no-changelog label Jul 30, 2024

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch 4 times, most recently from c736171 to a6fc50b Compare July 30, 2024 18:15

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch 2 times, most recently from c61bc98 to 29092de Compare July 30, 2024 19:17

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch 2 times, most recently from 04645dc to 56b736b Compare August 2, 2024 12:58

paulcacheux added the team/agent-security label Aug 5, 2024

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch from 56b736b to 76bf59e Compare August 5, 2024 14:40

paulcacheux marked this pull request as ready for review August 5, 2024 14:40

paulcacheux requested review from a team as code owners August 5, 2024 14:40

spikat approved these changes Aug 5, 2024

View reviewed changes

FlorentClarret approved these changes Aug 5, 2024

View reviewed changes

tasks/security_agent.py Show resolved Hide resolved

daniel-taf approved these changes Aug 5, 2024

View reviewed changes

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch from 76bf59e to e3dc307 Compare August 6, 2024 18:16

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch 2 times, most recently from 35d0549 to 3cc346d Compare August 9, 2024 13:46

L3n41c approved these changes Aug 9, 2024

View reviewed changes

sblumenthal approved these changes Aug 9, 2024

View reviewed changes

stanistan approved these changes Aug 12, 2024

View reviewed changes

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch 2 times, most recently from 7b017a2 to 56bcfa7 Compare August 21, 2024 18:02

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch from 56bcfa7 to a8d764e Compare August 24, 2024 17:27

brycekahle approved these changes Aug 27, 2024

View reviewed changes

paulcacheux added 5 commits August 28, 2024 10:12

make workloadmeta non-optional

3e22241

fix test

7873943

provide fx config for workloadmeta

73dd71a

build system probe with remotewmonly tag

d66a153

fix more servicediscovery tests

2c5af03

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch from a8d764e to 2c5af03 Compare August 28, 2024 08:12

fix wm params passing

e06bd03

paulcacheux force-pushed the paulcacheux/wm-sysprobe branch from b277b9b to e06bd03 Compare August 28, 2024 09:07

dd-mergequeue bot merged commit c6fe8f4 into main Aug 28, 2024
287 of 311 checks passed

dd-mergequeue bot deleted the paulcacheux/wm-sysprobe branch August 28, 2024 10:14

github-actions bot added this to the 7.58.0 milestone Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CWS] make use of the remote workloadmeta in the system-probe #28070

[CWS] make use of the remote workloadmeta in the system-probe #28070

paulcacheux commented Jul 30, 2024 •

edited

Loading

cit-pr-commenter bot commented Jul 30, 2024 •

edited

Loading

pr-commenter bot commented Jul 30, 2024 •

edited

Loading

pr-commenter bot commented Jul 30, 2024 •

edited

Loading

Fine details of change detection per experiment

Explanation

brycekahle commented Aug 5, 2024

paulcacheux commented Aug 6, 2024

L3n41c left a comment

stanistan left a comment

paulcacheux commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

paulcacheux commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

[CWS] make use of the remote workloadmeta in the system-probe #28070

[CWS] make use of the remote workloadmeta in the system-probe #28070

Conversation

paulcacheux commented Jul 30, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

cit-pr-commenter bot commented Jul 30, 2024 • edited Loading

Go Package Import Differences

pr-commenter bot commented Jul 30, 2024 • edited Loading

Test changes on VM

pr-commenter bot commented Jul 30, 2024 • edited Loading

Regression Detector

Regression Detector Results

Significant changes in experiment optimization goals

Fine details of change detection per experiment

Bounds Checks

Explanation

brycekahle commented Aug 5, 2024

paulcacheux commented Aug 6, 2024

L3n41c left a comment

Choose a reason for hiding this comment

stanistan left a comment

Choose a reason for hiding this comment

paulcacheux commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

What to do next?

paulcacheux commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

dd-devflow bot commented Aug 28, 2024

paulcacheux commented Jul 30, 2024 •

edited

Loading

cit-pr-commenter bot commented Jul 30, 2024 •

edited

Loading

pr-commenter bot commented Jul 30, 2024 •

edited

Loading

pr-commenter bot commented Jul 30, 2024 •

edited

Loading