x-pack/filebeat/input/internal/private: add field redaction package

[efd6]

Proposed commit message

Initial donation of code from github.com/kortschak/dex/internal/private.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

diff --git a/x-pack/filebeat/input/httpjson/input.go b/x-pack/filebeat/input/httpjson/input.go
index 67daa7ef84..41b46c1878 100644
--- a/x-pack/filebeat/input/httpjson/input.go
+++ b/x-pack/filebeat/input/httpjson/input.go
@@ -33,6 +33,7 @@ import (
        "github.com/elastic/beats/v7/libbeat/version"
        "github.com/elastic/beats/v7/x-pack/filebeat/input/internal/httplog"
        "github.com/elastic/beats/v7/x-pack/filebeat/input/internal/httpmon"
+       "github.com/elastic/beats/v7/x-pack/filebeat/input/internal/private"
        "github.com/elastic/elastic-agent-libs/logp"
        "github.com/elastic/elastic-agent-libs/mapstr"
        "github.com/elastic/elastic-agent-libs/monitoring"
@@ -82,6 +83,19 @@ func (log *retryLogger) Warn(msg string, keysAndValues ...interface{}) {
        log.log.Warnw(msg, keysAndValues...)
 }
 
+type redact struct {
+       value  mapstr.M
+       fields []string
+}
+
+func (r redact) MarshalLogObject(enc zapcore.ObjectEncoder) error {
+       v, err := private.Redact(r.value, "", r.fields)
+       if err != nil {
+               return fmt.Errorf("could not redact value: %v", err)
+       }
+       return v.MarshalLogObject(enc)
+}
+
 func Plugin(log *logp.Logger, store inputcursor.StateStore) v2.Plugin {
        return v2.Plugin{
                Name:       inputName,
diff --git a/x-pack/filebeat/input/httpjson/request.go b/x-pack/filebeat/input/httpjson/request.go
index 6cc46b27f2..5dfb91dbdb 100644
--- a/x-pack/filebeat/input/httpjson/request.go
+++ b/x-pack/filebeat/input/httpjson/request.go
@@ -465,7 +465,7 @@ func (rf *requestFactory) newRequest(ctx *transformContext) (transformable, erro
                }
        }
 
-       rf.log.Debugf("new request: %#v", req)
+       rf.log.Debugw("new request", "req", redact{value: mapstr.M(req), fields: []string{"header.Authorization"}})
 
        return req, nil
 }

Could also be used in place of the current redactor in CEL, though that has a non-deletion redaction that this does not have. Adding that requires thought since this will redact non-string values.

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @efd6? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[kortschak]

Approving as copyright holder of the original code.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[kcreddy]

Overall LGTM.

[cmacknz]

FYI @michel-laterman in case this can help with the secrets redaction work you're doing in agent diagnostics.

[cmacknz]

Is there a reason this is in the x-pack/filebeat/input directory instead of say libbeat?

Or is this intended to be specific to an input that doesn't quite exist yet? On that note, it'd be helpful to have this used somewhere in Beats but I'm not opposed to merging this despite that.

[efd6]

@cmacknz This is intended to be used by the httpjson (and to replace the cel redact code). If it wants to be made more widely used, I'm ok with that, but I'd like to let it soak with httpjson and cel first if that's OK (I have some ideas to improve the ergonomics of the API and I'd like to have the freedom that internal gives to allow that work to happen more easily).

[michel-laterman]

Very interesting approach, I think we should get something in elastic-agent-libs to redact structs for us (in the future).

For context, we've added a secret_paths field that is injected into policies that fleet-server sends to the elastic-agent, these are paths that the agent should redact when logging. fleet-server pr.
The agent redaction logic (PR here) uses go-ucfg to replace strings.