From 5966ca4eef85f32ade12eb2b54ae8caf56109467 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Fri, 4 Dec 2020 16:43:24 -0800 Subject: [PATCH 01/12] first pass to disambiguate some points in the spec --- specs/agents/sanitization.md | 46 +++++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 7ca0ce85..05d92af9 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -1,12 +1,17 @@ +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", +"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to +be interpreted as described in +[RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). + ## Data sanitization ### `sanitize_field_names` configuration Sometimes it is necessary to sanitize, i.e., remove, sensitive data sent to Elastic APM. -This config accepts a list of wildcard patterns of field names which should be sanitized. -These apply to HTTP headers (including cookies) and `application/x-www-form-urlencoded` data (POST form fields). -The query string and the captured request body (such as `application/json` data) will not get sanitized. + +This config accepts a list of wildcard patterns of field names which should be +sanitized. | | | |----------------|---| @@ -14,3 +19,38 @@ The query string and the captured request body (such as `application/json` data) | Default | `password, passwd, pwd, secret, *key, *token*, *session*, *credit*, *card*, authorization, set-cookie` | | Dynamic | `true` | | Central config | `true` | + +## Configuration + +Agents MUST provide a minimum default configuration of + + [ 'password', 'passwd', 'pwd', 'secret', '*key', '*token*', '*session*', + '*credit*','*card*', 'authorization', 'set-cookie'] + +for the `sanitize_field_names` configuration value. Agent's MAY include the +following extra fields in their default configuration to avoid breaking changes + + ['pw','pass','connect.sid'] + +If an end-user configures different values, these values MUST **replace** the +above values. An end-user's configured values MUST NOT be merged with these +default values. + +## Sanitizing Values + +If a payload field's name (a header key, a form key) matches a configured +wildcard, that field's _value_ MUST be removed/redacted and the key itself +MUST still be reported in the agent payload. Agents MAY chose the string +they use to replace the value so long as it's consistent and does not reveal +the value it has replaced. Some example replacement strings +include `REDACTED`, `**********`, `-----------`, etc. + +Fields that MUST be sanitized are the HTTP Request headers, HTTP Response +headers, and form fields in a `application/x-www-form-urlencoded` request +body. No fields (including `set-cookie` headers) are exempt from this. + +The query string and other captured request bodies (such as `application/json`) +MUST NOT be sanitized. + +Agents MAY choose to further sanitize fields based on the _value_ of a +particular field, including the keys and values store in a cookie header. From 2f33854537bfc7cfcd4cfd386e5c55dc6df8d146 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Fri, 4 Dec 2020 16:48:19 -0800 Subject: [PATCH 02/12] fix: removing a should --- specs/agents/sanitization.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 05d92af9..19d9c226 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -10,8 +10,8 @@ be interpreted as described in Sometimes it is necessary to sanitize, i.e., remove, sensitive data sent to Elastic APM. -This config accepts a list of wildcard patterns of field names which should be -sanitized. +This config accepts a list of wildcard patterns of field names which control +how an agent will sanitize data. | | | |----------------|---| From f8b2b9fe974a10066ae7470dd2726554508d5a3d Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Fri, 4 Dec 2020 16:50:11 -0800 Subject: [PATCH 03/12] fix: grammar --- specs/agents/sanitization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 19d9c226..f6c79c27 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -46,7 +46,7 @@ the value it has replaced. Some example replacement strings include `REDACTED`, `**********`, `-----------`, etc. Fields that MUST be sanitized are the HTTP Request headers, HTTP Response -headers, and form fields in a `application/x-www-form-urlencoded` request +headers, and form fields in an `application/x-www-form-urlencoded` request body. No fields (including `set-cookie` headers) are exempt from this. The query string and other captured request bodies (such as `application/json`) From 18e31938e39ab81341a2e42fe6dbcea06a6cd26b Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Fri, 4 Dec 2020 16:51:36 -0800 Subject: [PATCH 04/12] fix: clarifying value sanitization --- specs/agents/sanitization.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index f6c79c27..aab9ed44 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -54,3 +54,5 @@ MUST NOT be sanitized. Agents MAY choose to further sanitize fields based on the _value_ of a particular field, including the keys and values store in a cookie header. +Agents SHOULD consider sanitization based on values to be a seperate +feature with its own configuration. From 498113879b09a8933da2326a844179c5aa0ab055 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Fri, 4 Dec 2020 16:52:17 -0800 Subject: [PATCH 05/12] fix: grammar --- specs/agents/sanitization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index aab9ed44..853e90fd 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -53,6 +53,6 @@ The query string and other captured request bodies (such as `application/json`) MUST NOT be sanitized. Agents MAY choose to further sanitize fields based on the _value_ of a -particular field, including the keys and values store in a cookie header. +particular field, including the keys and values stored in a cookie header. Agents SHOULD consider sanitization based on values to be a seperate feature with its own configuration. From 4ed30f4defc1345a492faf74c8211f8b3e7acb39 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 16:54:54 -0800 Subject: [PATCH 06/12] Update specs/agents/sanitization.md Co-authored-by: eyalkoren <41850454+eyalkoren@users.noreply.github.com> --- specs/agents/sanitization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 853e90fd..24d21dba 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -39,7 +39,7 @@ default values. ## Sanitizing Values If a payload field's name (a header key, a form key) matches a configured -wildcard, that field's _value_ MUST be removed/redacted and the key itself +wildcard, that field's _value_ MUST be redacted and the key itself MUST still be reported in the agent payload. Agents MAY chose the string they use to replace the value so long as it's consistent and does not reveal the value it has replaced. Some example replacement strings From 218ccb6edbbbafe5f5099ff54ed1d7c12fd9bba6 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 16:55:14 -0800 Subject: [PATCH 07/12] Update specs/agents/sanitization.md Co-authored-by: eyalkoren <41850454+eyalkoren@users.noreply.github.com> --- specs/agents/sanitization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 24d21dba..5662bd2d 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -40,7 +40,7 @@ default values. If a payload field's name (a header key, a form key) matches a configured wildcard, that field's _value_ MUST be redacted and the key itself -MUST still be reported in the agent payload. Agents MAY chose the string +MUST still be reported in the agent payload. Agents MAY choose the string they use to replace the value so long as it's consistent and does not reveal the value it has replaced. Some example replacement strings include `REDACTED`, `**********`, `-----------`, etc. From a14bd8abab2f8493e0f52df077b264b598566113 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 16:57:46 -0800 Subject: [PATCH 08/12] chore: change query string parameters from a MUST NOT to a SHOULD NOT --- specs/agents/sanitization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 5662bd2d..21fdd026 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -50,7 +50,7 @@ headers, and form fields in an `application/x-www-form-urlencoded` request body. No fields (including `set-cookie` headers) are exempt from this. The query string and other captured request bodies (such as `application/json`) -MUST NOT be sanitized. +SHOULD NOT be sanitized. Agents MAY choose to further sanitize fields based on the _value_ of a particular field, including the keys and values stored in a cookie header. From 7629a1ea128ab9689ca13ac0dded8157dcdffe70 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 17:09:21 -0800 Subject: [PATCH 09/12] chore: remove remove/merge clarification --- specs/agents/sanitization.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 21fdd026..50b79016 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -32,10 +32,6 @@ following extra fields in their default configuration to avoid breaking changes ['pw','pass','connect.sid'] -If an end-user configures different values, these values MUST **replace** the -above values. An end-user's configured values MUST NOT be merged with these -default values. - ## Sanitizing Values If a payload field's name (a header key, a form key) matches a configured From ff04c63e493813e1a43ba053b6462529b67c9983 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 17:16:03 -0800 Subject: [PATCH 10/12] fix: recommend a specific REDACTED string but still allow for others. --- specs/agents/sanitization.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 50b79016..0c6d5887 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -38,8 +38,7 @@ If a payload field's name (a header key, a form key) matches a configured wildcard, that field's _value_ MUST be redacted and the key itself MUST still be reported in the agent payload. Agents MAY choose the string they use to replace the value so long as it's consistent and does not reveal -the value it has replaced. Some example replacement strings -include `REDACTED`, `**********`, `-----------`, etc. +the value it has replaced. The replacement string SHOULD be `[REDACTED]`. Fields that MUST be sanitized are the HTTP Request headers, HTTP Response headers, and form fields in an `application/x-www-form-urlencoded` request From 997594f8ca69d6f076e2e4665a90b157e6f7d588 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 17:20:55 -0800 Subject: [PATCH 11/12] fix: h2 to h4 --- specs/agents/sanitization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index 0c6d5887..d93ab211 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -20,7 +20,7 @@ how an agent will sanitize data. | Dynamic | `true` | | Central config | `true` | -## Configuration +#### Configuration Agents MUST provide a minimum default configuration of From cf9732f40b991a1db9aba1dd9e5d6c3235e02fa3 Mon Sep 17 00:00:00 2001 From: Alan Storm Date: Tue, 15 Dec 2020 17:23:11 -0800 Subject: [PATCH 12/12] fix: clarify that we're moving away from value sanitization --- specs/agents/sanitization.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/specs/agents/sanitization.md b/specs/agents/sanitization.md index d93ab211..09bfdea7 100644 --- a/specs/agents/sanitization.md +++ b/specs/agents/sanitization.md @@ -47,7 +47,4 @@ body. No fields (including `set-cookie` headers) are exempt from this. The query string and other captured request bodies (such as `application/json`) SHOULD NOT be sanitized. -Agents MAY choose to further sanitize fields based on the _value_ of a -particular field, including the keys and values stored in a cookie header. -Agents SHOULD consider sanitization based on values to be a seperate -feature with its own configuration. +Agents SHOULD NOT sanitize fields based on the _value_ of a particular field. \ No newline at end of file