[RFC] Email - Stage 1 Proposal #1219

P1llus · 2021-01-12T12:01:45Z

Moving the Email RFC to Stage 1

Criteria for Stage 1

Opened pull request for this proposal revising the existing strawperson
Identified "sponsor" at Elastic who will participate in RFC process and take ownership of the change after the process completes
Outlined initial field definitions
High-level description of examples of usage
High-level description of example sources of data
Identified potential concerns and implementation challenges/complexity
Subject matter experts identified and weighed in on the high level utility of these changes in the pull request
ECS team weighed in on appropriateness of these changes in the pull request

ebeahan · 2021-01-27T21:27:21Z

We've recently removed Stage 4 from the RFC process.

Updated proposal stages and their requirements: https://elastic.github.io/ecs/stages.html

This PR was initially targeting the now "legacy" stage two, and I'm thinking we update to target stage 1. Stage 1 (draft) fields will still be added as experimental fields as was done with legacy stage 2 fields.

ebeahan

Thanks, @P1llus, for opening this. I listed some of the areas we want to go over and capture to move forward.

Field definitions

I suggest we replace any instance of wildcard with keyword since we've paused on introducing wildcard field support into ECS for now.

Can we create a directory using the RFC's number and add a YAML for the proposed set of field definitions? Here's an example: https://github.com/elastic/ecs/tree/master/rfcs/text/0009

Example data

We've added a couple of great examples from M365, and it'd be great to capture one or two more examples from other email data sources.

Concerns

Are there any new concerns that have come up to be captured? Or, do any of the existing concerns have updates or resolutions we can capture?

Feedback

Any individuals or teams we should ask for their feedback around the approach, fields, etc.?

rfcs/text/0010-email.md

ebeahan · 2021-05-28T22:23:59Z

ECS team will review and determine the next steps to continue moving this forward.

@jamiehynds, are you still willing to act as a sponsor?

jamiehynds · 2021-05-31T08:45:33Z

Thanks @ebeahan - yes, I'll continue to sponsor the RFC.

ericwbentley · 2021-07-29T22:31:57Z

rfcs/text/0010-email.md

+| `email.bcc` | keyword (array) | The email address(es) of the blind carbon copy (CC) recipient(s) |
+| `email.content_type` | keyword | Information about how the message is to be displayed. Typically a MIME type |
+| `email.message_id` | wildcard | Unique identifier for the email message that refers to a particular version of a particular message |
+| `email.reply_to` | keyword | Address that replies should be delivered to |


It should be clarified that this is referring to RFC5322.ReplyTo and is not the same as Return-Path or RFC5321.MailFrom

Thanks @ericwbentley

So, something like this:
| - | - | - |
| - | - | - |
| email.reply_to | keyword | Stores the email address provided in the "Reply-To" originator field |

Trying to improve and clarify this point in a856c0a. Let me know what you think.

ericwbentley · 2021-07-29T22:33:18Z

rfcs/text/0010-email.md

-| `tls.*` | Used for TLS related information for the connection to for example a SMTP server over TLS |
-
-
+| `email.from` | keyword | Stores the `from` email address |


It should be clarified that this is referring to the RFC5322.From or "Header From" address specifically

There should be another field for RFC5321.MailFrom "Envelope From" and/or Return-Path as RFC5321.MailFrom and RFC5322.From often differ

Thanks @ericwbentley

So, something like this:
| - | - | - |
| - | - | - |
| email.from | keyword | Stores the email address provided in the "From" originator field |

Reference: https://datatracker.ietf.org/doc/html/rfc5322#section-3.6.2

@ericwbentley Makes sense to reference RFC 5322 - thanks for suggesting the distinction.

Like I put in the Concerns section, right now, this proposal focuses on the IMF from RFC5322. However, I like the idea of also introducing an smtp field set later that focuses on details of the protocol (and other email protocols, if helpful).

I see smtp.* fields pairing well with email.*, but I want to avoid increasing the scope here too much. Open to feedback, though.

ericwbentley · 2021-07-29T22:38:14Z

rfcs/text/0010-email.md

+| `email.direction` | keyword | Direction of the message based on the sending and receiving domains |
+| `email.x_mailer` | keyword | What application was used to draft and send the original email.
+
+### Additional event categorization allowed values


Can we consider adding another ID field, for example email.local-id? Exchange uses it's own IDs in addition to message-id for example internal-message-id or network-message-id. Other email servers will have their own session IDs as well.

peasead · 2021-08-02T18:28:32Z

What happened to this E-mail fieldset? #999

I see it was merged, but I don't see it in master.

ebeahan · 2021-08-02T18:39:06Z

@peasead This picks up on the same work from #999 that @P1llus @jamiehynds started. @P1llus has been focused on other commitments, and I'm helping to move the proposal forward.

The field changes proposed in #999 aren't anywhere in the schema yet. Once we have a consensus and merge this PR, the agreed changes will be added to the experimental schema.

ebeahan · 2021-08-04T14:40:43Z

will a source ever be an array of values?

Corrected

are cc and bcc fields arrays and not keywords?

The cc and bcc are arrays of keyword values.

how will we account for attachments?

I added a nested attachment object and updated the Proofpoint TAP sample mapping to include an attachment. Let me know what you think.

ebeahan · 2021-08-13T20:13:32Z

Thanks, all, for the comments so far! I think I've addressed all the outstanding feedback.

Would appreciate additional looks to see if this is set for stage one.

@kgeller @devonakerr @peasead @P1llus @jamiehynds

peasead

I think this looks good for a Stage 1.

kgeller

LGTM 👍

ebeahan · 2021-08-17T19:03:14Z

I overlooked including field definition files for stage 1. I will include those with the stage 2 PR, but the proposed definitions are also now included in the experimental schema.

JamisonWhite · 2021-10-08T20:58:27Z

Hi @ebeahan,
I just discovered this today (and the ECS RFC process, so I apologize in advance if this is out of line), but I'm concerned that to, from, cc, and bcc are treated as emails. These values usually include the display name as well.
Thanks Jamie
https://datatracker.ietf.org/doc/html/rfc5322#section-3.4 (edit updated to latest RFC)

peasead · 2021-10-12T20:16:15Z

Hi @ebeahan, I just discovered this today (and the ECS RFC process, so I apologize in advance if this is out of line), but I'm concerned that to, from, cc, and bcc are treated as emails. These values usually include the display name as well. Thanks Jamie https://datatracker.ietf.org/doc/html/rfc5322#section-3.4 (edit updated to latest RFC)

Thanks, Jamie.

How would you like to see them reflected and where would do you think that the actual user@domain.tld would go for the "to", "from", "cc", and "bcc"?

Something like this?

email.to.address: user1@domain.tld
email.to.display_name: "User 1"
email.cc.address: user2@domain.tld
email.cc.display_name: "User 2"
...

JamisonWhite · 2021-10-12T22:03:32Z

@peasead It's definitely on the right track and works for "from". However "to", "cc", and "bcc" can be a list of addresses and names, so maybe it would look like the email.attachments nested objects. I'm not sure about the naming standards. "to_addresses" seems long, but "tos" or "toes" doesn't seem right either.

field	type	description
email.to_addresses	nested	nested object of addresses
email.to_addresses.address	keyword	email address
email.to_addresses.display_name	keyword	email display name
same for "cc" and "bcc"

wasserman · 2021-10-18T00:22:26Z

@JamisonWhite @peasead In email the to/cc/bcc is always a singular label regardless of having multiples. I would vote not to include the _addresses part since this would be understood and a bit redundant. Hopefully aggregations by email addresses will not be impacted by this structure.

JamisonWhite · 2021-10-21T15:02:18Z

Updated based on @wasserman feedback.

field	type	description
email.to	nested	nested object of addresses
email.to.address	keyword	email address
email.to.display_name	keyword	email display name
same for "cc" and "bcc"

ebeahan · 2021-11-17T23:00:58Z

Thanks, @wasserman @JamisonWhite, for the feedback!

I will incorporate the thinking from this discussion into the stage 2 PR #1593

ebeahan added the RFC label Jan 12, 2021

ebeahan reviewed Feb 5, 2021

View reviewed changes

rfcs/text/0010-email.md Outdated Show resolved Hide resolved

rfcs/text/0010-email.md Show resolved Hide resolved

epixa changed the title ~~Moving Email RFC to Stage 2~~ [RFC] Email - Stage 2 Proposal Apr 22, 2021

ebeahan changed the title ~~[RFC] Email - Stage 2 Proposal~~ [RFC] Email - Stage 1 Proposal May 28, 2021

ebeahan requested review from jamiehynds, devonakerr and a team July 27, 2021 17:19

ebeahan self-assigned this Jul 27, 2021

P1llus and others added 19 commits July 27, 2021 12:22

Moving Email RFC to Stage 2

e18940f

adjust stage to align with updated RFC stages

badcf5e

space

6b19ed5

replace wildcard with keyword

2261d58

update people

31dadd0

section headers

de5f68c

formatting of concerns

986ec0f

typo

2c84aba

reset date and adding co-sponsor

d1134ba

removing stage 4 sections

95d4411

trim down field list (at least initially)

b90fcdd

event.category: email

4cd5dd5

flesh out successful O365 example

daeaf9b

more mapping

069f3ce

add example field defs

e7849f4

event allowed value def for email

e16045a

improve message_id description

38a5e9c

add email.direction

ada6bec

clean up concerns section

e51a832

ericwbentley reviewed Jul 29, 2021

View reviewed changes

ebeahan added 9 commits August 3, 2021 15:45

rename and improve description of origination timestamp field

351fea8

add delivery_timestamp

5cb8bc0

reference rfc5322 from:

6ea47e7

adding external_id field

23b9895

attachment proposal

81e73ca

add file hash fields

2200bfa

remove attachments concern

f032bd1

it's a question

fc56be2

add smtp vs. imf use case

dad89f1

ebeahan added 2 commits August 4, 2021 12:24

improve reply_to description

a856c0a

opt to use local_id field naming

6368af1

Merge branch 'master' into master

f72fa4e

peasead approved these changes Aug 16, 2021

View reviewed changes

kgeller approved these changes Aug 16, 2021

View reviewed changes

ebeahan added 2 commits August 16, 2021 15:19

Merge branch 'master' into master

ecfdfaa

set date for stage 1

87e8122

ebeahan merged commit d56e181 into elastic:master Aug 16, 2021

ebeahan mentioned this pull request Aug 16, 2021

RFC 0010 Stage 1 - Add email.* fields to experimental schema #1569

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Email - Stage 1 Proposal #1219

[RFC] Email - Stage 1 Proposal #1219

P1llus commented Jan 12, 2021 •

edited by ebeahan

Loading

ebeahan commented Jan 27, 2021 •

edited

Loading

ebeahan left a comment

ebeahan commented May 28, 2021

jamiehynds commented May 31, 2021

ericwbentley Jul 29, 2021

peasead Aug 2, 2021

ericwbentley Aug 3, 2021 •

edited

Loading

ebeahan Aug 13, 2021

ericwbentley Jul 29, 2021

peasead Aug 2, 2021 •

edited

Loading

ericwbentley Aug 3, 2021

ebeahan Aug 3, 2021

ericwbentley Jul 29, 2021

peasead commented Aug 2, 2021

ebeahan commented Aug 2, 2021

ebeahan commented Aug 4, 2021

ebeahan commented Aug 13, 2021

peasead left a comment

kgeller left a comment

ebeahan commented Aug 17, 2021

JamisonWhite commented Oct 8, 2021 •

edited

Loading

peasead commented Oct 12, 2021

JamisonWhite commented Oct 12, 2021

wasserman commented Oct 18, 2021 •

edited

Loading

JamisonWhite commented Oct 21, 2021

ebeahan commented Nov 17, 2021

	\| `email.reply_to` \| keyword \| Address that replies should be delivered to \|
	\| `email.reply_to` \| keyword \| Address that replies should be delivered to (RFC5322.ReplyTo) \|

		\| `tls.*` \| Used for TLS related information for the connection to for example a SMTP server over TLS \|


		\| `email.from` \| keyword \| Stores the `from` email address \|

	\| `email.from` \| keyword \| Stores the `from` email address \|
	\| `email.from` \| keyword \| Stores the header `from` email address (RFC5322.From) \|
	\| `email.envelope_from` \| keyword \| Stores the envelope `from` email address (RFC5321.MailFrom) \|

[RFC] Email - Stage 1 Proposal #1219

[RFC] Email - Stage 1 Proposal #1219

Conversation

P1llus commented Jan 12, 2021 • edited by ebeahan Loading

ebeahan commented Jan 27, 2021 • edited Loading

ebeahan left a comment

Choose a reason for hiding this comment

Field definitions

Example data

Concerns

Feedback

ebeahan commented May 28, 2021

jamiehynds commented May 31, 2021

ericwbentley Jul 29, 2021

Choose a reason for hiding this comment

peasead Aug 2, 2021

Choose a reason for hiding this comment

ericwbentley Aug 3, 2021 • edited Loading

Choose a reason for hiding this comment

ebeahan Aug 13, 2021

Choose a reason for hiding this comment

ericwbentley Jul 29, 2021

Choose a reason for hiding this comment

peasead Aug 2, 2021 • edited Loading

Choose a reason for hiding this comment

ericwbentley Aug 3, 2021

Choose a reason for hiding this comment

ebeahan Aug 3, 2021

Choose a reason for hiding this comment

ericwbentley Jul 29, 2021

Choose a reason for hiding this comment

peasead commented Aug 2, 2021

ebeahan commented Aug 2, 2021

ebeahan commented Aug 4, 2021

ebeahan commented Aug 13, 2021

peasead left a comment

Choose a reason for hiding this comment

kgeller left a comment

Choose a reason for hiding this comment

ebeahan commented Aug 17, 2021

JamisonWhite commented Oct 8, 2021 • edited Loading

peasead commented Oct 12, 2021

JamisonWhite commented Oct 12, 2021

wasserman commented Oct 18, 2021 • edited Loading

JamisonWhite commented Oct 21, 2021

ebeahan commented Nov 17, 2021

P1llus commented Jan 12, 2021 •

edited by ebeahan

Loading

ebeahan commented Jan 27, 2021 •

edited

Loading

ericwbentley Aug 3, 2021 •

edited

Loading

peasead Aug 2, 2021 •

edited

Loading

JamisonWhite commented Oct 8, 2021 •

edited

Loading

wasserman commented Oct 18, 2021 •

edited

Loading