[RFC] Email - Stage 1 Proposal #1219
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -1,68 +1,54 @@ | ||||||||
| # 0010: Email | ||||||||
| <!-- Leave this ID at 0000. The ECS team will assign a unique, contiguous RFC number upon merging the initial stage of this RFC. --> | ||||||||
|
|
||||||||
| - Stage: **1 (draft)** <!-- Update to reflect target stage. See https://elastic.github.io/ecs/stages.html --> | ||||||||
| - Date: **2021-08-16** <!-- The ECS team sets this date at merge time. This is the date of the latest stage advancement. --> | ||||||||
|
|
||||||||
| This RFC proposes a new top-level field set to facilitate email use cases, `email.*`. The `email.*` field set adds fields for the sender, recipient, message header fields, and other attributes of an email message typically seen logs produced by mail transfer agent (MTA) and email gateway applications. | ||||||||
|
|
||||||||
| ## Fields | ||||||||
|
|
||||||||
| <!-- | ||||||||
| Stage 1: Describe at a high level how this change affects fields. Which fieldsets will be impacted? How many fields overall? Are we primarily adding fields, removing fields, or changing existing fields? The goal here is to understand the fundamental technical implications and likely extent of these changes. ~2-5 sentences. | ||||||||
| --> | ||||||||
|
|
||||||||
| ### Email specific fields | ||||||||
|
|
||||||||
| | field | type | description | | ||||||||
| | --- | --- | --- | | ||||||||
| | `email.from` | keyword | Stores the `from` email address from the RFC5322 `From:` header field. | | ||||||||
| | `email.origination_timestamp` | date | The date and time the email message was composed. Many email clients will fill this in automatically when the message is sent by a user. | | ||||||||
| | `email.delivery_timestamp` | date | The date and time the email message was received by the service or client. | | ||||||||
| | `email.to` | keyword (array) | The email address(es) of the message recipient(s) | | ||||||||
| | `email.subject` | keyword; `.text` text multi-field | A brief summary of the topic of the message | | ||||||||
| | `email.cc` | keyword (array) | The email address(es) of the carbon copy (CC) recipient(s) | | ||||||||
| | `email.bcc` | keyword (array) | The email address(es) of the blind carbon copy (CC) recipient(s) | | ||||||||
| | `email.content_type` | keyword | Information about how the message is to be displayed. Typically a MIME type | | ||||||||
| | `email.message_id` | wildcard | Identifier from the RFC5322 `Message-ID:` header field that refers to a particular version of a particular message. | | ||||||||
| | `email.local_id` | keyword | Unique identifier given to the email by the source (MTA, gateway, etc.) that created the event and is not persistent across hops (for example, the `X-MS-Exchange-Organization-Network-Message-Id` id). | | ||||||||
| | `email.reply_to` | keyword | The address that replies should be delivered to from the RFC 5322 `Reply-To:` header field. | | ||||||||
| | `email.direction` | keyword | Direction of the message based on the sending and receiving domains | | ||||||||
| | `email.x_mailer` | keyword | What application was used to draft and send the original email. | | ||||||||
| | `email.attachments` | nested | Nested object of attachments on the email. | | ||||||||
| | `email.attachments.file.mime_type` | keyword | MIME type of the attachment file. | | ||||||||
| | `email.attachments.file.name` | keyword | Name of the attachment file including the extension. | | ||||||||
| | `email.attachments.file.extension` | keyword | Attachment file extension, excluding the leading dot. | | ||||||||
| | `email.attachments.file.size` | long | Attachment file size in bytes. | | ||||||||
| | `email.attachments.hash.md5` | keyword | MD5 hash of the file attachment. | | ||||||||
| | `email.attachments.hash.sha1` | keyword | SHA-1 hash of the file attachment. | | ||||||||
| | `email.attachments.hash.sha256` | keyword | SHA-256 hash of the file attachment. | | ||||||||
|
|
||||||||
| ### Additional event categorization allowed values | ||||||||
|
There was a problem hiding this comment. Should we provide a field for attachments, attachment names and MIME types? There was a problem hiding this comment. I agree we should include fields for emails attachments eventually. Still, I'm undecided whether to have them or not in this initial round of the If folks feel there's a strong case for including them from the start, I'm open to adding. Part of my hesitation - I haven't thought of a great option for multiple attachments in a single document without resorting to an array of objects and using a There was a problem hiding this comment. Can we consider adding another ID field, for example There was a problem hiding this comment. That sounds like a good idea, @ericwbentley Could you use the "insert suggestion" option to share how you think that'd need to be structured? There was a problem hiding this comment. Without using language too specific towards one product or another I think this covers what the field would be used for
Suggested change
There was a problem hiding this comment. @ericwbentley @peasead I had added I adjust the description a bit: 6368af1. Let me know if you have any feedback. |
||||||||
|
|
||||||||
| Email events may benefit from an additional ECS allowed event categorization value: `event.category: email`. | ||||||||
|
|
||||||||
| ## Usage | ||||||||
|
|
||||||||
| <!-- | ||||||||
| Stage 1: Describe at a high-level how these field changes will be used in practice. Real world examples are encouraged. The goal here is to understand how people would leverage these fields to gain insights or solve problems. ~1-3 paragraphs. | ||||||||
| --> | ||||||||
|
|
||||||||
| Email use cases stretch across all three Elastic solutions - Search, Observe, Protect. Whether it's searching for content within email, ensuring email infrastructure is operational or detecting email based attacks, there are many possibilities for email fields within ECS. | ||||||||
|
|
||||||||
| ## Source data | ||||||||
|
|
||||||||
|
|
@@ -78,6 +64,199 @@ Stage 1: Provide a high-level description of example sources of data. This does | |||||||
| Stage 2: Included a real world example source document. Ideally this example comes from the source(s) identified in stage 1. If not, it should replace them. The goal here is to validate the utility of these field changes in the context of a real world example. Format with the source name as a ### header and the example document in a GitHub code block with json formatting. | ||||||||
| --> | ||||||||
|
|
||||||||
| ### Office365 - Successful Delivery | ||||||||
|
|
||||||||
| #### Original log | ||||||||
|
|
||||||||
| ```json | ||||||||
| { | ||||||||
| "EndDate": "2020-11-10T22:12:34.8196921Z", | ||||||||
| "FromIP": "8.8.8.8", | ||||||||
| "Index": 25, | ||||||||
| "MessageId": "\\u003c95689d8d5e7f429390a4e3646eef75e8-JFBVALKQOJXWILKBK4YVA7APGM3DKTLFONZWCZ3FINSW45DFOJ6EAQ2ENFTWK43UL4YTCMBYGIYHYU3NORYA====@microsoft.com\\u003e", | ||||||||
| "MessageTraceId": "ff1a64a3-cafb-41b7-1efb-08d8848aedc3", | ||||||||
| "Organization": "testdomain.onmicrosoft.com", | ||||||||
| "Received": "2020-11-09T04:50:06.3312635", | ||||||||
| "RecipientAddress": "john@testdomain.onmicrosoft.com", | ||||||||
| "SenderAddress": "o365mc@microsoft.com", | ||||||||
| "Size": 64329, | ||||||||
| "StartDate": "2020-11-08T22:12:34.8196921Z", | ||||||||
| "Status": "Delivered", | ||||||||
| "Subject": "Weekly digest: Microsoft service updates", | ||||||||
| "ToIP": null | ||||||||
| } | ||||||||
| ``` | ||||||||
|
|
||||||||
| #### Mapped event | ||||||||
|
|
||||||||
| ```json | ||||||||
ebeahan marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||
| { | ||||||||
| "@timestamp": 1626984241830, | ||||||||
| "email": { | ||||||||
| "timestamp": "2020-11-08T22:12:34.8196921Z", | ||||||||
| "from": [ | ||||||||
| "o365mc@microsoft.com" | ||||||||
| ], | ||||||||
| "to": [ | ||||||||
| "john@testdomain.onmicrosoft.com" | ||||||||
| ], | ||||||||
| "subject": "Weekly digest: Microsoft service updates", | ||||||||
| "message_id": "\\u003c95689d8d5e7f429390a4e3646eef75e8-JFBVALKQOJXWILKBK4YVA7APGM3DKTLFONZWCZ3FINSW45DFOJ6EAQ2ENFTWK43UL4YTCMBYGIYHYU3NORYA====@microsoft.com\\u003e" | ||||||||
| }, | ||||||||
| "event": { | ||||||||
| "action": "delivered", | ||||||||
| "kind": "event", | ||||||||
| "category": [ | ||||||||
| "email", | ||||||||
| "network" | ||||||||
| ] | ||||||||
| } | ||||||||
| } | ||||||||
| ``` | ||||||||
|
|
||||||||
| ### Office365 - Undeliverable | ||||||||
|
|
||||||||
| #### Original log | ||||||||
|
|
||||||||
| ```json | ||||||||
| { | ||||||||
| "EndDate": "2020-11-10T22:12:34.8196921Z", | ||||||||
| "FromIP": null, | ||||||||
| "Index": 8, | ||||||||
| "MessageId": "\\u003c72872e16-f4c2-4eef-a393-e5621748a0ff@AS8P19vMB1605.EURP191.PROD.OUTLOOK.COM\\u003e", | ||||||||
| "MessageTraceId": "a4bd8c4c-3a4f-427f-8952-08d8850f9c20", | ||||||||
| "Organization": "testdomain.onmicrosoft.com", | ||||||||
| "Received": "2020-11-10T00:28:56.3306834", | ||||||||
| "RecipientAddress": "o365mc@microsoft.com", | ||||||||
| "SenderAddress": "postmaster@testdomain.onmicrosoft.com", | ||||||||
| "Size": 96627, | ||||||||
| "StartDate": "2020-11-08T22:12:34.8196921Z", | ||||||||
| "Status": "Delivered", | ||||||||
| "Subject": "Undeliverable: Message Center Major Change Update Notification", | ||||||||
| "ToIP": "8.8.8.8" | ||||||||
| } | ||||||||
| ``` | ||||||||
|
|
||||||||
| #### Mapped event | ||||||||
|
|
||||||||
| ```json | ||||||||
| { | ||||||||
| "@timestamp": 1626984241830, | ||||||||
| "email": { | ||||||||
| "timestamp": "2020-11-10T22:12:34.8196921Z", | ||||||||
| "from": [ | ||||||||
| "postmaster@testdomain.onmicrosoft.com" | ||||||||
| ], | ||||||||
| "to": [ | ||||||||
| "o365mc@microsoft.com" | ||||||||
| ], | ||||||||
| "subject": "Undeliverable: Message Center Major Change Update Notification", | ||||||||
| "message_id": "\\u003c72872e16-f4c2-4eef-a393-e5621748a0ff@AS8P19vMB1605.EURP191.PROD.OUTLOOK.COM\\u003e" | ||||||||
| }, | ||||||||
| "event": { | ||||||||
| "action": "delivered", | ||||||||
| "kind": "event", | ||||||||
| "category": [ | ||||||||
| "email", | ||||||||
| "network" | ||||||||
| ] | ||||||||
| } | ||||||||
| } | ||||||||
| ``` | ||||||||
|
|
||||||||
| ### Proofpoint Tap | ||||||||
|
|
||||||||
| #### Original log | ||||||||
|
|
||||||||
| ``` | ||||||||
| <38>1 2016-06-24T21:00:08Z - ProofpointTAP - MSGBLK [tapmsg@21139 messageTime="2016-06-24T21:18:38.000Z" messageID="20160624211145.62086.mail@evil.zz" recipient="clark.kent@pharmtech.zz, diana.prince@pharmtech.zz" sender="e99d7ed5580193f36a51f597bc2c0210@evil.zz" senderIP="192.0.2.255" phishScore="46" spamScore="4" QID="r2FNwRHF004109" GUID="c26dbea0-80d5-463b-b93c-4e8b708219ce" subject="Please find a totally safe invoice attached." quarantineRule="module.sandbox.threat" quarantineFolder="Attachment Defense" policyRoutes="default_inbound,executives" modulesRun="sandbox,urldefense,spam,pdr" headerFrom="\"A. Badguy\" <badguy@evil.zz>" headerTo="\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>" headerCC="\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>" headerReplyTo="null" toAddresses="clark.kent@pharmtech.zz,diana.prince@pharmtech.zz" ccAddresses="bruce.wayne@university-of-education.zz" fromAddress="badguy@evil.zz" replyToAddress="null" clusterId="pharmtech_hosted" messageParts="[{\"contentType\":\"text/plain\",\"disposition\":\"inline\",\"filename\":\"text.txt\",\"md5\":\"008c5926ca861023c1d2a36653fd88e2\",\"oContentType\":\"text/plain\",\"sandboxStatus\":\"unsupported\",\"sha256\":\"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281\"},{\"contentType\":\"application/pdf\",\"disposition\":\"attached\",\"filename\":\"Invoice for Pharmtech.pdf\",\"md5\":\"5873c7d37608e0d49bcaa6f32b6c731f\",\"oContentType\":\"application/pdf\",\"sandboxStatus\":\"threat\",\"sha256\":\"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca\"}]" xmailer="Spambot v2.5"] | ||||||||
| ``` | ||||||||
|
|
||||||||
| #### Mapped event | ||||||||
|
|
||||||||
| ```json | ||||||||
| { | ||||||||
| "@timestamp": "2016-06-24T21:00:08Z", | ||||||||
| "email": { | ||||||||
| "timestamp": "2016-06-24T21:18:38.000Z", | ||||||||
| "message_id": "20160624211145.62086.mail@evil.zz", | ||||||||
| "local_id": "c26dbea0-80d5-463b-b93c-4e8b708219ce", | ||||||||
| "to": [ | ||||||||
| "clark.kent@pharmtech.zz", | ||||||||
| "diana.prince@pharmtech.zz" | ||||||||
| ], | ||||||||
| "cc": [ | ||||||||
| "bruce.wayne@university-of-education.zz" | ||||||||
| ], | ||||||||
| "from": [ | ||||||||
| "badguy@evil.zz" | ||||||||
| ], | ||||||||
| "subject": "Please find a totally safe invoice attached.", | ||||||||
| "reply_to": "null", | ||||||||
| "x_mailer": "Spambot v2.5", | ||||||||
| "attachments": [ | ||||||||
| { | ||||||||
| "file": { | ||||||||
| "mime_type": "application/pdf", | ||||||||
| "name": "Invoice for Pharmtech.pdf", | ||||||||
| "extension": "pdf" | ||||||||
| }, | ||||||||
| "hash": { | ||||||||
| "md5": "5873c7d37608e0d49bcaa6f32b6c731f", | ||||||||
| "sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca" | ||||||||
| } | ||||||||
| } | ||||||||
| ] | ||||||||
| }, | ||||||||
| "event": { | ||||||||
| "id": "c26dbea0-80d5-463b-b93c-4e8b708219ce", | ||||||||
| "kind": "event", | ||||||||
| "category": "email", | ||||||||
| "action": "MSGBLK" | ||||||||
| }, | ||||||||
| "source": { | ||||||||
| "address": 192.0.2.255, | ||||||||
| "ip": 192.0.2.255 | ||||||||
| } | ||||||||
| } | ||||||||
| ``` | ||||||||
|
|
||||||||
| ### Mimecast Receipt log | ||||||||
|
|
||||||||
| #### Original log | ||||||||
|
|
||||||||
| ``` | ||||||||
| datetime=2017-05-26T16:47:41+0100|aCode=7O7I7MvGP1mj8plHRDuHEA|acc=C0A0|SpamLimit=0|IP=123.123.123.123|Dir=Internal|MsgId=<81ce15$8r2j59@mail01.example.com>|Subject=\message subject\|headerFrom=from@mimecast.com|Sender=from@mimecast.com|Rcpt=auser@mimecast.com|SpamInfo=[]|Act=Acc|TlsVer=TLSv1|Cphr=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|SpamProcessingDetail={"spf":{"info":"SPF_FAIL","allow":true},"dkim":{"info":"DKIM_UNKNOWN","allow":true}}|SpamScore=1 | ||||||||
| ``` | ||||||||
|
|
||||||||
| #### Mapped event | ||||||||
|
|
||||||||
| ```json | ||||||||
| { | ||||||||
| "@timestamp": "2017-05-26T16:47:41+0100", | ||||||||
| "source": { | ||||||||
| "address": 123.123.123.123, | ||||||||
| "ip": 123.123.123.123 | ||||||||
| }, | ||||||||
| "email": { | ||||||||
| "message_id": "<81ce15$8r2j59@mail01.example.com>", | ||||||||
| "from": [ | ||||||||
| "from@mimecast.com" | ||||||||
| ], | ||||||||
| "to": [ | ||||||||
| "auser@mimecast.com" | ||||||||
| ], | ||||||||
| "subject": "message subject", | ||||||||
| "direction": "internal" | ||||||||
| }, | ||||||||
| "tls": { | ||||||||
| "cipher": "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", | ||||||||
| "version": "1.0", | ||||||||
| "version_protocol": "tls" | ||||||||
| } | ||||||||
| } | ||||||||
| ``` | ||||||||
|
|
||||||||
| <!-- | ||||||||
| Stage 3: Add more real world example source documents so we have at least 2 total, but ideally 3. Format as described in stage 2. | ||||||||
| --> | ||||||||
|
|
@@ -97,37 +276,46 @@ The goal here is to research and understand the impact of these changes on users | |||||||
| <!-- | ||||||||
| Stage 1: Identify potential concerns, implementation challenges, or complexity. Spend some time on this. Play devil's advocate. Try to identify the sort of non-obvious challenges that tend to surface later. The goal here is to surface risks early, allow everyone the time to work through them, and ultimately document resolution for posterity's sake. | ||||||||
| --> | ||||||||
|
|
||||||||
| ### Email messages vs. protocols | ||||||||
|
|
||||||||
| The fields proposed in this document are focused on the contents of an email message but not on specific fields for email protocols. Do protocols like SMTP, POP3, IMAP, etc. be represented in ECS? | ||||||||
|
|
||||||||
| For example, users may need to compare the email address from the SMTP (envelope) sender to the `From:` header email address. | ||||||||
|
|
||||||||
| ### Email metrics and observability use caes | ||||||||
|
|
||||||||
| Does the initial set of `email` fields need to consider observability and email monitoring use cases, for example spam, metrics, deliverables, and logging. | ||||||||
|
|
||||||||
| ### Additional event categorization values | ||||||||
|
|
||||||||
| Should a new event.category field (email) be created, and, if so, which `event.type` values the `email` category should be combined with? | ||||||||
|
|
||||||||
| ### Display names | ||||||||
|
|
||||||||
| Should the display name be captured separately from the email address for senders and recipients. If so, how do we accomplish this in a document while keeping the 1:1 of a display name to email address. | ||||||||
|
|
||||||||
| ### Spam processing details | ||||||||
|
|
||||||||
| Should fields intended to capture details around spam processing like sender policy framework (SPF), domainkeys identified mail (DKIM), or domain-based message authentication, reporting, and conformance (DMARC) be in scope for this proposal as well? | ||||||||
|
|
||||||||
| <!-- | ||||||||
| Stage 2: Document new concerns or resolutions to previously listed concerns. It's not critical that all concerns have resolutions at this point, but it would be helpful if resolutions were taking shape for the most significant concerns. | ||||||||
| --> | ||||||||
|
|
||||||||
| <!-- | ||||||||
| Stage 3: Document resolutions for all existing concerns. Any new concerns should be documented along with their resolution. The goal here is to eliminate the risk of churn and instability by resolving outstanding concerns. | ||||||||
| --> | ||||||||
|
|
||||||||
| ## People | ||||||||
|
|
||||||||
| The following are the people that consulted on the contents of this RFC. | ||||||||
|
|
||||||||
| * @ebeahan | Co-author | ||||||||
| * @P1llus | Co-author, subject matter expert | ||||||||
| * @jamiehynds | Co-sponsor | ||||||||
| * @devonakerr | Co-sponsor | ||||||||
|
|
||||||||
|
|
||||||||
| <!-- | ||||||||
| Who will be or has been consulted on the contents of this RFC? Identify authorship and sponsorship, and optionally identify the nature of involvement of others. Link to GitHub aliases where possible. This list will likely change or grow stage after stage. | ||||||||
|
|
@@ -158,5 +346,6 @@ e.g.: | |||||||
|
|
||||||||
| <!-- An RFC should link to the PRs for each of it stage advancements. --> | ||||||||
|
|
||||||||
| * Stage 1 (formerly proposal stage): https://github.com/elastic/ecs/pull/999 | ||||||||
| * RFC ID correction: https://github.com/elastic/ecs/pull/1157 | ||||||||
| * Stage 1 (draft): https://github.com/elastic/ecs/pull/1219 | ||||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added a nested object at
email.attachmentswith each object having these fields available for each attachment:file.mime_typefile.namefile.extensionfile.sizehash.md5hash.sha1hash.sha256I definitely can expand this list if there are other fields we should account for. I don't think completely nesting
file.*atemail.attachments.*is necessary, but I did try to match the naming to the existingfile.*field set for consistency.