[Security Solution][Detection Engine] Adds threat matching to the rule creator

[FrankHassanabad]

Summary

This adds threat matching rule type to the rule creator.

Screen shot of creating a threat match

---

Screen shot of the description after creating one

---

Screen shot of first creating a threat match without values filled out

Additions and bug fixes:

• Changes the threat index to be an array
• Adds a threat_language to the REST schema so that we can use KQL, Lucene, (others in the future)
• Adds plumbing for threat_list to work with the other REST endpoints such as PUT, PATCH, etc...
• Adds the AND, OR dialog and user interface

Usage
If you are a team member using the team servers you can skip this usage section of creating threat index. Otherwise if you want to know how to create a mock threat index, instructions are below.

Go to the folder:

/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts

And post a small ECS threat mapping to the index called mock-threat-list:

./create_threat_mapping.sh

Then to post a small number of threats that represent simple port numbers you can run:

./create_threat_data.sh

However, feel free to also manually create them directly in your dev tools like so:

# Posts a threat list item called some-name with an IP but change these out for valid data in your system
PUT mock-threat-list-1/_doc/9999
{
  "@timestamp": "2020-09-09T20:30:45.725Z",
  "host": {
    "name": "some-name",
    "ip": "127.0.0.1"
  }
}

# Posts a destination port number to watch
PUT mock-threat-list-1/_doc/10000
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "destination": {
    "port": "443"
  }
}

# Posts a source port number to watch
PUT mock-threat-list-1/_doc/10001
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "source": {
    "port": "443"
  }
}

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
• This was checked for cross-browser compatibility

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[spong]

I'm thinking we should hide the other rule type cards when on the Edit Rule page since we can't currently switch rule types and it's a lot of real estate. This is the view when first clicking Edit Rule 😅

[spong]

Seeing the status get stuck in going to run when disabled:

And as we chatted about, looks like we're missing the running state (stays in going to run for duration of rule execution):

[yctercero]

Could this be made to be a non empty array? I just thought of that when I saw the containsEmptyItem helper. I remember having to do lots of checks like that with exceptions and finally just changed it so that it's required to not be empty since at the very least (in the UI) there's one item with empty values.

[FrankHassanabad]

Let me add that to a potential follow up. I totally agree, just don't know about the time I have left with the other tasks but I will add it to my list as I will have to make one of those icky specific types and all the tests for it.

[spong]

2️⃣ 1️⃣ ! 📈

[FrankHassanabad]

I was hoping you wouldn't see that one. :-) Yeah, someone needs to go in and refactor that. I kind of didn't want to add too much more additional things to this PR but we are accumulating more complexity debt and I left it as is for the next person. Probably me you know. hehe.

[spong]

Checked out, tested locally, and did a high level review of the code. So much goodness in this PR @FrankHassanabad!! Fantastic job exposing the backend threat matching features via this UI, and ++ for all the tests and re-use from the exceptions components.

Did quite a bit of testing and all error scenarios I could find are looking good. Good call on your last couple PR's focusing on bubbling up internal errors as they were extremely helpful in testing and watching where things fail when running out of resources. The one outstanding bug I was seeing was in reference to the last response getting stuck in going to run, but other than that most everything else looked 👍 and not worth holding up this PR.

May the Threat Matching begin! 🔥L🔥G🔥T🔥M🔥!🔥

[yctercero]

🚀 Focused on the code as I saw Garrett did a deep dive testing it. Had a few comments, but nothing to hold this up. This is really awesome and the code was so nice to read through!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 32783a3

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
securitySolution	1978	1989	+11

async chunks size

id	before	after	diff
securitySolution	10.3MB	10.3MB	+48.3KB

page load bundle size

id	before	after	diff
securitySolution	583.6KB	585.0KB	+1.3KB

History

• 💚 Build #78873 succeeded 65d8bc0
• 💚 Build #78577 succeeded 84f94db
• 💚 Build #78539 succeeded ea66ac8
• 💚 Build #78528 succeeded a4fc2ad
• 💚 Build #78525 succeeded 6f27124

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)