Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[7.x] [Security Solutions][Detection Engine] Adds threat matching API and rule type (#77395) #77978

Merged
merged 1 commit into from
Sep 20, 2020
Merged

[7.x] [Security Solutions][Detection Engine] Adds threat matching API and rule type (#77395) #77978

merged 1 commit into from
Sep 20, 2020

Conversation

FrankHassanabad
Copy link
Contributor

Backports the following commits to 7.x:

@FrankHassanabad
[Security Solutions][Detection Engine] Adds threat matching API and r…
2c7f889 
…ule type (elastic#77395)

## Summary

This is the backend, first iteration of threat matching API and rule type. You see elements using the backend API on the front end but cannot use the UI to add or edit a threshold rule with this PR.

Screen shots of it running in the UI elements that do work:
<img width="1862" alt="Screen Shot 2020-09-16 at 10 34 26 AM" src="https://user-images.githubusercontent.com/1151048/93366465-6e2b9c00-f808-11ea-923b-78e8d0fdfbaa.png">

<img width="1863" alt="Screen Shot 2020-09-16 at 10 34 48 AM" src="https://user-images.githubusercontent.com/1151048/93366476-71268c80-f808-11ea-8247-d2091ff1599a.png"> 

**Usage**
Since this is only backend API work and does not have the front end add/edit at the moment, you can use the existing UI's (for the most part) to validate the work here through CURL scripts below:

Go to the folder:
```ts
/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts
```

And post a small ECS threat mapping to the index called `mock-threat-list`:
```ts
./create_threat_mapping.sh
```

Then to post a small number of threats that represent simple port numbers you can run:
```ts
./create_threat_data.sh
```

However, feel free to also manually create them directly in your dev tools like so:

```ts
# Posts a threat list item called some-name with an IP but change these out for valid data in your system
PUT mock-threat-list-1/_doc/9999
{
  "@timestamp": "2020-09-09T20:30:45.725Z",
  "host": {
    "name": "some-name",
    "ip": "127.0.0.1"
  }
}
```

```ts
# Posts a destination port number to watch
PUT mock-threat-list-1/_doc/10000
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "destination": {
    "port": "443"
  }
}
```

```ts
# Posts a source port number to watch
PUT mock-threat-list-1/_doc/10001
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "source": {
    "port": "443"
  }
}
```

Then you can post a threat match rule:
```ts
./post_rule.sh ./rules/queries/query_with_threat_mapping.json
```
<details>
 <summary>Click here to see Response</summary>

```ts
{
  "actions": [],
  "author": [],
  "created_at": "2020-09-16T04:25:58.041Z",
  "created_by": "yo",
  "description": "Query with a threat mapping",
  "enabled": true,
  "exceptions_list": [],
  "false_positives": [],
  "from": "now-6m",
  "id": "f4226ab0-6f88-49c3-8f09-84cf5946ee7a",
  "immutable": false,
  "interval": "5m",
  "language": "kuery",
  "max_signals": 100,
  "name": "Query with a threat mapping",
  "output_index": ".siem-signals-hassanabad3-default",
  "query": "*:*",
  "references": [],
  "risk_score": 1,
  "risk_score_mapping": [],
  "rule_id": "threat-mapping",
  "severity": "high",
  "severity_mapping": [],
  "tags": [
    "tag_1",
    "tag_2"
  ],
  "threat": [],
  "threat_index": "mock-threat-list-1",
  "threat_mapping": [
    {
      "entries": [
        {
          "field": "host.name",
          "type": "mapping",
          "value": "host.name"
        },
        {
          "field": "host.ip",
          "type": "mapping",
          "value": "host.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "destination.ip"
        },
        {
          "field": "destination.port",
          "type": "mapping",
          "value": "destination.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.port",
          "type": "mapping",
          "value": "source.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "source.ip"
        }
      ]
    }
  ],
  "threat_query": "*:*",
  "throttle": "no_actions",
  "to": "now",
  "type": "threat_match",
  "updated_at": "2020-09-16T04:25:58.051Z",
  "updated_by": "yo",
  "version": 1
}
```
</details>

**Structure**

You can see the rule structure in the file:
```ts
x-pack/plugins/security_solution/server/lib/detection_engine/scripts/rules/queries/query_with_threat_mapping.json
```
<details>
 <summary>Click here to see JSON</summary>

```ts
{
  "name": "Query with a threat mapping",
  "description": "Query with a threat mapping",
  "rule_id": "threat-mapping",
  "risk_score": 1,
  "severity": "high",
  "type": "threat_match",
  "query": "*:*",
  "tags": ["tag_1", "tag_2"],
  "threat_index": "mock-threat-list",
  "threat_query": "*:*",
  "threat_mapping": [
    {
      "entries": [
        {
          "field": "host.name",
          "type": "mapping",
          "value": "host.name"
        },
        {
          "field": "host.ip",
          "type": "mapping",
          "value": "host.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "destination.ip"
        },
        {
          "field": "destination.port",
          "type": "mapping",
          "value": "destination.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.port",
          "type": "mapping",
          "value": "source.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "source.ip"
        }
      ]
    }
  ]
}
```

</details>

Structural elements that are new:

New type enum called "threat_match"
```ts
"type": "threat_match",
```

New `threat_index` string which can be set to a single threat index (This might change to an array in the near future before release):
```ts
"threat_index": "mock-threat-list"
```

New `threat_query` string which can be set any valid query to filter the threat list before executing the rule. This can be undefined, if you are only pushing in filters from the API.

```ts
"threat_query": "*:*",
```

New `threat_filters` array which can be set to any valid filter like `filters`. This can be `undefined` if you are only using the query from the API.
```ts
threat_filter": []
```

New `threat_mapping` array which can be set to a valid mapping between the threat list and the ECS list. This structure has an inner array called `entries` which represent a 2 level tree of 1st level OR elements followed by 2nd level AND elements.

For example, if you want to find all threat matches where ECS documents will match against some ${threatList} index where it would be like so:

<details>
 <summary>Click here to see array from the boolean</summary>

```ts
"threat_mapping": [
    {
      "entries": [
        {
          "field": "host.name",
          "type": "mapping",
          "value": "host.name"
        },
        {
          "field": "host.ip",
          "type": "mapping",
          "value": "host.ip"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "destination.ip",
          "type": "mapping",
          "value": "destination.ip"
        },
        {
          "field": "destination.port",
          "type": "mapping",
          "value": "destination.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.port",
          "type": "mapping",
          "value": "source.port"
        }
      ]
    },
    {
      "entries": [
        {
          "field": "source.ip",
          "type": "mapping",
          "value": "source.ip"
        }
      ]
    }
  ]
```

</details>

What that array represents in pseudo boolean logic is: 

<details>
 <summary>Click here to see pseduo logic</summary>

```ts
(host.name: ${threatList.host.name} AND host.ip: ${threatList.host.name}) OR
(destination.ip: ${threatList.destination.ip} AND destination.port: ${threatList.destination.port}) OR
(source.port ${threatList.source.port}) OR
(source.ip ${threatList.source.ip})
```

</details>

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

async chunks size

id value diff baseline
securitySolution 10.2MB +508.0B 10.2MB

page load bundle size

id value diff baseline
securitySolution 793.4KB +18.0B 793.3KB

distributable file count

id value diff baseline
default 47662 +11 47651

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@FrankHassanabad FrankHassanabad merged commit e04438a into elastic:7.x Sep 20, 2020
@FrankHassanabad FrankHassanabad deleted the backport/7.x/pr-77395 branch September 20, 2020 22:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@FrankHassanabad @kibanamachine