[Security Solution] Design and implement the API contract

[maximpn]

Epic: https://github.com/elastic/security-team/issues/2905 (internal)

Summary

Design and implement the API contract to cover the MVP functionality for Protections/Detections Coverage Overview feature.

Details

The popular way to represent coverage is using MITRE ATT&CK(TM) framework. It is a knowledge base of adversary tactics and techniques based on real-world observations. Currently it consists of 14 Tactics, 191 Techniques and 385 Sub-techniques. It is used by different teams and tools as a common reference.

Tactics represent an adversary tactical goal (eg Credential access), and in general can be viewed as an attack progression stages. Techniques represent how the attacker is achieving their goal.

We map our pre-built protections to ATT&CK tactics/techniques/sub-techniques where applicable. When creating custom rules, users can also map them to ATT&CK.

Protections/Detections Coverage Overview response API contract should cover the following items

• Provide a break down information about installed rules by ATT&CK techniques
• Each technique has to be broken down into sub-techniques
• Aggregated information (Enabled rules coverage, Potential coverage, Trend vs last week)

Request API should cover the following items

• Allow to filter rules by tactic, ATT&CK technique rule name, index pattern
• Allow to apply filters (by rule status, rule type, prebuilt rules vs custom, log source, index pattern)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[maximpn]

MITRE ATT&CK is a globally-accessible knowledge base of adversary tactics, techniques and sub-techniques based on real-world observations. It's not a tree like structure but rather a graph as one technique may belong to more than one tactic.

API contract design takes it into account and also based on the following principles

• UI already "knows" about MITRE ATT&CK tactics, techniques and sub-techniques so it's possible to render an empty grid
• Backend should return only necessary data but doesn't have to perform all the calculations as it can be done on the UI side and easily cached
• API response should be easily extensible
• It should be easy to a newer MITRE ATT&CK Framework's version (update the tactics and techniques lists)

Taking it into account request format is a simple combination of filters (the API endpoint returns all rules if the filter is not provided)

interface MitreCoverageRequest {
  filter?: MitreCoverageFilter;
}

interface MitreCoverageFilter {
  searchTerm?: string;
  status?: number; // a bit mask, any combination of RuleStatus values
  type?: number;  // a bit mask, any combination of RuleType values
}

enum RuleStatus {
  Enabled = 1 << 0, // 1
  Disabled = 1 << 1, // 2
  Available = 1 << 2, // 4
}

enum RuleType {
  Prebuilt = 1 << 0, // 1
  Custom = 1 << 1, // 2
  // Customized prebuilt rules
  Customized = 1 << 2, // 4
}

and the response format is a map with an additional map containing rule's data as it allows to reduce duplicated rules.

interface MitreCoverageResponse {
  // uses  tactic id (e.g. TA0009), technique id (e.g. T1531) and sub-technique id (e.g. T1596.003) as the key
  // and an array of rule SO's ids (not ruleId) as the value
  coverage: Record<string, string[]>;
  // uses rule SO's id (not ruleId) as the key
  rulesData: Record<string, Array<MitreCoverageRuleData | MitreCoverageAvailableRuleData>>;
}

interface MitreCoverageRuleData {
  name: string;
  enabled: boolean;
}

interface MitreCoverageAvailableRuleData {
  name: string;
  available: true;
}

A request example would look like

{
  "status": 3, // Only enabled and disabled rules RuleStatus.Enabled & RuleStatus.Disabled
}

A response example would look like

{
  "coverage": {
    "TA0009": ["889418e0-f325-11ed-af47-c3aaee15b7f5", "8cf5ec60-f325-11ed-af47-c3aaee15b7f5", "8f68e830-f325-11ed-af47-c3aaee15b7f5"],
    "T1596": ["8f68e830-f325-11ed-af47-c3aaee15b7f5"],
    "T1596.003": ["8f68e830-f325-11ed-af47-c3aaee15b7f5"]
  },
  "rulesData": {
    "889418e0-f325-11ed-af47-c3aaee15b7f5": {
      "name": "Multiple Alerts Involving a User",
      "enabled": true,
    },
    "889418e0-f325-11ed-af47-c3aaee15b7f5": {
      "name": "Unusual Linux Process Calling the Metadata Service",
      "enabled": false,
    },
    "889418e0-f325-11ed-af47-c3aaee15b7f5": {
      "name": "Unusual Windows User Calling the Metadata Service",
      "available": true,
    }
  }
}

This way UI should be able to build tactics to techniques relations graph based on the data in mitre_tactics_techniques.ts. Then fetch the MitreCoverageResponse, iterate over the graph and fill in the dashboard info and calculating the other metrics like the total number of enabled rules along the way. Having this graph filled in it should be straightforward to render the grid.

MITRE Coverage graph on the UI side may look the following way

interface MitreTactic {
  name: string;
  reference: string;
  techniques: MitreTechnique[];
  enabledRules: MitreRuleData[];
  disabledRules: MitreRuleData[];
  availableRules: MitreRuleData[];
}

interface MitreTechnique {
  name: string;
  reference: string;
  numOfCoveredSubtechniques: number;
  numOfSubtechniques: number;
  enabledRules: MitreRuleData[];
  disabledRules: MitreRuleData[];
  availableRules: MitreRuleData[];
}

type MitreRuleData = string; // rule SO's ids (not ruleId)

// or

interface MitreRuleData {
  id: string; // rule SO's ids (not ruleId)
  name: string;
}

[xcrzx]

Hey @maximpn, I'm curious about the reasoning behind using bitmaps. Bitmaps seem to add complexity to the API design.

Consider this request:

{
  "status": 3,
  "type": 5,
}

What do 3 and 5 mean there? For someone unfamiliar with the endpoint, it's impossible to decode their meanings without inspecting the implementation or referring to documentation (if available). Additionally, forming a request using a client such as Postman or Curl will require finding the field values and performing bit math to determine the correct number to pass, which complicates the use of the API further.

Contrast this with:

{
  "status": ['enabled', 'disabled'],
  "type": ['prebuilt', 'customized'],
}

This format explicitly states what is being sent to the backend and what the response should include. Explicit API parameters should be the preferred choice, as they reduce cognitive load for developers. This approach eliminates the need to parse or decode sent parameters and simplifies request formation, especially when parameter values match those in UI 1:1.

[banderror]

Awesome, the design LGTM in general, thanks for working it @maximpn! 👍 I just have a couple of comments.

First of all, I agree with @xcrzx about bitmaps. While bitmaps can be reasonable to use for in-memory calculations in statically typed languages like C# (because of a good DX) or for solving low-level or performance-sensitive problems (say communication between two stock exchange microservices), it seems to be an overkill and inferior DX for a FE-to-BE communication via an HTTP API. ++ to the suggested alternative:

{
  "status": ['enabled', 'disabled'],
  "type": ['prebuilt', 'customized'],
}

Secondly, let's try to come up with a better naming for RuleStatus and RuleType to disambiguate them from rule execution statuses and actual rule types. FWIW for the "type" I was thinking about the term "origin".

Also, let's not forget that our API contract and domain models exposed from the API should be underscore_cased.

Overall, the idea that the endpoint should return normalized and minimal data, and then on the FE side we denormalize it to something like that, sounds good to me:

interface MitreTactic {
  name: string;
  reference: string;
  techniques: MitreTechnique[];
  enabledRules: MitreRuleData[];
  disabledRules: MitreRuleData[];
  availableRules: MitreRuleData[];
}

interface MitreTechnique {
  name: string;
  reference: string;
  numOfCoveredSubtechniques: number;
  numOfSubtechniques: number;
  enabledRules: MitreRuleData[];
  disabledRules: MitreRuleData[];
  availableRules: MitreRuleData[];
}

@maximpn Please proceed with opening a PR with the implementation. In the PR we should have both the API contract and the FE-side domain model that @dplumlee could start using when he starts working on #158243

[maximpn]

@xcrzx and @banderror thank you for reviewing my initial API contract proposal. TBH I had doubts about using bit maps as it's definitely a trade off and there are a lot of another options though I like bitmaps simplicity and minimality. While a string array, for example, is too permissive and allows repeating values like

{
  "status": ['enabled', 'enabled', 'enabled'],
}

Anyway it's a solvable problem and a simpler option with better readability wins here so I agree string arrays is a good option for filters. I'll proceed with a PR.

[banderror]

Implemented in #159993