Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution][Alerts] Suppression rule fails after disabling and re-enabling #155242

Closed
vitaliidm opened this issue Apr 19, 2023 · 3 comments · Fixed by #155055
Closed

[Security Solution][Alerts] Suppression rule fails after disabling and re-enabling #155242

vitaliidm opened this issue Apr 19, 2023 · 3 comments · Fixed by #155055
Labels
bug Fixes for quality problems that affect the customer experience Team:Detection Alerts Security Detection Alerts Area Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@vitaliidm
Copy link
Contributor

vitaliidm commented Apr 19, 2023 •
edited
Loading

Describe the bug:
Suppression rule fails after disabling and re-enabling

Kibana/Elasticsearch Stack version:
8.7+

Steps to reproduce:

  1. Create custom query rule with suppression fields
  2. Ensure at least one alert will be suppressed
  3. Run rule, ensure suppressed alert is generated
  4. Disable rule, run rule again. It might require to do this few times. If it didn't work after first re-enabling. Add more data that would generate suppressed alert on the next rule run. Enable rule again
  5. Rule is failing

Screenshots (if relevant):
Screenshot 2023-04-19 at 11 56 10

Provide logs and/or server output (if relevant):

[2023-04-19T11:55:51.349+01:00][DEBUG][plugins.alerting.siem.queryRule] executing query for rule siem.queryRule:6d324ca0-dea0-11ed-bfe6-67c499fa8904 in space default - {"allow_no_indices":true,"runtime_mappings":{},"index":["suprression_1"],"size":0,"ignore_unavailable":true,"body":{"query":{"bool":{"filter":[{"bool":{"must":[],"filter":[{"query_string":{"query":"*"}}],"should":[],"must_not":[]}},{"range":{"@timestamp":{"lte":"2023-04-19T10:55:51.130Z","gte":"2023-03-08T13:55:51.130Z","format":"strict_date_optional_time"}}},{"bool":{"must_not":[{"bool":{"filter":[{"term":{"agent.name":"agent-0"}},{"range":{"@timestamp":{"lte":"2023-04-12T10:15:50.000Z","gte":"2023-03-08T13:55:51.130Z","format":"strict_date_optional_time"}}}]}},{"bool":{"filter":[{"term":{"agent.name":"agent-1"}},{"range":{"@timestamp":{"lte":"2023-04-12T09:18:50.000Z","gte":"2023-03-08T13:55:51.130Z","format":"strict_date_optional_time"}}}]}},{"bool":{"filter":[{"must_not":{"exists":{"field":"agent.name"}}},{"range":{"@timestamp":{"lte":"2023-04-15T09:52:50.000Z","gte":"2023-03-08T13:55:51.130Z","format":"strict_date_optional_time"}}}]}},{"bool":{"filter":[{"term":{"agent.name":"agent-1"}},{"range":{"@timestamp":{"lte":"2023-04-12T09:45:50.000Z","gte":"2023-03-08T13:55:51.130Z","format":"strict_date_optional_time"}}}]}}]}}]}},"fields":[{"field":"*","include_unmapped":true},{"field":"@timestamp","format":"strict_date_optional_time"}],"aggregations":{"eventGroups":{"composite":{"sources":[{"agent.name":{"terms":{"field":"agent.name","missing_bucket":true}}}],"size":100},"aggs":{"topHits":{"top_hits":{"size":1,"sort":[{"@timestamp":{"order":"asc","unmapped_type":"date"}}]}},"max_timestamp":{"max":{"field":"@timestamp"}},"min_timestamp":{"min":{"field":"@timestamp"}}}}},"runtime_mappings":{},"sort":[{"@timestamp":{"order":"asc","unmapped_type":"date"}}]}} - with options {"meta":true}
[2023-04-19T11:55:51.355+01:00][ERROR][plugins.securitySolution.ruleExecution] [-] nextSearchAfter threw an error ResponseError: x_content_parse_exception
	Caused by:
		x_content_parse_exception: [1:672] [bool] failed to parse field [must_not]
	Root causes:
		parsing_exception: unknown query [must_not] [siem.queryRule][suppression test][rule id 6d324ca0-dea0-11ed-bfe6-67c499fa8904][rule uuid 5ab5f9c2-611c-41ba-9df1-568c68731983][exec id 7774719f-684f-4827-b0ce-82b2426a96c1][space default]
[2023-04-19T11:55:51.357+01:00][ERROR][plugins.securitySolution.ruleExecution] Changing rule status to "failed". Bulk Indexing of signals failed: x_content_parse_exception
	Caused by:
		x_content_parse_exception: [1:672] [bool] failed to parse field [must_not]
	Root causes:
		parsing_exception: unknown query [must_not] [siem.queryRule][suppression test][rule id 6d324ca0-dea0-11ed-bfe6-67c499fa8904][rule uuid 5ab5f9c2-611c-41ba-9df1-568c68731983][exec id 7774719f-684f-4827-b0ce-82b2426a96c1][space default]

Any additional context (logs, chat logs, magical formulas, etc.):

it looks like, generated query is not valid 
    POST /suprression_1/_search?allow_no_indices=true&ignore_unavailable=true
    {
       "runtime_mappings":{
          
       },
       "size":0,
    
          "query":{
             "bool":{
                "filter":[
                   {
                      "bool":{
                         "must":[
                            
                         ],
                         "filter":[
                            {
                               "query_string":{
                                  "query":"*"
                               }
                            }
                         ],
                         "should":[
                            
                         ],
                         "must_not":[
                            
                         ]
                      }
                   },
                   {
                      "range":{
                         "@timestamp":{
                            "lte":"2023-04-19T10:55:51.130Z",
                            "gte":"2023-03-08T13:55:51.130Z",
                            "format":"strict_date_optional_time"
                         }
                      }
                   },
                   {
                      "bool":{
                         "must_not":[
                            {
                               "bool":{
                                  "filter":[
                                     {
                                        "term":{
                                           "agent.name":"agent-0"
                                        }
                                     },
                                     {
                                        "range":{
                                           "@timestamp":{
                                              "lte":"2023-04-12T10:15:50.000Z",
                                              "gte":"2023-03-08T13:55:51.130Z",
                                              "format":"strict_date_optional_time"
                                           }
                                        }
                                     }
                                  ]
                               }
                            },
                            {
                               "bool":{
                                  "filter":[
                                     {
                                        "term":{
                                           "agent.name":"agent-1"
                                        }
                                     },
                                     {
                                        "range":{
                                           "@timestamp":{
                                              "lte":"2023-04-12T09:18:50.000Z",
                                              "gte":"2023-03-08T13:55:51.130Z",
                                              "format":"strict_date_optional_time"
                                           }
                                        }
                                     }
                                  ]
                               }
                            },
                            {
                               "bool":{
                                  "filter":[
                                     {
                                        "must_not":{
                                           "exists":{
                                              "field":"agent.name"
                                           }
                                        }
                                     },
                                     {
                                        "range":{
                                           "@timestamp":{
                                              "lte":"2023-04-15T09:52:50.000Z",
                                              "gte":"2023-03-08T13:55:51.130Z",
                                              "format":"strict_date_optional_time"
                                           }
                                        }
                                     }
                                  ]
                               }
                            },
                            {
                               "bool":{
                                  "filter":[
                                     {
                                        "term":{
                                           "agent.name":"agent-1"
                                        }
                                     },
                                     {
                                        "range":{
                                           "@timestamp":{
                                              "lte":"2023-04-12T09:45:50.000Z",
                                              "gte":"2023-03-08T13:55:51.130Z",
                                              "format":"strict_date_optional_time"
                                           }
                                        }
                                     }
                                  ]
                               }
                            }
                         ]
                      }
                   }
                ]
             }
          },
          "fields":[
             {
                "field":"*",
                "include_unmapped":true
             },
             {
                "field":"@timestamp",
                "format":"strict_date_optional_time"
             }
          ],
          "aggregations":{
             "eventGroups":{
                "composite":{
                   "sources":[
                      {
                         "agent.name":{
                            "terms":{
                               "field":"agent.name",
                               "missing_bucket":true
                            }
                         }
                      }
                   ],
                   "size":100
                },
                "aggs":{
                   "topHits":{
                      "top_hits":{
                         "size":1,
                         "sort":[
                            {
                               "@timestamp":{
                                  "order":"asc",
                                  "unmapped_type":"date"
                               }
                            }
                         ]
                      }
                   },
                   "max_timestamp":{
                      "max":{
                         "field":"@timestamp"
                      }
                   },
                   "min_timestamp":{
                      "min":{
                         "field":"@timestamp"
                      }
                   }
                }
             }
          },
          "runtime_mappings":{
             
          },
          "sort":[
             {
                "@timestamp":{
                   "order":"asc",
                   "unmapped_type":"date"
                }
             }
          ]
       
    }

Response

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "unknown query [must_not]",
        "line": 86,
        "col": 48
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[86:48] [bool] failed to parse field [filter]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[86:48] [bool] failed to parse field [must_not]",
      "caused_by": {
        "type": "x_content_parse_exception",
        "reason": "[86:48] [bool] failed to parse field [filter]",
        "caused_by": {
          "type": "parsing_exception",
          "reason": "unknown query [must_not]",
          "line": 86,
          "col": 48,
          "caused_by": {
            "type": "named_object_not_found_exception",
            "reason": "[86:48] unknown field [must_not]"
          }
        }
      }
    }
  },
  "status": 400
}
@vitaliidm vitaliidm added bug Fixes for quality problems that affect the customer experience triage_needed Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Apr 19, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@vitaliidm vitaliidm added the Team:Detection Alerts Security Detection Alerts Area Team label Apr 19, 2023
@MadameSheema MadameSheema added the Team:Detections and Resp Security Detection Response Team label Apr 19, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@marshallmain
Copy link
Contributor

I think the problem is here - the must_not clause should be a sibling to filter instead of part of the filter array. Something like this should work instead:

return [
    {
      bool: {
        must_not: bucketHistory.map((bucket) => ({
          bool: {
            must_not: Object.entries(bucket.key)
              .filter(([_, value]) => value == null)
              .map(([field, _]) => ({
                exists: {
                  field,
                },
              })),
            filter: [
              ...Object.entries(bucket.key)
                .filter(([_, value]) => value != null)
                .map(([field, value]) => ({
                  term: {
                    [field]: value,
                  },
                })),
              buildTimeRangeFilter({
                to: bucket.endDate,
                from: from.toISOString(),
                primaryTimestamp,
                secondaryTimestamp,
              }),
            ],
          },
        })),
      },
    },
  ];

vitaliidm added a commit to vitaliidm/kibana that referenced this issue Apr 20, 2023
@vitaliidm
fix for elastic#155242
ef737a2
@vitaliidm vitaliidm mentioned this issue Apr 21, 2023
Merged
6 tasks
vitaliidm added a commit that referenced this issue Apr 25, 2023
@vitaliidm
[Security Solution][Alerts] adds suppression for missing fields optio…
5fb93a1 
…ns for security rules (#155055)

## Summary

- adresses #150101
- fixes #155242
- introduces UI options for selections 2 modes for suppressing or not
suppressing alerts with missing **Group By** fields
- adds accordion that contains all suppression related logic

### UX changes
#### Rule edit page
<details>
<summary>Accordion closed</summary>
<img width="1042" alt="Screenshot 2023-04-21 at 16 09 44"
src="https://user-images.githubusercontent.com/92328789/233700543-8a5091e0-6455-4d76-b6b6-7a280d747d0c.png">
</details>

<details>
<summary>Accordion opened</summary>
<img width="1017" alt="Screenshot 2023-04-24 at 19 44 33"
src="https://user-images.githubusercontent.com/92328789/234087516-58b88dab-0285-47ca-a016-bfff31dbebae.png">
</details>

#### Rule Details page

<img width="2293" alt="Screenshot 2023-04-19 at 18 50 13"
src="https://user-images.githubusercontent.com/92328789/234004667-d879bfff-0d11-4bc9-ab5b-7ad904e29d1f.png">


### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [x] Any UI touched in this PR does not create any new axe failures
(run axe in browser:
[FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/),
[Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [x] This renders correctly on smaller devices using a responsive
layout. (You can test this [in your
browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [x] This was checked for [cross-browser
compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Fixes for quality problems that affect the customer experience Team:Detection Alerts Security Detection Alerts Area Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
4 participants
@elasticmachine @MadameSheema @marshallmain @vitaliidm