[Security Solution]: Auto suggestion custom query filters are not displaying while user click on custom query filter bar.

[ghost]

Describe the bug:
Auto suggestion custom query filters are not displaying while user click on custom query filter bar.

Build Details:

VERSION: 8.6.0-SNAPSHOT
BUILD: 57808
COMMIT: 796751e2d3f36499cb8eb61b436c03006632f1ef

Preconditions

1. Kibana should be running.

Steps to Reproduce

1. Navigate to Rules tab.
2. Now, click on create new rule.
3. Create a custom query rule
4. Now, click on the custom query filter bar.
5. Observe that auto suggestion custom query filters are not displaying while user click on custom query filter bar.

Actual Result
Auto suggestion custom query filters are not displaying while user click on custom query filter bar.

Expected Result
Auto suggestion custom query filters should display while user click on custom query filter bar.

Screen-Recording:

Rules.-.Kibana.-.Google.Chrome.2022-10-31.14-18-06.mp4

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[ghost]

@karanbirsingh-qasource Please review.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[dhurley14]

Why was this upgraded from impact:medium to impact:high? @marshallmain @MadameSheema

[marshallmain]

This significantly affects the UX in multiple parts of the app where we expose the query bar so I think it's important to address in this release.

[dhurley14]

Related: #81299

[dhurley14]

After researching this issue, we can break it down in two parts:

1. The autocomplete for the fields is not working.
2. The autosuggest for the values is not working.

Issue 1. will be resolved here by updating the format of the index pattern to include two additional fields "type": "title", "value": "auditbeat-*,packetbeat-*..."

Issue 2. is proving more difficult to resolve. I believe between the change introduced here where unified search switched from this.props.deps.data.indexPatterns to this.props.deps.data.dataViews and this PR #143389 essentially introduced a requirement that only values from data views be offered up as suggestions. If a data view is found to contain a title that matches the title provided to the autosuggest function, values will appear. However if no title is discovered the code will default to querying for values against whatever data view is set as the default data view, which in the security solution case is the data view containing the title .alerts-security.alerts-default,apm-*-transaction*,auditbeat-*,endgame-*,filebeat-*,logs-*,packetbeat-*,traces-apm*,winlogbeat-*,-*elastic-cloud-logs-*

Now the issue here is twofold. One, we may not want values to be suggested from a data view with a title that matches or partially matches (although partial matching of titles does not seem to work) the multi-target index pattern provided to the autosuggest function.

The more immediate issue is that since there is no .alerts-security.alerts-default index set up until an alert is written, the autosuggest function returns a 404 because that index does not exist, as shown in the screenshot below.

After briefly examining the cypress tests I found this autosuggest value issue did not fail our cypress tests because our cypress tests are testing creating a rule with a data view. I did not see a test where we create a rule with a multi-target index pattern.

I will point out that the autosuggest does work when the data source selected is a data view (unless the selected data view is the security solution default dataview which contains the .alerts index, that fails for the same reasons described above). Selecting a data view directly from the create rules app succeeds because it follows a different code path (outlined here) So when a string "title" is provided the fetchIndexPattern from unified search tries to find an accompanying data view. But since we are passing in a data view object when we select a data view in the create rules form, we skip this step.

@rylnd @marshallmain @peluja1012 any suggestions for resolving this issue? Happy to jump on a call and walk everyone through the code as well.

[dhurley14]

Also in regards to @marshallmain comment here

This significantly affects the UX in multiple parts of the app where we expose the query bar so I think it's important to address in this release.

After testing locally, it seems to me that every place in the app which utilizes the unified search bar is running into this issue. Briefly looking at the code I believe the autosuggest values for overview / timeline page are coming from a bsearch in timeline but I could be mistaken. Either way it might be good to have the other areas in threat hunting corroborate these findings.

[ghost]

Hi @MadameSheema

We have tested this issue on 8.6.0-BC1 build and have checked all the pages where we can use the unified search bar and observed that the auto suggested fields are not populating on the below shared screenshot pages.

Screenshot:

• Timeline Template page

• Alerts page

• Rules Page

• Timeline page

• Host page

• Network Page

• Users Page

• Overview Page

• Kubernetes Page

Please do let us know if anything else is required from our end.
Thanks!

[MadameSheema]

@dhurley14 please take a look at the above observations, thanks!

[dhurley14]

Update from today: A few of us (cc: @stephmilovic @YulNaumenko @XavierM ) met today to look at the differences between the change needed for the rule query bar vs the query bar (SearchBar component) on the other app pages in the Security Solution. After inspecting the functional changes necessary for the SearchBar component to function in the Security Solution we decided it would be best to pursue a much smaller and less risky change in the unified search plugin code. The degree of difficulty to make the required changes in sourcerer is fairly high and would require extensive testing to reduce the likelihood of introducing new bugs. Seeing as this is functionality that is broken in 8.6 and with the second BC set to come tomorrow, the smaller and less risky choice makes the most sense when considering the time restrictions we are operating under. With that in mind, @mattkime said he would take a look at it over the next day or so and try to determine an optimal fix. For now, @stephmilovic has opened a draft PR available here. Currently, search remains broken across the app.

[stephmilovic]

@samratbhadra-qasource can you please confirm this fix?

[ghost]

Hi @stephmilovic

We have validated this issue on the latest 8.6.0-BC4 build and observed that the issue is fixed. Please find below the testing details:

Build Details:
VERSION: 8.6.0-BC4
BUILD: 58612
COMMIT: 218162f

Screen-Recording:

Home.-.Elastic.-.Google.Chrome.2022-12-01.11-01-21.mp4

Hence, closing this issue and marking it issue as QA Validated.

cc: @MadameSheema
Thanks!