[Security Solution] Nested fields don't load autocomplete values

[MadameSheema]

Originally reported by: @dplumlee

Info:

• in Master

Steps to reproduce:

1. Load an alert into the detections alert table
2. Open either Add exception modal
3. Choose a non-boolean nested field such as file.Ext.code_signature.subject_name as the field name

Current behavior:

• No autocomplete values are returned when the values box is focused

Expected behavior:

• The correct autocomplete values are returned and displayed

Dev notes:

• In attempting to triage this earlier with @peluja1012 we discovered that the kibana autocomplete value service used in this file was returning nothing, and the way we currently implement the hook/api call, the backend seemingly isn't recognizing the field as nested.

Commented by @yctercero on 28th Jul:
@dplumlee did you try this with the timeline Add Filter and KQL search bar? Trying it there, it seems to be the same issue (as we're using the same service).

Commented by @dplumlee on 28th Jul:
@yctercero i looked through the KQL code and saw they were passing nested fields down a slightly different way but the Add Filter seems to be getting the same issue for me, yeah

[yctercero]

For whoever grabs this one - needs to be revisited as it was a known issue (core issue) when originally coded up.

[yctercero]

This sees to be related to #137709 - could QA confirm that the steps outlined in that other issue resolve this as well?

cc @MadameSheema

[MadameSheema]

@samratbhadra-qasource can you please validate if the above issue is still happening after following this?: #137709 (comment) Thanks!!! :)

[ghost]

Hi @MadameSheema

We have tested this issue on the latest 8.5.0-Snapshot build and observed that the issue is still occurring. Please find below the testing details:

Build Details:
VERSION: 8.5.0-SNAPSHOT
BUILD: 55993
COMMIT: 436b287

Screenshot:

• When turned on the autocomplete use time range option.

• When turned off the autocomplete use time range option.

Thanks!

[MadameSheema]

Thanks @samratbhadra-qasource!!

@yctercero can you please take a look at the above comment when you have the chance?

[yctercero]

Thanks! I'll put it in for 8.6 - I'm not sure we'll get to it 8.5 but will prioritize if we can.

[dhurley14]

Related: #144229

[dhurley14]

@WafaaNasr Can you take a look at this bug? It looks like the autosuggest is populating but the validation for the "Add Exception" button is preventing us from adding this nested exception. I know you were in the validation code recently so hopefully this is an easy fix.

To test you will need endpoint data which you can generate by using the following script:

cd ~/kibana/x-pack/plugins/security_solution/scripts/endpoint && nvm use && yarn test:generate

Here is a screenshot:

[cybersecdiva]

Tested in current 8.7.0 deployment:

Preconditions:

• Alerts generated with both Endpoint Security and non-endpoint data
• Verify that Autocomplete is on in Stack Management
  • To ✅ if Autocomplete is enabled, Navigate to Stack Management -> Advanced Settings
  • In the search box, enter first three characters "auto" to bring up Autocomplete section search results
  • In the Autocomplete section, Under the Use time range section, on the right check ✅ that "autocomplete:useTimeRange" button is On

Steps to reproduce behavior:

1. Trigger an alert for the Endpoint Security rule
   • In this case, I triggered mimikatz
2. Navigate to Security -> Alerts
3. Next to the triggered alert Under Actions column click on the three dots⚈⚈⚈icon
4. Click on Add Rule exception enter an exception name
5. In the Field section, key in file.Ext.code and select from the drop down list menu options 🔽 File.Ext.code_signature.subject_name for the field.
6. In the Value section, click on the drop down list 🔽

Results:

Autocomplete values for the nested field does not load and displays a message "There aren't any options available" when drop down list is selected

Expected results:

Autocomplete values are returned with drop down list options 🔽

Screen video capture:

Bug_.Nested.fields.don.t.load.autocomplete.values.mp4

Observations:

• Autocomplete values in Nested fields don't load when a rule or endpoint exception after selection of Add rule exception or Add endpoint exception
• Behavior occurs even though Autocomplete "autocomplete:useTimeRange" button 🔘 is On
  • Note: I also verified and checked behavior with the Autocomplete "autocomplete:useTimeRange" button 🔘 is Off and the results are the same

Screenshots:

Rule Exception Field File.Ext.code_signature.subject_name that shows Autocomplete values not generating:

Endpoint Exception Field File.Ext.code_signature.subject_name that shows Autocomplete values not generating:
Note: I deleted the automatic populated fields that display and generate when creating an Endpoint Exception and added under the Field sectionFile.Ext.code_signature.subject_name

Endpoint Exception Field File.Ext.code_signature.subject_namewith boolean operator that shows Autocomplete values not generating:

Conclusion:

Validated that the behavior is still occurring in 8.7.0

cc: @MadameSheema @dhurley14 @WafaaNasr Update FYI Observations

[cybersecdiva]

Tested in8.9.0:

Build Details:
VERSION: 8.9.0 BC5
BUILD: 64715
COMMIT: beb56356c5c037441f89264361302513ff5bd9f8

Preconditions:

• Alerts generated with both Endpoint Security and non-endpoint data
• Verify that Autocomplete is on in Stack Management
  • To ✅ if Autocomplete is enabled, Navigate to Stack Management -> Advanced Settings
  • In the search box, enter first three characters "auto" to bring up Autocomplete section search results
  • In the Autocomplete section, Under the Use time range section, on the right check ✅ that "autocomplete:useTimeRange" button is On

Steps to reproduce behavior:

1. Trigger an alert for the Endpoint Security rule
   • In this case, I triggered mimikatz
2. Navigate to Security -> Alerts
3. Next to the triggered alert Under Actions column click on the three dots⚈⚈⚈icon
4. Click on Add Rule exception enter an exception name
5. In the Field section, key in file.Ext.code and select from the drop down list menu options 🔽 File.Ext.code_signature.subject_name for the field.
6. In the Value section, click on the drop down list 🔽

Results:

Autocomplete values for the non-nested field does not load and displays a message "There aren't any options available" when drop down list is selected

Expected results:

Autocomplete values are returned with drop down list options 🔽

Screen video capture:

Non-nested.fields.bug_.mp4

Observations:

• Autocomplete values for non-nested fields don't load when a rule or endpoint exception after selection of Add rule exception or Add endpoint exception
• Behavior occurs even though Autocomplete "autocomplete:useTimeRange" button 🔘 is On
  • Note: I also verified and checked behavior with the Autocomplete "autocomplete:useTimeRange" button 🔘 is Off and the results are the same

Screenshots:

Rule Exception Field File.Ext.code_signature.subject_name that shows Autocomplete values not generating for
non-nested fields:

Endpoint Exception Field File.Ext.code_signature.subject_name that shows Autocomplete values not generating for
non-nested fields:
Note: I deleted the automatic populated fields that display and generate when creating an Endpoint Exception and added under the Field sectionFile.Ext.code_signature.subject_name

.

Conclusion:

• Validated that the behavior of non-nested fields values not generating is still occurring in 8.9.0.

• Autocomplete values for nested fields in 8.9.0 do generate for both rule or endpoint exception and are working, the bug 🐛 that needs to be fixed is for non-nested fields

cc: @MadameSheema @dhurley14 @WafaaNasr Updated FYI Observations