First stab at UBI dashboards

[alexeyrodriguez]

Description

Dashboards to visualize UBI query events

Issues Resolved

https://github.com/orgs/o19s/projects/4/views/1?pane=issue&itemId=82044054

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Notes for reviewer/contributor

The ticket above should be converted to an issue (I wanted to do so, but different organizations o19s vs opensearch-project).

There are remaining issues in this first iteration of Dashboards:

• In the panes for Searches by day of week and hour of day, there can be missing entries if searches do not happen in specific days of hours.
• In the same panes as above, the ordering is not satisfactory, currently days are ordered by decreasing number of queries, this is also the case for hours. For hours one could use alphanumeric ordering if hours would be strings of hours formatted as double digits.

Approaches that didn't work:

• Use the formatting facility in the scripted fields page (in index patterns). First, left padding zeroes doesn't work: Support leading zeros in number field formatter elastic/kibana#18625
• It's not clear formatting would help, as I don't know if Dashboards would sort on the numerical value or on the string representation (perhaps the latter, but it doesn't work).
• I think I tried to use datw formatting, but that didn't work: https://discuss.elastic.co/t/how-can-i-zero-pad-a-string-in-painless/101914
• Manually creating strings with 0 left-padding. This probably can still be fixed. I used the following query to test the idea in the Dev pane:

GET ubi_queries/_search
{
  "query": { "match_all": {} },
  "script_fields": {
    "hour": {
      "script": {
        "lang": "painless",
        "source": "String.valueOf(doc['timestamp'].value.getHour()).padLeft(2, '0')"
      }
    }
  }
}

[wrigleyDan]

Did some searching for the "day of week" issue we have:
Someone asked a similar question regarding some kind of custom sorting functionality in the Elastic forum and mentions two solutions - unfortunately without going into any detail:

one is by selecting Aggregation as Filters and filtering in that particular order which I want.

And second one is assigned weight to data based by sorting preference and created scripted field for it. Then use the new scripted field in aggregations to sort it.

The second sounds like it's close to our use case with scripted fields but I haven't found any clues how we would create the aggregations on one scripted field and sort by another.
We'd somehow need to achieve displaying the values that are created by the following script:

doc['timestamp'].value.getDayOfWeekEnum().getDisplayName(TextStyle.FULL, Locale.ROOT)

And to sort we'd want to use the numbers of these days, not their names:

doc['timestamp'].value.getDayOfWeek()

A "not-so-nice" alternative would be to have the aggregations based on the numbers of the days. That way we'd see 1 instead of Monday, 2 instead of Tuesday, etc.

[wrigleyDan]

Update
Working with filter aggregations works and does the trick for the mentioned issues:

• In the panes for Searches by day of week and hour of day, there can be missing entries if searches do not happen in specific days of hours.
• In the same panes as above, the ordering is not satisfactory, currently days are ordered by decreasing number of queries, this is also the case for hours. For hours one could use alphanumeric ordering if hours would be strings of hours formatted as double digits.

[wrigleyDan]

Change the field names in the visualization for Top Searches Without Results and Most common queries to match the mappings and the actually existing fields when ubi_queries is properly set up:

• user_query.keyword --> user_query
• query_response_object_ids.keyword--> query_response_hit_ids

The visualization with the most frequently returned documents currently has the challenge that there is an inconsistency between the mapping we apply for the ubi_queries index and the implementation how the ubi query event is being built.

The mappings say the field is called query_response_hit_ids (see https://github.com/opensearch-project/user-behavior-insights/blob/main/src/main/resources/queries-mapping.json#L8) while the implementation stores the ids in the field query_response_object_ids (see https://github.com/opensearch-project/user-behavior-insights/blob/main/src/main/java/org/opensearch/ubi/UbiActionFilter.java#L238). Opened an issue for that in the UBI project: #68

I suggest we exclude that visualization from the Dashboard for now until the bug is fixed and include that afterwards in the next iteration.

[wrigleyDan]

With #68 fixed we can update the visualization for the most frequently returned docs.

@alexeyrodriguez I can't push to this branch so I sent you an updated dashboards file for your review. If that looks alright from your perspective you can update this PR and remove the "draft" state from it and we can get moving again.

Thanks!

[wrigleyDan]

LGTM!

[epugh]

LGTM. We need to figure out a solution around sample data, but that can be in another PR