Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Discover content for 2.0 features #8177

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c72cf75
Update Discover content for 2.0 features
vagimeli Sep 5, 2024
6f8f1d0
Update index.md, add enhancements, move text to new page
vagimeli Sep 16, 2024
48bfa61
Update index.md, add enhancements, move text to new page
vagimeli Sep 16, 2024
fe2589f
Merge branch 'main' into discover-2.0
vagimeli Oct 8, 2024
9f891f9
Update exploring-query-enhancements.md
vagimeli Oct 14, 2024
5432c4f
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 14, 2024
b0b1b2f
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 14, 2024
438b376
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 16, 2024
6f1a197
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 16, 2024
6beaba9
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 22, 2024
c921d47
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 22, 2024
268d517
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
39c5f94
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
3579519
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
a8152c5
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
feb7db8
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
a390fe2
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 29, 2024
7cb37bd
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 29, 2024
0ffbd4d
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 29, 2024
c0be285
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 29, 2024
20d4dcf
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
150aa58
Update _dashboards/discover/defining-analyzing-searches.md
vagimeli Oct 29, 2024
6fc3866
Update _dashboards/discover/exploring-query-enhancements.md
vagimeli Oct 29, 2024
8037daa
Merge branch 'main' into discover-2.0
vagimeli Nov 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions _dashboards/discover/defining-analyzing-searches.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
layout: default
title: Defining and analyzing searches
parent: Analyzing data
nav_order: 10
---

# Defining and analyzing searches

Discovering and exploring data in OpenSearch Dashboards is a powerful way to gain insights from your data. The Discover application within OpenSearch Dashboards provides a flexible interface for defining and analyzing searches across your data.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Defining a search

To define a search, follow these steps:

1. On the OpenSearch Dashboards navigation menu, select **Discover**.
2. Choose the data you want to work with. In this case, choose `opensearch_dashboards_sample_data_flights` from the upper-left dropdown menu.
3. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/calendar-oui.png" class="inline-icon" alt="calendar icon"/>{:/} icon to change the time range of your search and then select **Refresh**.

## Analyzing document tables

In OpenSearch, a document table stores unstructured data. In a document table, each row represents a single document, and each column contains document attributes.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

### View document attributes
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To review document attributes, follow these steps:

1. From the data table's left column, choose the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/inspect-icon.png" class="inline-icon" alt="inspect icon"/>{:/} icon to open the **Document Details** window. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/minimize-icon.png" class="inline-icon" alt="minimize icon"/>{:/} icon to close the **Document Details** window.
2. Examine the metadata. You can switch between the **Table** and **JSON** tabs to view the data in your preferred format.
3. Select **View surrounding documents** to view data for other log entries either preceding or following your current document or select **View single document** to view a particular log entry.

### Add or delete fields in document tables
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To add or delete fields in a document table, follow these steps:

1. View the data fields listed under **Available fields** and select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/plus-icon.png" class="inline-icon" alt="plus icon"/>{:/} icon to add the desired fields to the document table. The field will be automatically added to both **Selected fields** and the document table. For this example, choose the fields `Carrier`, `AvgTicketPrice`, and `Dest`.
2. Select **Sort fields** > **Pick fields to sort by**. Drag and drop the chosen fields in the desired sort order.

## Searching data

You can use the search toolbar to enter a [DQL]({{site.url}}{{site.baseurl}}/dashboards/discover/dql/) or [query string]({{site.url}}{{site.baseurl}}/query-dsl/full-text/query-string/) query. The search toolbar is best for basic queries; for full query and filter capability, use [query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/query-dsl/index/) in the [Dev Tools console]({{site.url}}{{site.baseurl}}/dashboards/dev-tools/index-dev/).
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

For more information, see [Discover and Dashboard search toolbar]({{site.url}}{{site.baseurl}}/dashboards/index/#discover-and-dashboard-search-bar).

## Filtering data

Filters allow you to narrow the results of a query by specifying certain criteria. You can filter by field, value, or range. The **Add filter** pop-up suggests the available fields and operators.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To filter your data, follow these steps:

1. Under the DQL search bar, choose **Add filter**.
2. Select the desired options from the **Field**, **Operator**, and **Value** dropdown lists. For example, select `Cancelled`, `is`, and `true`.
3. Choose **Save**.
4. To remove a filter, choose the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/cross-icon.png" class="inline-icon" alt="cross icon"/>{:/} icon to the right of the filter name.

## Saving a search

To save your search, including the query text, filters, and current data view, follow these steps:

1. Select **Save** on the upper-right toolbar.
2. Add a title, and then choose **Save**.
3. Select **Open** on the upper-right toolbar to access your saved searches.

## Visualizing data findings

To visualize your data findings, follow these steps:

1. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/inspect-icon.png" class="inline-icon" alt="inspect icon"/>{:/} icon to the right of the field you want to visualize.
2. Select the **Visualize** button. When the **Visualize** application is launched, a visualization appears.

## Setting alerts

Set alerts to notify you when your data exceeds your specified thresholds. Go to [Alerting dashboards and visualizations]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/dashboards-alerting/) to learn about creating and managing alerts.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
73 changes: 73 additions & 0 deletions _dashboards/discover/exploring-query-enhancements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
layout: default
title: Exploring query enhancements
parent: Analyzing data
nav_order: 20
---

# Exploring query enhancements
Introduced 2.17
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
{: .label .label-purple }

Starting with OpenSearch 2.17, query enhancements have been made. These enhancements are experimental and may be subject to change or instability. Enhancements include the following:

- Query languages PPL and SQL, with **Query Assist** for PPL
- Multiline query editor for PPL and SQL and autocomplete for PPL and DQL
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did we wanted to mention the expand and collapse feature of the query editor? they can go from multiline to a single line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kavilla Please see revised text. You're welcome to make any necessary changes to the text. Thank you!

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
- Data selector with **Data Explorer** that supports index patterns, indexes, and Amazon S3 connections and data configuration that supports selecting the appropriate query language based on the data type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it there enough to call out that index patterns aren't required for exploring your data within your indexes within OpenSearch Dashboards but still required if you want to create visualizations

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this communicate the appropriate message: OpenSearch Dashboards allows for exploring your data within your indexes without using index patterns

vagimeli marked this conversation as resolved.
Show resolved Hide resolved
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
- Link sharing through URLs without needing write permission to create an index pattern
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

creating an index pattern with write permission is required which was a blocker for exploring data. non-privileged users would have to depend on someone with permission to create the index pattern for the data to be explored leading to suboptimal index patterns to be created as it would be easier to just do something like logs-* instead of the specific logs. so i think it is right but there's a couple number of features that get unlocked without the required write permissions

Copy link
Contributor Author

@vagimeli vagimeli Oct 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kavilla @sejli Should the link sharing bullet be modified to read: Link sharing via URLs eliminates the need for write permissions to create index patterns, removing a significant barrier to data exploration. This allows non-privileged users to access and interact with specific data directly, without relying on broad, potentially suboptimal index patterns, such as logs-*, created by users with higher permissions

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, @kavilla?


All existing functionality remains in OpenSearch 2.17, and the new features are designed to improve your data exploration experience. The **Enable query enhancements** is currently not on a minimum distribution. <SME: Do we need to add text about plugins?>. You can leave your feedback at <insert-link> to help the OpenSearch Project improve.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sejli Will you provide me with the link for users to leave feedback about these features? Also, please review this paragraph. Do we need to add text about plugins?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sejli Confirming this is on for 2.18. Do we have updates to the draft text?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sejli No one has provided any updates. The developers write the initial drafts for new features and the tech writer refines the content. Any updates developers can make in the PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • In a distribution of just OpenSearch and OpenSearch Dashboards, query enhancements will not provide access to PPL or SQL. This requires the OpenSearch SQL plugin. I think we can make it known that query enhancements does work with just OpenSearch, but will require the SQL plugin and others to make SQL, PPL, and external datasource queries work.
  • Other than making a note for getting SQL and PPL to work, looks good to me. May need some more input from @kavilla as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revised the Note text to add this information.

{: .note}

## Enabling query enhancements

To enable the query enhancements through OpenSearch Dashboards, follow these steps:

1. Go to **Dashboards Management** > **Advanced settings** > **Search** and toggle on **Enable query enhancements**. Tip: You can select the **Search** pane from the **Category** dropdown menu in the upper-right search bar.
2. Select the **Save** button to save your changes.
3. Reload the page as prompted in the pop-up message.

Alternatively, you can override the setting on startup by running the following command:

```
./bin/opensearch-dashboards --uiSettings.overrides['query:enhancements:enabled']=true
```
{% include copy-curl.html %}

## Using the experimental features

The following tutorials guide you through some of the experimental features and capabilities.

### Query language enhancements

You can now use PPL in **Discover**. Follow these steps to try out the feature:

1. Go to **Discover** and select **PPL** from the query language dropdown menu in the upper-right search bar. You should see a dashboard containing the query editor, histogram, and data table panes.
2. Select a sample dataset. For this example, select `opensearch_dashboards_sample_data_ecommerce` from the data source dropdown menu above the query editor and adjust the time filter to **Last 1 year**.
3. Enter the following example PPL query:

```json
source = opensearch_dashboards_sample_data_logs
| where tags = "success"
| where geo.dest = "US"
```
{% include copy-curl.html %}

4. View the resulting output that shows the number of successful log entries originating from the United States. You should see an updated histogram and data table following the query editor.
5. Select the **Recent queries** option within the query editor toolbar to display your recent queries.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we mention this available for non-DQL/Lucene queries?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
5. Select the **Recent queries** option within the query editor toolbar to display your recent queries.
5. Select the **Recent queries** option within the query editor toolbar to display your recent queries. This is available for non-DQL and Lucene queries.


PPL and DQL provide an autocomplete option that suggests field names, functions, and syntax.


## Data sources and data types enhancements
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

You can now select your data sources and types from within the **Discover** dashboard. Follow these steps to try out the feature.

1. From the **Discover** page, select a data source from the dropdown menu in the upper toolbar.
2. Select the **View all available data** button to display a list of your available data sources. You may need to refresh your page to display any newly added data sources.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: @vagimeli might have better context on the correct terms but worried about saying data sources given some users got confused. They are similar but different. Ideally data is broad strokes enough to be correct but like i said not sure what we wanted to communicate going forward to end users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kavilla I'm open to suggestions. Please make a suggested rewrite and I'll accept changes. Thank you.

3. Select the desired data source and follow steps displayed in the data sources window to manage your data source.

You can now use **Query Assist** with PPL queries. With **Query Assist**, you can ask questions like _Are there any errors in my logs?_. The assistant includes predefined prompts. Follow these steps to try out the feature:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't see it but should we call out that Query Assist for PPL but not necessarily available for all data sources

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kavilla Should we tell user which specific data sources/types it's available for?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query Assist is only available in query enhancements when the data source itself supports it. It's dependent on if the user adds a cluster with query assist enabled/set up. I think we can call out that query assist is required on the OpenSearch data source, something like query assist is available when included data source has it available?


1. Select **PPL** from the dropdown menu in the query toolbar.
2. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/dashboards/query-assist.png" class="inline-icon" alt="query assist icon"/>{:/} icon and choose a predefined question. The resulting output is displaying in the query editor pane.
99 changes: 3 additions & 96 deletions _dashboards/discover/index-discover.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,105 +9,12 @@ redirect_from:

# Analyzing data

To analyze your data in OpenSearch and visualize key metrics, you can use the **Discover** application in OpenSearch Dashboards. An example of data analysis in **Discover** is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/discover.png" alt="A Discover default page" width="700">
You can analyze your data in OpenSearch and visualize key metrics using the **Discover** application in OpenSearch Dashboards. Using **Discover**, you can explore and visualize data from various data sources, data types, and query languages.

## Getting started

In this tutorial, you'll learn about using **Discover** to:

- Add data.
- Interpret and visualize data.
- Share data findings.
- Set alerts.

Before getting started, make sure you:
Before getting started with exploring and visualizing your data using **Discover**, make sure you:

- Install [OpenSearch Dashboards](https://opensearch.org/downloads.html).
- Add sample data or import your own data into OpenSearch. Go to the [OpenSearch Dashboards quickstart guide]({{site.url}}{{site.baseurl}}/dashboards/quickstart/) to learn about adding sample datasets. Go to [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/) to learn about importing your own data.
- Add sample data or import your own data into OpenSearch. See [OpenSearch Dashboards quickstart guide]({{site.url}}{{site.baseurl}}/dashboards/quickstart/) to learn about adding sample datasets or [Managing indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/) to learn about importing your own data.
- Have a foundational understanding of [OpenSearch documents and indexes]({{site.url}}{{site.baseurl}}/im-plugin/index/).

## Defining the search

To define a search, follow these steps:

1. On the OpenSearch Dashboards navigation menu, select **Discover**.
2. Choose the data you want to work with. In this case, choose `opensearch_dashboards_sample_data_flights` from the upper-left dropdown menu.
3. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/calendar-oui.png" class="inline-icon" alt="calendar icon"/>{:/} icon to change the time range of your search and then select **Refresh**.

The resulting view is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/define-search.png" alt="Discover interface showing search of flight sample data for Last 7 days" width="700">

## Analyzing document tables

In OpenSearch, a document table stores unstructured data. In a document table, each row represents a single document, and each column contains document attributes.

To examine document attributes, follow these steps:

1. From the data table's left column, choose the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/inspect-icon.png" class="inline-icon" alt="inspect icon"/>{:/} icon to open the **Document Details** window. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/minimize-icon.png" class="inline-icon" alt="minimize icon"/>{:/} icon to close the **Document Details** window.
2. Examine the metadata. You can switch between the **Table** and **JSON** tabs to view the data in your preferred format.
3. Select **View surrounding documents** to view data for other log entries either preceding or following your current document or select **View single document** to view a particular log entry.

The resulting view is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/doc-details.png" alt="Document attributes" width="700">

To add or delete fields in a document table, follow these steps:

1. View the data fields listed under **Available fields** and select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/plus-icon.png" class="inline-icon" alt="plus icon"/>{:/} icon to add the desired fields to the document table. The field will be automatically added to both **Selected fields** and the document table. For this example, choose the fields `Carrier`, `AvgTicketPrice`, and `Dest`.
2. Select **Sort fields** > **Pick fields to sort by**. Drag and drop the chosen fields in the desired sort order.

The resulting view is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/add-data-fields.png" alt="Adding and deleting data fields" width="700">

## Searching data

You can use the search toolbar to enter a [DQL]({{site.url}}{{site.baseurl}}/dashboards/discover/dql/) or [query string]({{site.url}}{{site.baseurl}}/query-dsl/full-text/query-string/) query. The search toolbar is best for basic queries; for full query and filter capability, use [query domain-specific language (DSL)]({{site.url}}{{site.baseurl}}/query-dsl/index/) in the [Dev Tools console]({{site.url}}{{site.baseurl}}/dashboards/dev-tools/index-dev/).

For more information, see [Discover and Dashboard search toolbar]({{site.url}}{{site.baseurl}}/dashboards/index/#discover-and-dashboard-search-bar).

## Filtering data

Filters allow you to narrow the results of a query by specifying certain criteria. You can filter by field, value, or range. The **Add filter** pop-up suggests the available fields and operators.

To filter your data, follow these steps:

1. Under the DQL search bar, choose **Add filter**.
2. Select the desired options from the **Field**, **Operator**, and **Value** dropdown lists. For example, select `Cancelled`, `is`, and `true`.
3. Choose **Save**.
4. To remove a filter, choose the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/cross-icon.png" class="inline-icon" alt="cross icon"/>{:/} icon to the right of the filter name.

The resulting view is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/discover-filter.png" alt="Visualize data findings interface" width="700"/>

## Saving a search

To save your search, including the query text, filters, and current data view, follow these steps:

1. Select **Save** on the upper-right toolbar.
2. Add a title, and then choose **Save**.
3. Select **Open** on the upper-right toolbar to access your saved searches.

## Visualizing data findings

To visualize your data findings, follow these steps:

1. Select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/icons/inspect-icon.png" class="inline-icon" alt="inspect icon"/>{:/} icon to the right of the field you want to visualize.

The resulting view is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/visualize-discover.png" alt="Visualize data findings interface" width="700"/>

2. Select the **Visualize** button. When the **Visualize** application is launched, a visualization appears.

The resulting view is shown in the following image.

<img src="{{site.url}}{{site.baseurl}}/images/dashboards/visualization-flight.png" alt="Data visualization of flight sample data field destination" width="700"/>

## Setting alerts

Set alerts to notify you when your data exceeds your specified thresholds. Go to [Alerting dashboards and visualizations]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/dashboards-alerting/) to learn about creating and managing alerts.
2 changes: 1 addition & 1 deletion _dashboards/discover/time-filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
layout: default
title: Time filter
parent: Analyzing data
nav_order: 20
nav_order: 50
redirect_from:
- /dashboards/get-started/time-filter/
- /dashboards/discover/time-filter/
Expand Down
Binary file added images/dashboards/query-assist.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading