[discover-2.0] update interfaces and services for query, data, languages

[kavilla]

Description

This PR stabilizes the Multi-Query Language (MQL), and Multiple-Data-Sources (MDS) features by updating types to utilize a generic structure, abstracting data querying, and introducing a Language service. These changes aim to resolve conflicts in data source definitions across different projects and provide a flexible framework for handling various data structures.

Notes

• The commits on this branch were reviewed individually.
• The size of this PR is primarily due to cleanup and consolidation efforts.

This PR represents a significant step towards a more flexible and extensible multi-query language system in OpenSearch Dashboards.

Key Changes

1. Generic Data Structure: Introduced a flexible DataStructure interface that can represent various data hierarchies without hard-coding specific fields.

2. Abstract Data Querying: Implemented a system where actual data querying is left to the project or developer to implement, allowing for diverse opinions and implementations.

3. Language Service: Added a Language service that registers more closely with the query object, improving the flow of queries through the application.

4. Consolidation: Cleaned up and consolidated existing code, resulting in a large PR despite individual commits being reviewed separately.

DataStructure Interface

The core of this update is the DataStructure interface:

export interface DataStructure {
  id: string;
  title: string;
  type: string;
  parent?: DataStructure;
  children?: DataStructure[];
  hasNext?: boolean;
  columnHeader?: string;
  meta?: DataStructureMeta;
}

This interface allows for representation of various data hierarchies, such as OpenSearch clusters, indices, or even complex nested structures like S3 data sources.

Benefits

• Flexibility: Developers can populate the DataStructure with values specific to their data source types without modifying the core interface.
• Consistency: Standardized structure for representing different data sources and their hierarchies.
• Extensibility: Easy to add new data source types without changing the underlying system.
  Improved Query Handling: The Language service allows for better integration of queries with different data structures.

Scope

Currently, these changes are only enabled for the Discover application. They lay the groundwork for potential future expansion to other parts of the system.

Screenshot

Testing the changes

local and CI

Changelog

• refactor: Multi-datasources and multi-query languages features to use generic structured types, abstract data querying, and the language service

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[github-actions]

ℹ️ Manual Changeset Creation Reminder

Please ensure manual commit for changeset file 7731.yml under folder changelogs/fragments to complete this PR.

If you want to use the available OpenSearch Changeset Bot App to avoid manual creation of changeset file you can install it in your forked repository following this link.

For more information about formatting of changeset files, please visit OpenSearch Auto Changeset and Release Notes Tool.

[github-actions]

❌ Changeset File Not Added Yet

Please ensure manual commit for changeset file 7731.yml under folder changelogs/fragments to complete this PR. File still missing.

[codecov]

Codecov Report

Attention: Patch coverage is 30.96774% with 214 lines in your changes missing coverage. Please review.

Project coverage is 64.16%. Comparing base (6931a41) to head (caf949c).
Report is 147 commits behind head on main.

Files with missing lines	Patch %	Lines
.../query_string/language_service/language_service.ts	21.27%	37 Missing ⚠️
...y_string/dataset_service/lib/index_pattern_type.ts	6.45%	29 Missing ⚠️
...ry/query_string/dataset_service/dataset_service.ts	18.18%	24 Missing and 3 partials ⚠️
...ery/query_string/dataset_service/lib/index_type.ts	13.79%	25 Missing ⚠️
.../public/query/query_string/query_string_manager.ts	50.00%	13 Missing and 1 partial ⚠️
...c/plugins/data/common/datasets/_structure_cache.ts	0.00%	11 Missing ⚠️
...public/utils/state_management/redux_persistence.ts	23.07%	9 Missing and 1 partial ⚠️
...blic/ui/search_bar/lib/use_query_string_manager.ts	0.00%	9 Missing ⚠️
src/plugins/data/public/search/search_service.ts	0.00%	8 Missing ⚠️
...c/plugins/data/public/ui/search_bar/search_bar.tsx	36.36%	6 Missing and 1 partial ⚠️
... and 17 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7731      +/-   ##
==========================================
+ Coverage   63.86%   64.16%   +0.30%     
==========================================
  Files        3665     3659       -6     
  Lines       81441    80796     -645     
  Branches    12994    12866     -128     
==========================================
- Hits        52010    51846     -164     
+ Misses      26243    25770     -473     
+ Partials     3188     3180       -8     

Flag	Coverage Δ
Linux_1	30.30% <12.85%> (+0.10%)	⬆️
Linux_2	55.89% <14.28%> (+0.01%)	⬆️
Linux_3	40.67% <31.98%> (+0.27%)	⬆️
Linux_4	31.46% <12.69%> (+0.09%)	⬆️
Windows_1	30.32% <12.85%> (+0.08%)	⬆️
Windows_2	55.85% <14.28%> (+0.01%)	⬆️
Windows_3	40.67% <31.98%> (+0.27%)	⬆️
Windows_4	31.46% <12.69%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

The merge-base changed after approval.

The merge-base changed after approval.

[kavilla]

I dont know why github seems to be dismissing this PR approvals. It's actually really annoying. I'm not pushing any changes and I'm not dismissing any of the reviews.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-7731-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1976ecfd223bb8cb2462b2779d1b7af97016c476
# Push it to GitHub
git push --set-upstream origin backport/backport-7731-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7731-to-2.x.

[AMoo-Miki]

Manually backported with #7863