Skip to content

Commit

Permalink
[Auto Suggest] Add MDS Support Along with A Few Cleanup and tests (op…
Browse files Browse the repository at this point in the history
…ensearch-project#7463)

* add tests for sql autocomplete rule processing

Signed-off-by: Eric <menwe@amazon.com>

* refer to monaco type directly

Signed-off-by: Eric <menwe@amazon.com>

* remove unnecessary antlr auto generated files

Signed-off-by: Eric <menwe@amazon.com>

* inital adoption of dataSet manager

Signed-off-by: Eric <menwe@amazon.com>

* mds support

Signed-off-by: Eric <menwe@amazon.com>

* remove test that are failed due to adopting dataSet manager

Signed-off-by: Eric <menwe@amazon.com>

* add changelog

Signed-off-by: Eric <menwe@amazon.com>

* fix(query assist): update reading data source id from dataset manager (opensearch-project#7464)

* revert to read datasource id from index pattern

Signed-off-by: Joshua Li <joshuali925@gmail.com>

* add dataset mock to query mock

Signed-off-by: Joshua Li <joshuali925@gmail.com>

* update query assist to use dataset manager

Signed-off-by: Joshua Li <joshuali925@gmail.com>

* use selected dataset state instead of relying on rerender

Signed-off-by: Joshua Li <joshuali925@gmail.com>

* remove skip 1 in dataset observable

Signed-off-by: Joshua Li <joshuali925@gmail.com>

* update dataset_manager tests

Signed-off-by: Joshua Li <joshuali925@gmail.com>

---------

Signed-off-by: Joshua Li <joshuali925@gmail.com>

* update utils

Signed-off-by: Eric <menwe@amazon.com>

* keep with observable and remove values suggestion

Signed-off-by: Eric <menwe@amazon.com>

* update unit tests

Signed-off-by: Eric <menwe@amazon.com>

* [Auto Suggest] DQL autosuggest with ANTLR (opensearch-project#7467)

* Antlr autocomplete (opensearch-project#7159)

* dql grammar with rudamentary testing parser

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* show suggestion of fields depending on current index pattern

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* basic code completion with fields populated

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* updated grammar and generated for better group handling

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* add ignored tokens

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* remove console logs

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

---------

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* dql Antlr autocomplete (opensearch-project#7160)

* re-add provider for sql

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* added temporary fix for language providor to appear for more than one language

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

---------

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* remove EOF in parser to fix suggestions

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* use custom version of cursor token index for dql

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* implemented value suggestions based on field

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* set param type

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* update grouping grammar

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* fix grammar for dots in field and value term search with spaces

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* value suggestions match field to avoid failing api call and to find assc keyword field

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* update value suggestions from partially formed value

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* refactor value suggestions and change fieldval listener to visitor

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* implement value suggestions within phrases

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* make grammar more readable

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* rename grammar parser rules

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* bring back minimal autocomplete optimized grammar

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* enable partially complete value suggestion for value groups

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* remove number as lexer rule

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* fix cursor import and clean up

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* fix completion item range to be current word

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* update cursor to use monaco position

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* cursor index to use position directly

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* move language registration into render function to handle new languages

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* include auto closing quotes and parenthesis for dql

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* rename generated file

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* include single line editor closing pairs

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* Changeset file for PR opensearch-project#7391 created/updated

* add license and fix linting

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* modify grammar

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* add tests for fields and keywords

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* move dql test constants to separate file

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* pass core setup from autocomplete constructor to query sugg provider and utilize selectionEnd if no position

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* update an import

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* use updated dataset for index pattern

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

* remove console log

Signed-off-by: Paul Sebastian <paulstn@amazon.com>

---------

Signed-off-by: Paul Sebastian <paulstn@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

* [tests][discover-next] update the tests and async nature of the dataset navigator (opensearch-project#7489)

* [tests][discover-next] update the tests and async nature of the dataset manager

Address test failures related to the dataset navigator.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* bad fingers accidentally hit the x button

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

---------

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

* resolve conflicts

Signed-off-by: Eric <menwe@amazon.com>

* fix one minor linting

Signed-off-by: Eric <menwe@amazon.com>

---------

Signed-off-by: Eric <menwe@amazon.com>
Signed-off-by: Joshua Li <joshuali925@gmail.com>
Signed-off-by: Paul Sebastian <paulstn@amazon.com>
Signed-off-by: Kawika Avilla <kavilla414@gmail.com>
Signed-off-by: Eric Wei <menwe@amazon.com>
Co-authored-by: Joshua Li <joshuali925@gmail.com>
Co-authored-by: Paul Sebastian <paulstn@amazon.com>
Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>
Co-authored-by: Kawika Avilla <kavilla414@gmail.com>
Co-authored-by: Ashwin P Chandran <ashwinpc@amazon.com>
  • Loading branch information
6 people committed Jul 27, 2024
1 parent 8441667 commit 43a0f65
Show file tree
Hide file tree
Showing 17 changed files with 334 additions and 20,973 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,6 @@ snapshots.js

# Yarn local mirror content
.yarn-local-mirror

# Ignore the generated antlr files
/src/plugins/data/public/antlr/opensearch_sql/grammar/.antlr
2 changes: 2 additions & 0 deletions changelogs/fragments/7463.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
feat:
- Add MDS support along with a few cleanup and tests update ([#7463](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7463))
42 changes: 11 additions & 31 deletions src/plugins/data/public/antlr/opensearch_sql/code_completion.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
* SPDX-License-Identifier: Apache-2.0
*/

import { monaco } from 'packages/osd-monaco/target';
import { monaco } from '@osd/monaco';
import { Lexer as LexerType, ParserRuleContext, Parser as ParserType } from 'antlr4ng';
import { CodeCompletionCore } from 'antlr4-c3';
import {
Expand All @@ -23,7 +23,7 @@ import { findCursorTokenIndex } from '../shared/cursor';
import { openSearchSqlAutocompleteData } from './opensearch_sql_autocomplete';
import { SQL_SYMBOLS } from './constants';
import { QuerySuggestion, QuerySuggestionGetFnArgs } from '../../autocomplete';
import { fetchColumnValues, fetchTableSchemas } from '../shared/utils';
import { fetchTableSchemas } from '../shared/utils';

export interface SuggestionParams {
position: monaco.Position;
Expand Down Expand Up @@ -52,60 +52,39 @@ export const getSuggestions = async ({
column: position?.column || selectionEnd,
});

const finalSuggestions = [];
const finalSuggestions = [] as QuerySuggestion[];

try {
// Fetch columns and values
if ('suggestColumns' in suggestions && (suggestions.suggestColumns?.tables?.length ?? 0) > 0) {
const tableNames = suggestions.suggestColumns?.tables?.map((table) => table.name) ?? [];
const schemas = await fetchTableSchemas(tableNames, api, services);
const schemas = await fetchTableSchemas(tableNames, api, dataSetManager);

schemas.forEach((schema) => {
if (schema.body?.fields?.length > 0) {
const columns = schema.body.fields.find((col: any) => col.name === 'COLUMN_NAME');
const fieldTypes = schema.body.fields.find((col: any) => col.name === 'DATA_TYPE');
if (columns && fieldTypes) {
finalSuggestions.push(
...columns.values.map((col: string, index: number) => ({
...columns.values.map((col: string) => ({
text: col,
type: 'field',
fieldType: fieldTypes.values[index],
type: monaco.languages.CompletionItemKind.Field,
}))
);
}
}
});

if (
'suggestValuesForColumn' in suggestions &&
/\S/.test(suggestions.suggestValuesForColumn as string) &&
suggestions.suggestValuesForColumn !== undefined
) {
const values = await fetchColumnValues(
tableNames,
suggestions.suggestValuesForColumn as string,
api,
services
);
values.forEach((value) => {
if (value.body?.fields?.length > 0) {
finalSuggestions.push(
...value.body.fields[0].values.map((colVal: string) => ({
text: `'${colVal}'`,
type: 'value',
}))
);
}
});
}
// later TODO: fetch column values, currently within the industry, it's not a common practice to suggest
// values due to different types of the columns as well as the performance impact. For now just avoid it.
}

// Fill in aggregate functions
if ('suggestAggregateFunctions' in suggestions && suggestions.suggestAggregateFunctions) {
finalSuggestions.push(
...SQL_SYMBOLS.AGREGATE_FUNCTIONS.map((af) => ({
text: af,
type: 'function',
type: monaco.languages.CompletionItemKind.Function,
}))
);
}
Expand All @@ -115,12 +94,13 @@ export const getSuggestions = async ({
finalSuggestions.push(
...(suggestions.suggestKeywords ?? []).map((sk) => ({
text: sk.value,
type: 'keyword',
type: monaco.languages.CompletionItemKind.Keyword,
}))
);
}
} catch (error) {
// TODO: pipe error to the UI
return [];
}

return finalSuggestions;
Expand Down
Loading

0 comments on commit 43a0f65

Please sign in to comment.