Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Search] Remove long-running query pop-up #75385

Merged
merged 115 commits into from
Sep 13, 2020
Merged
Changes from 1 commit
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
9ce2275
[Search] Remove long-running query pop-up
lukasolson Aug 19, 2020
b647c34
Merge branch 'master' into search-remove-popup
lukasolson Aug 20, 2020
c1217b0
Don't timeout if requestTimeout isn't configured
lukasolson Aug 20, 2020
3299406
Merge branch 'master' into search-remove-popup
lukasolson Aug 20, 2020
b9aa07d
Remove unused kibanaUtils
lukasolson Aug 20, 2020
c2c0237
Remove unused kibanaReact
lukasolson Aug 21, 2020
7927af6
Re-add reference to kibanaUtils
lukasolson Aug 21, 2020
0f69bac
Remove unused translations and update documentation
lukasolson Aug 21, 2020
c29ab1b
Add new x-pack advanced setting searchTimeout and use it in the Enhan…
Aug 23, 2020
4f603ce
docs
Aug 23, 2020
13ea90e
Merge branch 'master' into search-remove-popup
lukasolson Aug 24, 2020
2df19ac
Merge branch 'master' of github.com:elastic/kibana into search/search…
Aug 25, 2020
25aae3a
Re-add toast when queries time out
lukasolson Aug 25, 2020
849ef85
Fix types
lukasolson Aug 25, 2020
de8107d
Update error message with capabilities
lukasolson Aug 25, 2020
d54b09a
Update docs
lukasolson Aug 25, 2020
20ea649
Update docs
lukasolson Aug 25, 2020
356281a
Move search server routes into a directory.
lukeelmers Aug 20, 2020
84ad62f
Add internal/_msearch route.
lukeelmers Aug 26, 2020
9a8de03
Remove legacy search API, rewrite default search strategy to use inte…
lukeelmers Aug 26, 2020
371a4df
Remove legacy es_client code.
lukeelmers Aug 26, 2020
2475467
Handle msearch options on server.
lukeelmers Aug 26, 2020
775586a
Remove elasticsearch-browser dependency.
lukeelmers Aug 26, 2020
d7df81e
Update generated docs.
lukeelmers Aug 26, 2020
e02a9dc
Merge branch 'master' of github.com:elastic/kibana into search/search…
Aug 26, 2020
2cb19ec
Merge remote-tracking branch 'lukeelmers/fix/remove-es-client' into s…
Aug 26, 2020
c07d923
Rely on server timeout in OSS (?)
Aug 26, 2020
1086b9c
Merge branch 'master' into search/search-timeout-setting
elasticmachine Aug 26, 2020
184a3e6
Merge branch 'master' of github.com:elastic/kibana into search/search…
Aug 26, 2020
8c75c64
Rename function
Aug 26, 2020
5b1af34
Merge branch 'master' of github.com:elastic/kibana into search/search…
Aug 26, 2020
17fd210
Merge branch 'search/search-timeout-setting' of github.com:lizozom/ki…
Aug 26, 2020
39605fe
Merge branch 'master' of github.com:elastic/kibana into search/search…
Aug 26, 2020
742f3ba
Add features to dependencies
lukasolson Aug 26, 2020
4056f40
Undefined check
lukasolson Aug 26, 2020
791a92f
Merge branch 'master' into fix/remove-es-client
elasticmachine Aug 26, 2020
82915ab
Merge branch 'master' into search/search-timeout-setting
elasticmachine Aug 27, 2020
059706e
doc
Aug 27, 2020
260d23c
Code review fixes
Aug 27, 2020
9540e4c
code review
Aug 27, 2020
6de4205
doc
Aug 27, 2020
5049b35
loading count
Aug 27, 2020
002c13c
simplify code review and fix jest tets
Aug 27, 2020
3c07a71
Merge branch 'master' of github.com:elastic/kibana into pr/75943
Aug 27, 2020
1dded0a
type check
Aug 27, 2020
75c1eb6
Merge branch 'master' of github.com:elastic/kibana into search/search…
Aug 27, 2020
a6a128f
Merge remote-tracking branch 'lukeelmers/fix/remove-es-client' into s…
Aug 27, 2020
05e9bdf
Remove esShard from client
Aug 27, 2020
8dd8f95
Merge branch 'master' into fix/remove-es-client
elasticmachine Aug 31, 2020
1dab018
Merge remote-tracking branch 'lukeelmers/fix/remove-es-client' into s…
Aug 31, 2020
dc19863
cleanup request parameters from FE
Aug 31, 2020
0499b21
doc
Aug 31, 2020
05c51b8
doc
Aug 31, 2020
39fc3fe
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 1, 2020
735ac5c
Align request parameters on server,
Sep 1, 2020
0edd897
docs
Sep 1, 2020
34a008f
add management docs
Sep 1, 2020
f9fab22
docs
Sep 1, 2020
eaa9c77
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 1, 2020
a601601
Remove import
Sep 1, 2020
cb1951e
Break circular dep + fix msearch test
Sep 1, 2020
78ea8a1
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 1, 2020
4921152
Remove deleted type
Sep 1, 2020
0b954df
Fix jest
Sep 1, 2020
5e2417d
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 2, 2020
3f8b19a
Bring toSnakeCase back
Sep 2, 2020
b000274
docs
Sep 2, 2020
504814d
fix jest
Sep 2, 2020
e520695
Merge branch 'search-timeout-setting' into search-remove-popup
lukasolson Sep 3, 2020
5e0959a
Add new x-pack advanced setting searchTimeout and use it in the Enhan…
Aug 23, 2020
a682fb7
docs
Aug 23, 2020
f2c4c16
Rely on server timeout in OSS (?)
Aug 26, 2020
2497e00
Rename function
Aug 26, 2020
a65596f
doc
Aug 27, 2020
1e9e9a0
Remove esShard from client
Aug 27, 2020
6d718c5
cleanup request parameters from FE
Aug 31, 2020
060ce08
doc
Aug 31, 2020
0310fb1
doc
Aug 31, 2020
1b4770a
Align request parameters on server,
Sep 1, 2020
70b5809
docs
Sep 1, 2020
f8aba5d
add management docs
Sep 1, 2020
f4d0064
docs
Sep 1, 2020
9c5c0ec
Remove import
Sep 1, 2020
befdf10
Break circular dep + fix msearch test
Sep 1, 2020
098a9ad
Remove deleted type
Sep 1, 2020
dac9d7a
Fix jest
Sep 1, 2020
f9273d6
Bring toSnakeCase back
Sep 2, 2020
3dc393e
docs
Sep 2, 2020
59f5cc1
fix jest
Sep 2, 2020
2841d57
Fix merge
Sep 3, 2020
313bbd7
Fix types
lukasolson Sep 5, 2020
ae3f8b6
Merge branch 'search/search-timeout-setting' into search-remove-popup
lukasolson Sep 5, 2020
2dea1e3
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 6, 2020
68c2604
Merge branch 'search/search-timeout-setting' into pr/75385
Sep 6, 2020
6111ac7
Allow timeout to be undefined
Sep 6, 2020
b1b466c
Fix jest test
Sep 6, 2020
5a7c9e0
Upldate docs
Sep 6, 2020
882c4fd
Fix msearch jest
Sep 6, 2020
6e91b8f
Merge branch 'search/search-timeout-setting' into pr/75385
Sep 6, 2020
95f0e9b
Merge correction
Sep 6, 2020
b9306c0
Merge branch 'master' into search/search-timeout-setting
elasticmachine Sep 7, 2020
bec86a5
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 7, 2020
887b3ea
docs
Sep 7, 2020
b9c7dfc
Merge branch 'search/search-timeout-setting' of github.com:lizozom/ki…
Sep 8, 2020
4bdbcc6
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 8, 2020
f6d4b5b
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 8, 2020
4405680
Fix rollup search merge
Sep 8, 2020
61d1f76
Merge branch 'master' of github.com:elastic/kibana into search/search…
Sep 8, 2020
790f2b0
Merge branch 'search/search-timeout-setting' into pr/75385
Sep 8, 2020
069390b
Merge branch 'master' of github.com:elastic/kibana into pr/75385
Sep 9, 2020
25d5e37
Fix merge
Sep 9, 2020
bc6eaae
Merge branch 'master' into search-remove-popup
elasticmachine Sep 9, 2020
c6508b0
Use i18n
lukasolson Sep 10, 2020
4ed7d31
Merge remote-tracking branch 'origin/search-remove-popup' into search…
lukasolson Sep 10, 2020
66dac98
Merge branch 'master' into search-remove-popup
lukasolson Sep 13, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cleanup request parameters from FE
  • Loading branch information
Liza K committed Sep 3, 2020
commit 6d718c50edaff8a180676ec9fbb07b19a0f5efaf
12 changes: 0 additions & 12 deletions src/plugins/data/public/search/fetch/get_search_params.test.ts
Original file line number Diff line number Diff line change
@@ -25,18 +25,6 @@ function getConfigStub(config: any = {}): GetConfigFn {
}

describe('getSearchParams', () => {
test('includes rest_total_hits_as_int', () => {
const config = getConfigStub();
const searchParams = getSearchParams(config);
expect(searchParams.rest_total_hits_as_int).toBe(true);
});

test('includes ignore_unavailable', () => {
const config = getConfigStub();
const searchParams = getSearchParams(config);
expect(searchParams.ignore_unavailable).toBe(true);
});

test('includes ignore_throttled according to search:includeFrozen', () => {
let config = getConfigStub({ [UI_SETTINGS.SEARCH_INCLUDE_FROZEN]: true });
let searchParams = getSearchParams(config);
2 changes: 0 additions & 2 deletions src/plugins/data/public/search/fetch/get_search_params.ts
Original file line number Diff line number Diff line change
@@ -24,8 +24,6 @@ const sessionId = Date.now();

export function getSearchParams(getConfig: GetConfigFn) {
return {
rest_total_hits_as_int: true,
ignore_unavailable: true,
ignore_throttled: getIgnoreThrottled(getConfig),
max_concurrent_shard_requests: getMaxConcurrentShardRequests(getConfig),
preference: getPreference(getConfig),
1 change: 1 addition & 0 deletions src/plugins/data/server/index.ts
Original file line number Diff line number Diff line change
@@ -212,6 +212,7 @@ export {
ISearchSetup,
ISearchStart,
getDefaultSearchParams,
getShardTimeout,
getTotalLoaded,
usageProvider,
SearchUsage,
Original file line number Diff line number Diff line change
@@ -19,9 +19,13 @@

import { SharedGlobalConfig } from '../../../../../core/server';

export function getShardTimeout(config: SharedGlobalConfig) {
return `${config.elasticsearch.shardTimeout.asMilliseconds()}ms`;
}

export function getDefaultSearchParams(config: SharedGlobalConfig) {
return {
timeout: `${config.elasticsearch.shardTimeout.asMilliseconds()}ms`,
timeout: getShardTimeout(config),
ignoreUnavailable: true, // Don't fail if the index/indices don't exist
restTotalHitsAsInt: true, // Get the number of hits as an int rather than a range
};
2 changes: 1 addition & 1 deletion src/plugins/data/server/search/es_search/index.ts
Original file line number Diff line number Diff line change
@@ -19,5 +19,5 @@

export { ES_SEARCH_STRATEGY, IEsSearchRequest, IEsSearchResponse } from '../../../common/search';
export { esSearchStrategyProvider } from './es_search_strategy';
export { getDefaultSearchParams } from './get_default_search_params';
export { getDefaultSearchParams, getShardTimeout } from './get_default_search_params';
export { getTotalLoaded } from './get_total_loaded';
2 changes: 1 addition & 1 deletion src/plugins/data/server/search/index.ts
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@

export { ISearchStrategy, ISearchSetup, ISearchStart, SearchEnhancements } from './types';

export { getDefaultSearchParams, getTotalLoaded } from './es_search';
export { getDefaultSearchParams, getTotalLoaded, getShardTimeout } from './es_search';

export { usageProvider, SearchUsage } from './collectors';

4 changes: 2 additions & 2 deletions src/plugins/data/server/search/routes/msearch.ts
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@ import { schema } from '@kbn/config-schema';
import { IRouter } from 'src/core/server';
import { UI_SETTINGS } from '../../../common';
import { SearchRouteDependencies } from '../search_service';
import { getDefaultSearchParams } from '..';
import { getShardTimeout } from '..';

interface MsearchHeaders {
index: string;
@@ -96,7 +96,7 @@ export function registerMsearchRoute(router: IRouter, deps: SearchRouteDependenc

// get shardTimeout
const config = await deps.globalConfig$.pipe(first()).toPromise();
const { timeout } = getDefaultSearchParams(config);
const timeout = getShardTimeout(config);

const body = convertRequestBody(request.body, { timeout });

Original file line number Diff line number Diff line change
@@ -5,8 +5,8 @@
*/

import { RequestHandlerContext } from '../../../../../src/core/server';
import { pluginInitializerContextConfigMock } from '../../../../../src/core/server/mocks';
import { enhancedEsSearchStrategyProvider } from './es_search_strategy';
import { BehaviorSubject } from 'rxjs';

const mockAsyncResponse = {
body: {
@@ -43,7 +43,15 @@ describe('ES search strategy', () => {
elasticsearch: { client: { asCurrentUser: { transport: { request: mockApiCaller } } } },
},
};
const mockConfig$ = pluginInitializerContextConfigMock<any>({}).legacy.globalConfig$;
const mockConfig$ = new BehaviorSubject<any>({
elasticsearch: {
shardTimeout: {
asMilliseconds: () => {
return 100;
},
},
},
});

beforeEach(() => {
mockApiCaller.mockClear();
@@ -64,10 +72,17 @@ describe('ES search strategy', () => {
await esSearch.search((mockContext as unknown) as RequestHandlerContext, { params });

expect(mockApiCaller).toBeCalled();
const { method, path, body } = mockApiCaller.mock.calls[0][0];

const { method, path, body, querystring } = mockApiCaller.mock.calls[0][0];

expect(method).toBe('POST');
expect(path).toBe('/logstash-*/_async_search');
expect(body).toEqual({ query: {} });
expect(querystring).toHaveProperty('wait_for_completion_timeout');
expect(querystring).toHaveProperty('track_total_hits');
expect(querystring).toHaveProperty('ignore_unavailable');
expect(querystring).toHaveProperty('keep_alive');
expect(querystring).toHaveProperty('batched_reduce_size');
});

it('makes a GET request to async search with ID when ID is provided', async () => {
@@ -79,10 +94,13 @@ describe('ES search strategy', () => {
await esSearch.search((mockContext as unknown) as RequestHandlerContext, { id: 'foo', params });

expect(mockApiCaller).toBeCalled();
const { method, path, body } = mockApiCaller.mock.calls[0][0];
const { method, path, body, querystring } = mockApiCaller.mock.calls[0][0];
expect(method).toBe('GET');
expect(path).toBe('/_async_search/foo');
expect(body).toEqual(undefined);
expect(querystring).not.toHaveProperty('batched_reduce_size');
expect(querystring).not.toHaveProperty('batched_reduce_size');
expect(querystring).not.toHaveProperty('batched_reduce_size');
});

it('encodes special characters in the path', async () => {
46 changes: 21 additions & 25 deletions x-pack/plugins/data_enhanced/server/search/es_search_strategy.ts
Original file line number Diff line number Diff line change
@@ -15,10 +15,10 @@ import {
Logger,
} from '../../../../../src/core/server';
import {
getDefaultSearchParams,
getTotalLoaded,
ISearchStrategy,
SearchUsage,
getDefaultSearchParams,
} from '../../../../../src/plugins/data/server';
import { IEnhancedEsSearchRequest } from '../../common';
import { shimHitsTotal } from './shim_hits_total';
@@ -38,18 +38,16 @@ export const enhancedEsSearchStrategyProvider = (
request: IEnhancedEsSearchRequest,
options?: ISearchOptions
) => {
logger.debug(`search ${JSON.stringify(request.params) || request.id}`);
const config = await config$.pipe(first()).toPromise();
const client = context.core.elasticsearch.client.asCurrentUser;
const defaultParams = getDefaultSearchParams(config);
const params = { ...defaultParams, ...request.params };

logger.debug(`search ${JSON.stringify(request.params) || request.id}`);

const isAsync = request.indexType !== 'rollup';

try {
const response = isAsync
? await asyncSearch(client, { ...request, params }, options)
: await rollupSearch(client, { ...request, params }, options);
? await asyncSearch(client, request)
: await rollupSearch(client, request, config$);

if (
usage &&
@@ -81,32 +79,26 @@ export const enhancedEsSearchStrategyProvider = (

async function asyncSearch(
client: ElasticsearchClient,
request: IEnhancedEsSearchRequest,
options?: ISearchOptions
request: IEnhancedEsSearchRequest
): Promise<IEsSearchResponse> {
const { timeout = undefined, restTotalHitsAsInt = undefined, ...params } = {
...request.params,
};

params.trackTotalHits = true; // Get the exact count of hits
const { params = {} } = request;

// If we have an ID, then just poll for that ID, otherwise send the entire request body
const { body = undefined, index = undefined, ...queryParams } = request.id ? {} : params;

const method = request.id ? 'GET' : 'POST';
const path = encodeURI(request.id ? `/_async_search/${request.id}` : `/${index}/_async_search`);

// Only report partial results every 64 shards; this should be reduced when we actually display partial results
const batchedReduceSize = request.id ? undefined : 64;

const asyncOptions = {
const querystring = toSnakeCase({
waitForCompletionTimeout: '100ms', // Wait up to 100ms for the response to return
keepAlive: '1m', // Extend the TTL for this search request by one minute
};

const querystring = toSnakeCase({
...asyncOptions,
...(batchedReduceSize && { batchedReduceSize }),
...(request.id
? {}
: {
trackTotalHits: true,
ignoreUnavailable: true,
batchedReduceSize: 64, // Only report partial results every 64 shards; this should be reduced when we actually display partial results
}),
...queryParams,
});

@@ -131,12 +123,16 @@ async function asyncSearch(
async function rollupSearch(
client: ElasticsearchClient,
request: IEnhancedEsSearchRequest,
options?: ISearchOptions
config$: Observable<SharedGlobalConfig>
): Promise<IEsSearchResponse> {
const config = await config$.pipe(first()).toPromise();
const { body, index, ...params } = request.params!;
const method = 'POST';
const path = encodeURI(`/${index}/_rollup_search`);
const querystring = toSnakeCase(params);
const querystring = toSnakeCase({
...getDefaultSearchParams(config),
...params,
});

const esResponse = await client.transport.request({
method,