Update ES client to canary.37 (#119791)

* upgrade es client to canary 37

* fix error in core

* mute error in test/

* mute incompatible errors

* unskip request_entity_too_large_exception test

* commit autofix

* unskip batch_size_bytes_exceeds_es_content_length test

* fix errors in Core code

* fix or mute errors in data plugin

* fix data_view_management

* fix error index_management

* fix x-pack/test errors

* fix watcher

* fix event_log

* fix data_enhanced

* fix uptime

* fix triggers_actions_ui

* fix transform

* fix reporting

* fix rule_registry

* fix timeline

* fix task_manager

* fix security_solution

* fix rule_registry

* fix fleet

* fix index_management

* fix lens

* fix maps

* fix ml

* fix observability

* bump to canary 8.1-2

* fix error in packages

* fix errors in core

* fix errors in data

* fix errors in discover

* fix some discover tests

* fix errors in telemetry

* fix alerting

* fix errors in apm

* fix errors in event_log

* fix errors in fleet

* fix errors in infra

* fix errors in lists

* mute errors in maps

* fix errors in ml

* fix errors in osquery

* fix errors in security_solution

* fix errors in stack_alerts

* fix errors in task_manager

* fix errors in timelines

* fix errors in transform

* fix errors in watcher

* fix errors in tests

* update docs

* adjust fleet code

* fix problem in fleet

* remove outdated _type from data plugin tests

* fix packages tests

* update type

* fix tests part 2

* Adds product header back to elasticsearch

* Updates API docs

* Fix Discover tests

* update alerting typings

* Fix type errors

* Import Sort type

* Update x-pack/plugins/security_solution/common/endpoint/data_loaders/index_fleet_server.ts

Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>

* Remove double negation

* Fix snapshot conflict issues after merge from "main"

Co-authored-by: Christiane Heiligers <christiane.heiligers@elastic.co>
Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
Co-authored-by: Steph Milovic <stephanie.milovic@elastic.co>
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>

docs/development/core/public/kibana-plugin-core-public.savedobjectsfindoptions.md

@@ -30,7 +30,7 @@ export interface SavedObjectsFindOptions
 |  [searchAfter?](./kibana-plugin-core-public.savedobjectsfindoptions.searchafter.md) | estypes.Id\[\] | <i>(Optional)</i> Use the sort values from the previous page to retrieve the next page of results. |
 |  [searchFields?](./kibana-plugin-core-public.savedobjectsfindoptions.searchfields.md) | string\[\] | <i>(Optional)</i> The fields to perform the parsed query against. See Elasticsearch Simple Query String <code>fields</code> argument for more information |
 |  [sortField?](./kibana-plugin-core-public.savedobjectsfindoptions.sortfield.md) | string | <i>(Optional)</i> |
-|  [sortOrder?](./kibana-plugin-core-public.savedobjectsfindoptions.sortorder.md) | estypes.SearchSortOrder | <i>(Optional)</i> |
+|  [sortOrder?](./kibana-plugin-core-public.savedobjectsfindoptions.sortorder.md) | estypes.SortOrder | <i>(Optional)</i> |
 |  [type](./kibana-plugin-core-public.savedobjectsfindoptions.type.md) | string \| string\[\] |  |
 |  [typeToNamespacesMap?](./kibana-plugin-core-public.savedobjectsfindoptions.typetonamespacesmap.md) | Map&lt;string, string\[\] \| undefined&gt; | <i>(Optional)</i> This map defines each type to search for, and the namespace(s) to search for the type in; this is only intended to be used by a saved object client wrapper. If this is defined, it supersedes the <code>type</code> and <code>namespaces</code> fields when building the Elasticsearch query. Any types that are not included in this map will be excluded entirely. If a type is included but its value is undefined, the operation will search for that type in the Default namespace. |
 

docs/development/core/public/kibana-plugin-core-public.savedobjectsfindoptions.sortorder.md

@@ -7,5 +7,5 @@
 <b>Signature:</b>
 
 ```typescript
-sortOrder?: estypes.SearchSortOrder;
+sortOrder?: estypes.SortOrder;
 ```

docs/development/core/server/kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfoundesunavailableerror.md

@@ -0,0 +1,23 @@
+<!-- Do not edit this file. It is automatically generated by API Documenter. -->
+
+[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md) &gt; [createGenericNotFoundEsUnavailableError](./kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfoundesunavailableerror.md)
+
+## SavedObjectsErrorHelpers.createGenericNotFoundEsUnavailableError() method
+
+<b>Signature:</b>
+
+```typescript
+static createGenericNotFoundEsUnavailableError(type?: string | null, id?: string | null): DecoratedError;
+```
+
+## Parameters
+
+|  Parameter | Type | Description |
+|  --- | --- | --- |
+|  type | string \| null |  |
+|  id | string \| null |  |
+
+<b>Returns:</b>
+
+DecoratedError
+

docs/development/core/server/kibana-plugin-core-server.savedobjectserrorhelpers.md

@@ -18,6 +18,7 @@ export declare class SavedObjectsErrorHelpers
 |  [createBadRequestError(reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.createbadrequesterror.md) | <code>static</code> |  |
 |  [createConflictError(type, id, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.createconflicterror.md) | <code>static</code> |  |
 |  [createGenericNotFoundError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfounderror.md) | <code>static</code> |  |
+|  [createGenericNotFoundEsUnavailableError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfoundesunavailableerror.md) | <code>static</code> |  |
 |  [createIndexAliasNotFoundError(alias)](./kibana-plugin-core-server.savedobjectserrorhelpers.createindexaliasnotfounderror.md) | <code>static</code> |  |
 |  [createInvalidVersionError(versionInput)](./kibana-plugin-core-server.savedobjectserrorhelpers.createinvalidversionerror.md) | <code>static</code> |  |
 |  [createTooManyRequestsError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.createtoomanyrequestserror.md) | <code>static</code> |  |

docs/development/core/server/kibana-plugin-core-server.savedobjectsfindoptions.md

@@ -30,7 +30,7 @@ export interface SavedObjectsFindOptions
 |  [searchAfter?](./kibana-plugin-core-server.savedobjectsfindoptions.searchafter.md) | estypes.Id\[\] | <i>(Optional)</i> Use the sort values from the previous page to retrieve the next page of results. |
 |  [searchFields?](./kibana-plugin-core-server.savedobjectsfindoptions.searchfields.md) | string\[\] | <i>(Optional)</i> The fields to perform the parsed query against. See Elasticsearch Simple Query String <code>fields</code> argument for more information |
 |  [sortField?](./kibana-plugin-core-server.savedobjectsfindoptions.sortfield.md) | string | <i>(Optional)</i> |
-|  [sortOrder?](./kibana-plugin-core-server.savedobjectsfindoptions.sortorder.md) | estypes.SearchSortOrder | <i>(Optional)</i> |
+|  [sortOrder?](./kibana-plugin-core-server.savedobjectsfindoptions.sortorder.md) | estypes.SortOrder | <i>(Optional)</i> |
 |  [type](./kibana-plugin-core-server.savedobjectsfindoptions.type.md) | string \| string\[\] |  |
 |  [typeToNamespacesMap?](./kibana-plugin-core-server.savedobjectsfindoptions.typetonamespacesmap.md) | Map&lt;string, string\[\] \| undefined&gt; | <i>(Optional)</i> This map defines each type to search for, and the namespace(s) to search for the type in; this is only intended to be used by a saved object client wrapper. If this is defined, it supersedes the <code>type</code> and <code>namespaces</code> fields when building the Elasticsearch query. Any types that are not included in this map will be excluded entirely. If a type is included but its value is undefined, the operation will search for that type in the Default namespace. |
 

docs/development/core/server/kibana-plugin-core-server.savedobjectsfindoptions.sortorder.md

@@ -7,5 +7,5 @@
 <b>Signature:</b>
 
 ```typescript
-sortOrder?: estypes.SearchSortOrder;
+sortOrder?: estypes.SortOrder;
 ```

package.json

@@ -105,7 +105,7 @@
     "@elastic/apm-synthtrace": "link:bazel-bin/packages/elastic-apm-synthtrace",
     "@elastic/charts": "40.2.0",
     "@elastic/datemath": "link:bazel-bin/packages/elastic-datemath",
-    "@elastic/elasticsearch": "npm:@elastic/elasticsearch-canary@^8.0.0-canary.35",
+    "@elastic/elasticsearch": "npm:@elastic/elasticsearch-canary@8.1.0-canary.2",
     "@elastic/ems-client": "8.0.0",
     "@elastic/eui": "43.1.1",
     "@elastic/filesaver": "1.1.2",

packages/kbn-es-archiver/src/lib/docs/generate_doc_records_stream.test.ts

@@ -28,7 +28,6 @@ interface SearchResponses {
         total: number;
         hits: Array<{
           _index: string;
-          _type: string;
           _id: string;
           _source: Record<string, unknown>;
         }>;
@@ -60,9 +59,9 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
             hits: {
               total: 5,
               hits: [
-                { _index: 'foo', _type: '_doc', _id: '0', _source: {} },
-                { _index: 'foo', _type: '_doc', _id: '1', _source: {} },
-                { _index: 'foo', _type: '_doc', _id: '2', _source: {} },
+                { _index: 'foo', _id: '0', _source: {} },
+                { _index: 'foo', _id: '1', _source: {} },
+                { _index: 'foo', _id: '2', _source: {} },
               ],
             },
           },
@@ -72,8 +71,8 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
             hits: {
               total: 5,
               hits: [
-                { _index: 'foo', _type: '_doc', _id: '3', _source: {} },
-                { _index: 'foo', _type: '_doc', _id: '4', _source: {} },
+                { _index: 'foo', _id: '3', _source: {} },
+                { _index: 'foo', _id: '4', _source: {} },
               ],
             },
           },
@@ -85,8 +84,8 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
             hits: {
               total: 2,
               hits: [
-                { _index: 'bar', _type: '_doc', _id: '0', _source: {} },
-                { _index: 'bar', _type: '_doc', _id: '1', _source: {} },
+                { _index: 'bar', _id: '0', _source: {} },
+                { _index: 'bar', _id: '1', _source: {} },
               ],
             },
           },
@@ -109,7 +108,6 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
       createMapStream((record: any) => {
         expect(record).toHaveProperty('type', 'doc');
         expect(record.value.source).toEqual({});
-        expect(record.value.type).toBe('_doc');
         expect(record.value.index).toMatch(/^(foo|bar)$/);
         expect(record.value.id).toMatch(/^\d+$/);
         return `${record.value.index}:${record.value.id}`;
@@ -221,7 +219,7 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
 
   describe('keepIndexNames', () => {
     it('changes .kibana* index names if keepIndexNames is not enabled', async () => {
-      const hits = [{ _index: '.kibana_7.16.0_001', _type: '_doc', _id: '0', _source: {} }];
+      const hits = [{ _index: '.kibana_7.16.0_001', _id: '0', _source: {} }];
       const responses = {
         ['.kibana_7.16.0_001']: [{ body: { hits: { hits, total: hits.length } } }],
       };
@@ -245,7 +243,7 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
     });
 
     it('does not change non-.kibana* index names if keepIndexNames is not enabled', async () => {
-      const hits = [{ _index: '.foo', _type: '_doc', _id: '0', _source: {} }];
+      const hits = [{ _index: '.foo', _id: '0', _source: {} }];
       const responses = {
         ['.foo']: [{ body: { hits: { hits, total: hits.length } } }],
       };
@@ -269,7 +267,7 @@ describe('esArchiver: createGenerateDocRecordsStream()', () => {
     });
 
     it('does not change .kibana* index names if keepIndexNames is enabled', async () => {
-      const hits = [{ _index: '.kibana_7.16.0_001', _type: '_doc', _id: '0', _source: {} }];
+      const hits = [{ _index: '.kibana_7.16.0_001', _id: '0', _source: {} }];
       const responses = {
         ['.kibana_7.16.0_001']: [{ body: { hits: { hits, total: hits.length } } }],
       };

packages/kbn-es-archiver/src/lib/docs/generate_doc_records_stream.ts

@@ -65,7 +65,6 @@ export function createGenerateDocRecordsStream({
                 // when it is loaded it can skip migration, if possible
                 index:
                   hit._index.startsWith('.kibana') && !keepIndexNames ? '.kibana_1' : hit._index,
-                type: hit._type,
                 id: hit._id,
                 source: hit._source,
               },

packages/kbn-rule-data-utils/src/alerts_as_data_rbac.ts

@@ -40,10 +40,10 @@ export const isValidFeatureId = (a: unknown): a is ValidFeatureId =>
  * Ref: https://github.com/elastic/elasticsearch/issues/28806#issuecomment-369303620
  *
  * return stringified Long.MAX_VALUE if we receive Number.MAX_SAFE_INTEGER
- * @param sortIds estypes.SearchSortResults | undefined
+ * @param sortIds estypes.SortResults | undefined
  * @returns SortResults
  */
-export const getSafeSortIds = (sortIds: estypes.SearchSortResults | null | undefined) => {
+export const getSafeSortIds = (sortIds: estypes.SortResults | null | undefined) => {
   if (sortIds == null) {
     return sortIds;
   }

src/core/public/public.api.md

@@ -1499,7 +1499,7 @@ export interface SavedObjectsFindOptions {
     // (undocumented)
     sortField?: string;
     // (undocumented)
-    sortOrder?: estypes.SearchSortOrder;
+    sortOrder?: estypes.SortOrder;
     // (undocumented)
     type: string | string[];
     typeToNamespacesMap?: Map<string, string[] | undefined>;

src/core/server/core_usage_data/core_usage_data_service.ts

@@ -13,9 +13,9 @@ import { hasConfigPathIntersection, ChangedDeprecatedPaths } from '@kbn/config';
 
 import { CoreService } from 'src/core/types';
 import { Logger, SavedObjectsServiceStart, SavedObjectTypeRegistry } from 'src/core/server';
-import {
-  AggregationsFiltersAggregate,
-  AggregationsFiltersBucketItem,
+import type {
+  AggregationsMultiBucketAggregateBase,
+  AggregationsSingleBucketAggregateBase,
   SearchTotalHits,
 } from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
 import { CoreContext } from '../core_context';
@@ -75,6 +75,13 @@ const kibanaOrTaskManagerIndex = (index: string, kibanaConfigIndex: string) => {
   return index === kibanaConfigIndex ? '.kibana' : '.kibana_task_manager';
 };
 
+interface UsageDataAggs extends AggregationsMultiBucketAggregateBase {
+  buckets: {
+    disabled: AggregationsSingleBucketAggregateBase;
+    active: AggregationsSingleBucketAggregateBase;
+  };
+}
+
 export class CoreUsageDataService
   implements CoreService<InternalCoreUsageDataSetup, CoreUsageDataStart>
 {
@@ -153,7 +160,10 @@ export class CoreUsageDataService
   private async getSavedObjectAliasUsageData(elasticsearch: ElasticsearchServiceStart) {
     // Note: this agg can be changed to use `savedObjectsRepository.find` in the future after `filters` is supported.
     // See src/core/server/saved_objects/service/lib/aggregations/aggs_types/bucket_aggs.ts for supported aggregations.
-    const { body: resp } = await elasticsearch.client.asInternalUser.search({
+    const { body: resp } = await elasticsearch.client.asInternalUser.search<
+      unknown,
+      { aliases: UsageDataAggs }
+    >({
       index: kibanaIndex,
       body: {
         track_total_hits: true,
@@ -179,10 +189,10 @@ export class CoreUsageDataService
 
     const { hits, aggregations } = resp;
     const totalCount = (hits.total as SearchTotalHits).value;
-    const aggregate = aggregations!.aliases as AggregationsFiltersAggregate;
-    const buckets = aggregate.buckets as Record<string, AggregationsFiltersBucketItem>;
-    const disabledCount = buckets.disabled.doc_count as number;
-    const activeCount = buckets.active.doc_count as number;
+    const aggregate = aggregations!.aliases;
+    const buckets = aggregate.buckets;
+    const disabledCount = buckets.disabled.doc_count;
+    const activeCount = buckets.active.doc_count;
     const inactiveCount = totalCount - disabledCount - activeCount;
 
     return { totalCount, disabledCount, activeCount, inactiveCount };

src/core/server/elasticsearch/client/mocks.ts

@@ -12,6 +12,7 @@ import type { DeeplyMockedKeys } from '@kbn/utility-types/jest';
 import type { PublicKeys } from '@kbn/utility-types';
 import { ElasticsearchClient } from './types';
 import { ICustomClusterClient } from './cluster_client';
+import { PRODUCT_RESPONSE_HEADER } from '../supported_server_response_check';
 
 const omittedProps = [
   'diagnostic',
@@ -22,9 +23,6 @@ const omittedProps = [
   'helpers',
 ] as Array<PublicKeys<KibanaClient>>;
 
-// the product header expected in every response from es
-const PRODUCT_RESPONSE_HEADER = 'x-elastic-product';
-
 // use jest.requireActual() to prevent weird errors when people mock @elastic/elasticsearch
 const { Client: UnmockedClient } = jest.requireActual('@elastic/elasticsearch');
 const createInternalClientMock = (res?: Promise<unknown>): DeeplyMockedKeys<KibanaClient> => {

src/core/server/elasticsearch/index.ts

@@ -42,3 +42,8 @@ export type {
 } from './client';
 export { getRequestDebugMeta, getErrorMessage } from './client';
 export { pollEsNodesVersion } from './version_check/ensure_es_version';
+export {
+  isSupportedEsServer,
+  isNotFoundFromUnsupportedServer,
+  PRODUCT_RESPONSE_HEADER,
+} from './supported_server_response_check';

src/core/server/elasticsearch/supported_server_response_check.test.ts

@@ -0,0 +1,41 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { isNotFoundFromUnsupportedServer } from './supported_server_response_check';
+
+describe('#isNotFoundFromUnsupportedServer', () => {
+  it('returns true with not found response from unsupported server', () => {
+    const rawResponse = {
+      statusCode: 404,
+      headers: {},
+    };
+
+    const result = isNotFoundFromUnsupportedServer(rawResponse);
+    expect(result).toBe(true);
+  });
+
+  it('returns false with not found response from supported server', async () => {
+    const rawResponse = {
+      statusCode: 404,
+      headers: { 'x-elastic-product': 'Elasticsearch' },
+    };
+
+    const result = isNotFoundFromUnsupportedServer(rawResponse);
+    expect(result).toBe(false);
+  });
+
+  it('returns false when not a 404', async () => {
+    const rawResponse = {
+      statusCode: 200,
+      headers: { 'x-elastic-product': 'Elasticsearch' },
+    };
+
+    const result = isNotFoundFromUnsupportedServer(rawResponse);
+    expect(result).toBe(false);
+  });
+});

src/core/server/elasticsearch/supported_server_response_check.ts

@@ -0,0 +1,35 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+export const PRODUCT_RESPONSE_HEADER = 'x-elastic-product';
+/**
+ * Response headers check to determine if the response is from Elasticsearch
+ * @param headers Response headers
+ * @returns boolean
+ */
+// This check belongs to the elasticsearch service as a dedicated helper method.
+export const isSupportedEsServer = (
+  headers: Record<string, string | string[] | undefined> | null | undefined
+) => {
+  return !!headers && headers[PRODUCT_RESPONSE_HEADER] === 'Elasticsearch';
+};
+
+/**
+ * Check to ensure that a 404 response does not come from Elasticsearch
+ *
+ * WARNING: This is a hack to work around for 404 responses returned from a proxy.
+ * We're aiming to minimise the risk of data loss when consumers act on Not Found errors
+ *
+ * @param response response from elasticsearch client call
+ * @returns boolean 'true' if the status code is 404 and the Elasticsearch product header is missing/unexpected value
+ */
+export const isNotFoundFromUnsupportedServer = (args: {
+  statusCode: number | null;
+  headers: Record<string, string | string[] | undefined> | null;
+}): boolean => {
+  return args.statusCode === 404 && !isSupportedEsServer(args.headers);
+};

src/core/server/saved_objects/migrations/actions/integration_tests/actions.test.ts

@@ -331,7 +331,7 @@ describe('migration actions', () => {
               // Allocate 1 replica so that this index stays yellow
               number_of_replicas: '1',
               // Disable all shard allocation so that the index status is red
-              'index.routing.allocation.enable': 'none',
+              index: { routing: { allocation: { enable: 'none' } } },
             },
           },
         })
@@ -395,7 +395,7 @@ describe('migration actions', () => {
               // Allocate 1 replica so that this index stays yellow
               number_of_replicas: '1',
               // Disable all shard allocation so that the index status is red
-              'index.routing.allocation.enable': 'none',
+              index: { routing: { allocation: { enable: 'none' } } },
             },
           },
         })
@@ -1450,7 +1450,7 @@ describe('migration actions', () => {
                 // Allocate 1 replica so that this index stays yellow
                 number_of_replicas: '1',
                 // Disable all shard allocation so that the index status is red
-                'index.routing.allocation.enable': 'none',
+                index: { routing: { allocation: { enable: 'none' } } },
               },
             },
           },
@@ -1567,8 +1567,8 @@ describe('migration actions', () => {
         }
       `);
     });
-    // TODO: unskip after https://github.com/elastic/kibana/issues/116111
-    it.skip('resolves left request_entity_too_large_exception when the payload is too large', async () => {
+
+    it('resolves left request_entity_too_large_exception when the payload is too large', async () => {
       const newDocs = new Array(10000).fill({
         _source: {
           title:

src/core/server/saved_objects/migrations/integration_tests/7_13_0_unknown_types.test.ts

@@ -97,8 +97,7 @@ describe('migration v2', () => {
         { index: '.kibana_7.13.0_001' },
         { meta: true }
       );
-      const settings = response['.kibana_7.13.0_001']
-        .settings as estypes.IndicesIndexStatePrefixedSettings;
+      const settings = response['.kibana_7.13.0_001'].settings as estypes.IndicesIndexSettings;
       expect(settings.index).not.toBeUndefined();
       expect(settings.index!.blocks?.write).not.toEqual('true');
     }