Introducing the concept of ES capabilities #164850

pgayvallet · 2023-08-25T14:31:29Z

Summary

We recently got problems because some index creation settings are rejected by stateless ES, causing the whole system to fail and Kibana to terminate.

We can't really use feature flags for this, given:

it doesn't really make sense to use manual flags for something that strictly depend on one of our dependency's capabilities
we're mixing the concept of "serverless" offering and "serverless" build. Atm we sometimes run "serverless" Kibana against traditional ES, meaning that the "serverless" info cannot be used to determine if we're connected against a default or serverless version of ES.

This was something that was agreed a few weeks back, but never acted upon.

Introducing ES capabilities

This PR introduces the concept of elasticsearch "capabilities".

Those capabilities are built exclusively from info coming from the ES cluster (and not by some config flag).

This first implementation simply exposes a serverless flag, that is populated depending on the build_flavor field of the info API (/ endpoint).

The end goal would be to expose a real capabilities (e.g "what is supported") list instead. But ideally this would be provided by some ES API and not by us guessing what is supported depending on the build flavor, so for now, just exposing whether we're connected to a default of serverless ES will suffice.

Using it to adapt some API calls during SO migration

This PR also adapts the createIndex and cloneIndex migration action to use this information and change their request against ES accordingly (removing some index creation parameters that are not supported).

…fix'

…ch-capabilities

pgayvallet

Self-review

pgayvallet · 2023-08-27T15:29:31Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/elasticsearch_service.ts

+      capabilities = getElasticsearchCapabilities({
+        clusterInfo: await firstValueFrom(this.clusterInfo$!),
+      });


Re-using the clusterInfo$ observable to retrieve the info required to build the capabilities, which avoids to perform an additional call to ES.

pgayvallet · 2023-08-27T15:31:54Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/elasticsearch_service.ts

    }

    return {
      client: this.client!,
      createClient: (type, clientConfig) => this.createClusterClient(type, config, clientConfig),
+      getCapabilities: () => capabilities,


So, technically, capabilities are bound to a given cluster, so a given client. However attaching the API to ClusterClient was way more complex, and made the caching way harder too.

I doubt we're connecting Kibana to multiple clusters, especially mixed serverless/non-serverless ones anytime soon, so I just KISS.

pgayvallet · 2023-08-27T15:32:31Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/get_capabilities.ts

+ *
+ * @internal
+ */
+export const getCapabilitiesFromClient = async (


This one is exposed only for integration tests (at least for now)

pgayvallet · 2023-08-27T15:34:01Z

packages/core/elasticsearch/core-elasticsearch-server-internal/src/get_cluster_info.ts

@@ -28,6 +29,7 @@ export function getClusterInfo$(internalClient: ElasticsearchClient): Observable
      cluster_name: info.cluster_name,
      cluster_uuid: info.cluster_uuid,
      cluster_version: info.version.number,
+      cluster_build_flavor: info.version.build_flavor,


As said previously, I reused the client.info call from this method to build the capabilities, which surfaced the cluster_build_flavor in a few files / snapshots. Overall, I don't think this is a problem so I didn't bother filtering / excluding it in some places.

pgayvallet · 2023-08-27T15:34:46Z

.../elasticsearch/core-elasticsearch-server-internal/src/register_analytics_context_provider.ts

@@ -27,6 +27,7 @@ export function registerAnalyticsContextProvider(
      cluster_name: { type: 'keyword', _meta: { description: 'The Cluster Name' } },
      cluster_uuid: { type: 'keyword', _meta: { description: 'The Cluster UUID' } },
      cluster_version: { type: 'keyword', _meta: { description: 'The Cluster version' } },
+      cluster_build_flavor: { type: 'keyword', _meta: { description: 'The Cluster build flavor' } },


(c.f previous comment)

pgayvallet · 2023-08-27T15:39:50Z

.../core/saved-objects/core-saved-objects-migration-server-internal/src/actions/create_index.ts


+    const indexSettings = {
+      // settings not being supported on serverless ES
+      ...(esCapabilities.serverless
+        ? {}
+        : {
+            // ES rule of thumb: shards should be several GB to 10's of GB, so
+            // Kibana is unlikely to cross that limit.
+            number_of_shards: INDEX_NUMBER_OF_SHARDS,
+            auto_expand_replicas: INDEX_AUTO_EXPAND_REPLICAS,
+            // Set an explicit refresh interval so that we don't inherit the
+            // value from incorrectly configured index templates (not required
+            // after we adopt system indices)
+            refresh_interval: '1s',
+            // Bump priority so that recovery happens before newer indices
+            priority: 10,
+          }),


Adapted the createIndex and cloneIndex actions to use these new capabilites.

Note that this specific work is not necessarily tested - it will be done in a follow up - the intent of the PR was to setup the feature and wire it down to the 2 migration algorithm so that we can effectively use it.

pgayvallet · 2023-08-27T15:43:53Z

packages/core/test-helpers/core-test-helpers-kbn-server/src/create_serverless_root.ts

+      // runServerless doesn't wait until the nodes are up
+      await waitUntilClusterReady(getServerlessESClient());


As explained, es.runServerless returns as soon as the docker containers as running, which causes tests to potentially fail if they need to access ES as soon as startES has been called.

This naive awaiting solves it.

pgayvallet · 2023-08-27T15:45:00Z

packages/core/test-helpers/core-test-helpers-kbn-server/src/create_serverless_root.ts

      return {
+        getClient: getServerlessESClient,


Same, the serverless ES test harness did not expose a client as the traditional one was doing. I'm just creating a default client to localhost:9200 for now, given that apparently the ports aren't configurable when calling es.runServerless yet. We will eventually need to adapt this if the ports become options later.

pgayvallet · 2023-08-27T15:45:45Z

src/core/server/integration_tests/elasticsearch/capabilities.test.ts

+  it('returns the correct capabilities', async () => {
+    const capabilities = await getCapabilitiesFromClient(client);
+    expect(capabilities).toEqual({
+      serverless: false,
+    });


testing capabilities for traditional ES

pgayvallet · 2023-08-27T15:46:28Z

src/core/server/integration_tests/elasticsearch/capabilities_serverless.test.ts

+// skipped because test serverless ES nodes are currently using static ports
+// causing parallel jest runners to fail for obvious port conflicts reasons.
+describe.skip('ES capabilities for serverless ES', () => {


See comment. The test passes locally, but I didn't want to introduce more flakiness right now, so I'm commenting it until the serverless test utils become more mature.

FWIW I am working on adding similar tests for all migration actions here: #164842

Still good to have the capabilities covered more specifically.

elasticmachine · 2023-08-27T15:47:06Z

Pinging @elastic/kibana-core (Team:Core)

rudolf

Looks great!

I guess the capabilities could eventually be used as a type guard to narrow the ES client types to only serverless so that we have more type safety against calling unsupported APIs.

This would also be a good mechanism for ES to communicate the full capabilities that a build flavour supports.

rudolf · 2023-08-27T21:16:49Z

...e/saved-objects/core-saved-objects-migration-server-internal/src/actions/clone_index.test.ts

@@ -51,11 +122,17 @@ describe('cloneIndex', () => {
  });

  it('re-throws non retry-able errors', async () => {
-    const task = setWriteBlock({
+    const task = cloneIndex({


good catch 😨

rudolf · 2023-08-27T21:18:34Z

...s/core/saved-objects/core-saved-objects-migration-server-internal/src/actions/clone_index.ts

+      .clone({
+        index: source,
+        target,
+        wait_for_active_shards: WAIT_FOR_ALL_SHARDS_TO_BE_ACTIVE,


not an index setting so it's not included in the latest stateless-es PR, but I found other docs stating that this option will be removed from several APIs. But given it's not blocked yet as far as I know, we can do that in a follow-up.

Yeah, the next step is to take a closer look at the whole migration and make sure it works with the latest serverless ES builds. I didn't want to do too much in a single PR to avoid the risk of never merging

…ch-capabilities

kibana-ci · 2023-08-28T08:20:07Z

💚 Build Succeeded

Buildkite Build
Commit: 3b746c9

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/core-elasticsearch-server-internal`	33	34	+1
`@kbn/core-elasticsearch-server-mocks`	15	13	-2
`@kbn/core-saved-objects-migration-server-internal`	90	91	+1
total			-0

Unknown metric groups

API count

id	before	after	diff
`@kbn/core-elasticsearch-server`	108	111	+3
`@kbn/core-elasticsearch-server-internal`	37	38	+1
`@kbn/core-elasticsearch-server-mocks`	15	13	-2
`@kbn/core-saved-objects-migration-server-internal`	124	125	+1
total			+3

History

💛 Build #153929 was flaky 0823495
💔 Build #153924 failed 608e834
💔 Build #153914 failed ff51ef0
💔 Build #153901 failed 1ee5853
💔 Build #153899 failed 84f1c0a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jloleysens · 2023-08-28T08:35:36Z

src/core/server/integration_tests/elasticsearch/capabilities_serverless.test.ts

+// skipped because test serverless ES nodes are currently using static ports
+// causing parallel jest runners to fail for obvious port conflicts reasons.
+describe.skip('ES capabilities for serverless ES', () => {


FWIW I am working on adding similar tests for all migration actions here: #164842

Still good to have the capabilities covered more specifically.

Introduce the concept of ES capabilities and use it for SO migration

d0dd190

pgayvallet added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc release_note:skip Skip the PR/issue when compiling release notes v8.11.0 Feature:elasticsearch labels Aug 25, 2023

pgayvallet added 2 commits August 25, 2023 17:04

first batch of fixing tests

88bd7af

second batch of fixing tests

ee1f3b8

pgayvallet force-pushed the kbn-xxx-elasticsearch-capabilities branch from dd441b1 to ee1f3b8 Compare August 25, 2023 15:15

kibanamachine and others added 5 commits August 25, 2023 15:21

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

4661716

…fix'

fix / adapt more things

6ccae02

fix init context

d32f077

fix more more things

6eb8d2d

fix snapshot

1c969f9

pgayvallet mentioned this pull request Aug 26, 2023

Try to make stateless ES work again #164919

Closed

pgayvallet added 2 commits August 27, 2023 10:41

stop using the config, detect from cluster

150e72c

Merge remote-tracking branch 'upstream/main' into kbn-xxx-elasticsear…

9381904

…ch-capabilities

pgayvallet changed the title ~~Introduce the concept of ES capabilities and use it for SO migration~~ Introducing the concept of ES capabilities Aug 27, 2023

pgayvallet added 7 commits August 27, 2023 10:59

fix collector

5603c63

revert auto indent

84f1c0a

renaming is hard

e3422f6

comments are hard too.

1ee5853

updating snapshot

ff51ef0

add integration tests for capabilities

608e834

fix condition with skipStartupConnectionCheck

0823495

pgayvallet commented Aug 27, 2023

View reviewed changes

pgayvallet marked this pull request as ready for review August 27, 2023 15:47

pgayvallet requested a review from a team as a code owner August 27, 2023 15:47

rudolf mentioned this pull request Aug 27, 2023

Don't set refresh interval as user templates can no longer affect it #164372

Closed

9 tasks

rudolf approved these changes Aug 27, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into kbn-xxx-elasticsear…

3b746c9

…ch-capabilities

pgayvallet merged commit 53173f1 into elastic:main Aug 28, 2023

kibanamachine added the backport:skip This commit does not require backporting label Aug 28, 2023

jloleysens approved these changes Aug 28, 2023

View reviewed changes

TinaHeiligers mentioned this pull request Aug 29, 2023

Remove Space Selector from Serverless Top Navigation #164461

Merged

This was referenced Sep 1, 2023

[share] hideEmbed serverless configuration to hide 'Embed code' share option in serverless #162354

Closed

[ML] File upload serverless compatibility #165337

Closed

pgayvallet mentioned this pull request Sep 4, 2023

[Fix] Load sample data in serverless #165157

Merged

1 task

awahab07 mentioned this pull request Mar 25, 2024

[Dataset quality] Hide estimated data in Serverless #179155

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing the concept of ES capabilities #164850

Introducing the concept of ES capabilities #164850

pgayvallet commented Aug 25, 2023 •

edited

Loading

pgayvallet left a comment

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023 •

edited

Loading

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

pgayvallet Aug 27, 2023

jloleysens Aug 28, 2023

elasticmachine commented Aug 27, 2023

rudolf left a comment

rudolf Aug 27, 2023

rudolf Aug 27, 2023

pgayvallet Aug 28, 2023

kibana-ci commented Aug 28, 2023

API count

jloleysens Aug 28, 2023

		// runServerless doesn't wait until the nodes are up
		await waitUntilClusterReady(getServerlessESClient());

Introducing the concept of ES capabilities #164850

Introducing the concept of ES capabilities #164850

Conversation

pgayvallet commented Aug 25, 2023 • edited Loading

Summary

Introducing ES capabilities

Using it to adapt some API calls during SO migration

pgayvallet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet Aug 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 27, 2023

rudolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Aug 28, 2023

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

Choose a reason for hiding this comment

pgayvallet commented Aug 25, 2023 •

edited

Loading

pgayvallet Aug 27, 2023 •

edited

Loading