Merge branch 'main' into ps241106-dontPassThreadpoolToComputer

docs/changelog/114548.yaml

@@ -0,0 +1,5 @@
+pr: 114548
+summary: Support IPinfo database configurations
+area: Ingest Node
+type: enhancement
+issues: []

docs/changelog/114934.yaml

@@ -0,0 +1,6 @@
+pr: 114934
+summary: "[ES|QL] To_DatePeriod and To_TimeDuration return better error messages on\
+  \ `union_type` fields"
+area: ES|QL
+type: bug
+issues: []

docs/reference/how-to/knn-search.asciidoc

@@ -95,13 +95,20 @@ and https://elasticsearch-benchmarks.elastic.co/#tracks/dense_vector[here] some
 of datasets and configurations that we use for our nightly benchmarks.
 
 [discrete]
+[[dense-vector-preloading]]
 include::search-speed.asciidoc[tag=warm-fs-cache]
 
 The following file extensions are used for the approximate kNN search:
+Each extension is broken down by the quantization types.
 
-* `vec` and `veq` for vector values
-* `vex` for HNSW graph
-* `vem`, `vemf`, and `vemq` for metadata
+* `vex` for the HNSW graph
+* `vec` for all non-quantized vector values. This includes all element types: `float`, `byte`, and `bit`.
+* `veq` for quantized vectors indexed with <<dense-vector-quantization,`quantization`>>: `int4` or `int8`
+* `veb` for binary vectors indexed with <<dense-vector-quantization,`quantization`>>: `bbq`
+* `vem`, `vemf`, `vemq`, and `vemb` for metadata, usually small and not a concern for preloading
+
+Generally, if you are using a quantized index, you should only preload the relevant quantized values and the HNSW graph.
+Preloading the raw vectors is not necessary and might be counterproductive.
 
 [discrete]
 === Reduce the number of index segments

docs/reference/index-modules/store.asciidoc

@@ -143,8 +143,8 @@ terms dictionaries, postings lists and points, which are the most important
 parts of the index for search and aggregations.
 
 For vector search, you use <<approximate-knn, approximate k-nearest neighbor search>>,
-you might want to set the setting to vector search files: `["vec", "vex", "vem"]`
-("vec" is used for vector values, "vex" – for HNSW graph, "vem" – for metadata).
+you might want to set the setting to vector search files. See <<dense-vector-preloading, vector preloading>> for a detailed
+list of the files.
 
 Note that this setting can be dangerous on indices that are larger than the size
 of the main memory of the host, as it would cause the filesystem cache to be

docs/reference/ingest/apis/delete-geoip-database.asciidoc → docs/reference/ingest/apis/delete-ip-location-database.asciidoc

@@ -1,30 +1,30 @@
-[[delete-geoip-database-api]]
-=== Delete geoip database configuration  API
+[[delete-ip-location-database-api]]
+=== Delete IP geolocation database configuration API
 ++++
-<titleabbrev>Delete geoip database configuration</titleabbrev>
+<titleabbrev>Delete IP geolocation database configuration</titleabbrev>
 ++++
 
-Deletes a geoip database configuration.
+Deletes a IP geolocation database configuration.
 
 [source,console]
 ----
-DELETE /_ingest/geoip/database/my-database-id
+DELETE /_ingest/ip_location/database/my-database-id
 ----
 // TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
 
 
-[[delete-geoip-database-api-request]]
+[[delete-ip-location-database-api-request]]
 ==== {api-request-title}
 
-`DELETE /_ingest/geoip/database/<database>`
+`DELETE /_ingest/ip_location/database/<database>`
 
-[[delete-geoip-database-api-prereqs]]
+[[delete-ip-location-database-api-prereqs]]
 ==== {api-prereq-title}
 
 * If the {es} {security-features} are enabled, you must have the
 `manage` <<privileges-list-cluster,cluster privilege>> to use this API.
 
-[[delete-geoip-database-api-path-params]]
+[[delete-ip-location-database-api-path-params]]
 ==== {api-path-parms-title}
 
 `<database>`::
@@ -35,21 +35,21 @@ DELETE /_ingest/geoip/database/my-database-id
 --
 
 
-[[delete-geoip-database-api-query-params]]
+[[delete-ip-location-database-api-query-params]]
 ==== {api-query-parms-title}
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
 
 
-[[delete-geoip-database-api-example]]
+[[delete-ip-location-database-api-example]]
 ==== {api-examples-title}
 
 
-[[delete-geoip-database-api-specific-ex]]
-===== Delete a specific geoip database configuration
+[[delete-ip-location-database-api-specific-ex]]
+===== Delete a specific IP geolocation database configuration
 
 [source,console]
 ----
-DELETE /_ingest/geoip/database/example-database-id
+DELETE /_ingest/ip_location/database/example-database-id
 ----
 // TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]

docs/reference/ingest/apis/get-geoip-database.asciidoc → docs/reference/ingest/apis/get-ip-location-database.asciidoc

@@ -1,33 +1,33 @@
-[[get-geoip-database-api]]
-=== Get geoip database configuration API
+[[get-ip-location-database-api]]
+=== Get IP geolocation database configuration API
 ++++
-<titleabbrev>Get geoip database configuration</titleabbrev>
+<titleabbrev>Get IP geolocation database configuration</titleabbrev>
 ++++
 
-Returns information about one or more geoip database configurations.
+Returns information about one or more IP geolocation database configurations.
 
 [source,console]
 ----
-GET /_ingest/geoip/database/my-database-id
+GET /_ingest/ip_location/database/my-database-id
 ----
 // TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
 
 
 
-[[get-geoip-database-api-request]]
+[[get-ip-location-database-api-request]]
 ==== {api-request-title}
 
-`GET /_ingest/geoip/database/<database>`
+`GET /_ingest/ip_location/database/<database>`
 
-`GET /_ingest/geoip/database`
+`GET /_ingest/ip_location/database`
 
-[[get-geoip-database-api-prereqs]]
+[[get-ip-location-database-api-prereqs]]
 ==== {api-prereq-title}
 
 * If the {es} {security-features} are enabled, you must have the
  `manage` <<privileges-list-cluster,cluster privilege>> to use this API.
 
-[[get-geoip-database-api-path-params]]
+[[get-ip-location-database-api-path-params]]
 ==== {api-path-parms-title}
 
 `<database>`::
@@ -38,22 +38,22 @@ supported.
 To get all database configurations, omit this parameter or use `*`.
 
 
-[[get-geoip-database-api-query-params]]
+[[get-ip-location-database-api-query-params]]
 ==== {api-query-parms-title}
 
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
 
-[[get-geoip-database-api-example]]
+[[get-ip-location-database-api-example]]
 ==== {api-examples-title}
 
 
-[[get-geoip-database-api-specific-ex]]
-===== Get information for a specific geoip database configuration
+[[get-ip-location-database-api-specific-ex]]
+===== Get information for a specific IP geolocation database configuration
 
 [source,console]
 ----
-GET /_ingest/geoip/database/my-database-id
+GET /_ingest/ip_location/database/my-database-id
 ----
 // TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
 

docs/reference/ingest/apis/index.asciidoc

@@ -25,23 +25,21 @@ Use the following APIs to get statistics about ingest processing:
 the <<geoip-processor,`geoip` processor>>.
 
 [discrete]
-[[ingest-geoip-database-apis]]
-=== Ingest GeoIP Database APIs
-
-preview::["The commercial IP geolocation database download management APIs are in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."]
+[[ingest-ip-location-database-apis]]
+=== Ingest IP Location Database APIs
 
 Use the following APIs to configure and manage commercial IP geolocation database downloads:
 
-* <<put-geoip-database-api>> to create or update a database configuration
-* <<get-geoip-database-api>> to retrieve a database configuration
-* <<delete-geoip-database-api>> to delete a database configuration
+* <<put-ip-location-database-api>> to create or update a database configuration
+* <<get-ip-location-database-api>> to retrieve a database configuration
+* <<delete-ip-location-database-api>> to delete a database configuration
 
 include::put-pipeline.asciidoc[]
 include::get-pipeline.asciidoc[]
 include::delete-pipeline.asciidoc[]
 include::simulate-pipeline.asciidoc[]
 include::simulate-ingest.asciidoc[]
 include::geoip-stats.asciidoc[]
-include::put-geoip-database.asciidoc[]
-include::get-geoip-database.asciidoc[]
-include::delete-geoip-database.asciidoc[]
+include::put-ip-location-database.asciidoc[]
+include::get-ip-location-database.asciidoc[]
+include::delete-ip-location-database.asciidoc[]

docs/reference/ingest/apis/put-ip-location-database.asciidoc

@@ -0,0 +1,92 @@
+[[put-ip-location-database-api]]
+=== Create or update IP geolocation database configuration API
+++++
+<titleabbrev>Create or update IP geolocation database configuration</titleabbrev>
+++++
+
+Creates or updates an IP geolocation database configuration.
+
+IMPORTANT: The Maxmind `account_id` shown below requires a license key. Because the license key is sensitive information,
+it is stored as a <<secure-settings,secure setting>> in {es} named `ingest.geoip.downloader.maxmind.license_key`. Only
+one Maxmind license key is currently allowed per {es} cluster. A valid license key must be in the secure settings in order
+to download from Maxmind. The license key setting does not take effect until all nodes are restarted or a
+<<cluster-nodes-reload-secure-settings, `reload_secure_settings`>> request is executed.
+
+[source,console]
+----
+PUT _ingest/ip_location/database/my-database-1
+{
+  "name": "GeoIP2-Domain",
+  "maxmind": {
+    "account_id": "1234567"
+  }
+}
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
+
+IMPORTANT: The IPinfo configuration shown below requires a token. Because the token is sensitive information,
+it is stored as a <<secure-settings,secure setting>> in {es} named `ingest.ip_location.downloader.ipinfo.token`. Only
+one IPinfo token is currently allowed per {es} cluster. A valid token must be in the secure settings in order
+to download from IPinfo. The token setting does not take effect until all nodes are restarted or a
+<<cluster-nodes-reload-secure-settings, `reload_secure_settings`>> request is executed.
+
+[source,console]
+----
+PUT _ingest/ip_location/database/my-database-2
+{
+  "name": "standard_location",
+  "ipinfo": {
+  }
+}
+----
+// TEST[skip:we don't want to leak the enterprise-geoip-downloader task, which touching these APIs would cause. Therefore, skip this test.]
+
+
+[[put-ip-location-database-api-request]]
+==== {api-request-title}
+
+`PUT /_ingest/ip_location/database/<database>`
+
+[[put-ip-location-database-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the
+`manage` <<privileges-list-cluster,cluster privilege>> to use this API.
+
+
+[[put-ip-location-database-api-path-params]]
+==== {api-path-parms-title}
+
+`<database>`::
++
+__
+(Required, string) ID of the database configuration to create or update.
+
+[[put-ip-location-database-api-query-params]]
+==== {api-query-parms-title}
+
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
+
+[[put-ip-location-database-api-request-body]]
+==== {api-request-body-title}
+
+// tag::ip-location-database-object[]
+`name`::
+(Required, string)
+The provider-assigned name of the IP geolocation database to download.
+
+`<provider>`::
+(Required, a provider object and its associated configuration)
+The configuration necessary to identify which IP geolocation provider to use to download
+the database, as well as any provider-specific configuration necessary for such downloading.
++
+At present, the only supported providers are `maxmind` and `ipinfo`. The maxmind provider
+requires that an `account_id` (string) is configured. The ipinfo provider does not require
+additional configuration in the request body.
+// end::ip-location-database-object[]
+
+[[ip-location-database-configuration-licensing]]
+==== Licensing
+
+Downloading databases from third party providers is a commercial feature that requires an
+appropriate license. For more information, refer to https://www.elastic.co/subscriptions.