- Added Realm settings
- Added RP/OP configuration classes
- Added realm configuration tests
- Completed the Prepare Auethentication API

Being an RP, we won't need to create JWTs but this will be useful
for testing

- Reworked builder
- Added parser that deserializes and parses a JWT

TODO: Signature validation

- Added support to PemUtils for reading public keys from files
- Added support for converting signatures from DER to JWS and vice
  versa
- Added tests

…ettings

…ic#37523)

Adding the migration guide and removing the deprecated in 6.x
constructor

relates elastic#35560
relates elastic#34488

* SQL: Rename SQL data type DATE to DATETIME

SQL data type DATE has only the date part (e.g.: 2019-01-14)
without any time information. Previously the SQL type DATE was
referring to the ES DATE which contains also the time part along
with TZ information. To conform with SQL data types the data type
`DATE` is renamed to `DATETIME`, since it includes also the time,
as a new runtime SQL `DATE` data type will be introduced down the road,
which only contains the date part and meets the SQL standard.

Closes: elastic#36440

* Address comments

…#37517)

Relates to elastic#37378

Relates to elastic#37567

…c#37560)

This test failed because the refresh at the end of the test is not guaranteed to run before the
indexing is completed, and therefore there's no guarantee that the refresh will free all operations.
This triggers an assertion failure in the test clean-up, which asserts that there are no more pending
operations.

…stic#37563)

This change adds a way to customize how phrase prefix queries should be created
on field types. The match phrase prefix query is exposed in field types in order
to allow optimizations based on the options set on the field.
For instance the text field uses the configured prefix field (if available) to
build a span near that mixes the original field and the prefix field on the last
position.
This change also contains a small refactoring of the match/multi_match query that
simplifies the interactions between the builders.

Closes elastic#31921

Relates elastic#37458

This commit adds a set_priority action to the hot, warm, and cold
phases for an ILM policy. This action sets the `index.priority`
on the managed index to allow different priorities between the
hot, warm, and cold recoveries.

This commit also includes the HLRC and documentation changes.

closes elastic#36905

…36137)

This is a continuation of elastic#28667 and has as goal to convert all executors to propagate errors to the
uncaught exception handler. Notable missing ones were the direct executor and the scheduler. This
commit also makes it the property of the executor, not the runnable, to ensure this property. A big
part of this commit also consists of vastly improving the test coverage in this area.

This commit ensures that we only change refreshListeners to a list if we're actually adding
something to the list.

relates elastic#26701

…#37550)

delete and close index actions threw IllegalArgumentExceptions
when attempting to run against an index that has a snapshot
in progress.

This change introduces a dedicated SnapshotInProgressException
for these scenarios. This is done to explicitly signal to clients that
this is the reason the action failed, and it is a retryable error.

relates to elastic#37541.

* Same as elastic#37464 but for the restore side

This is related to elastic#35975. This commit implements rate limiting on the
follower side using a new class `CombinedRateLimiter`.

Commit elastic#37535 removed an internal restore request in favor of the
RestoreSnapshotRequest. Commit elastic#37449 added a new test that used the
internal restore request. This commit modifies the new test to use the
RestoreSnapshotRequest.

* This was muted needlessly, the problem in elastic#26701 only applies to `6.x`
* Relates elastic#26701

Not all terminals are accessed via SSH.

This change adds deprecation warning to the indices.get_mapping API in case the
"inlcude_type_name" parameter is set to "true" and changes the parsing code in
GetMappingsResponse to parse the type-less response instead of the one
containing types. As a consequence the HLRC client doesn't need to force
"include_type_name=true" any more and the GetMappingsResponseTests can be
adapted to the new format as well. Also removing some "include_type_name"
parameters in yaml test and docs where not necessary.

The "include_type_name" parameter was temporarily introduced in elastic#37285 to facilitate
moving the default parameter setting to "false" in many places in the documentation
code snippets. Most of the places can simply be reverted without causing errors.
In this change I looked for asciidoc files that contained the
"include_type_name=true" addition when creating new indices but didn't look
likey they made use of the "_doc" type for mappings. This is mostly the case
e.g. in the analysis docs where index creating often only contains settings. I
manually corrected the use of types in some places where the docs still used an
explicit type name and not the dummy "_doc" type.

This test was actually passing, for the wrong reason: it asserts a
`MasterNotDiscoveredException` is thrown, expecting this to be due to a failure
to perform state recovery, but in fact it's thrown because the node is not
correctly bootstrapped.

* Fixes `testTwoNodeFirstNodeCleared` by manipulating voting config exclusions.

* Removes `testRecoveryDifferentNodeOrderStartup` since state recovery is now
  handled entirely on the elected master, so the order in which the data nodes
  start is irrelevant.

As of today the Close Index API does its best to close indices,
but closing an index with ongoing recoveries might or might not
be acknowledged depending of the values of the max seq number
and global checkpoint at the time the
TransportVerifyShardBeforeClose action is executed.

These tests failed because they always expect that the index is
correctly closed on the first try, which is not always the case.
Instead we need to retry the closing until it succeed.

Closes elastic#37571

Currently when adding a response header, we do some de-duplication, and
maybe drop the header on the floor if we have reached capacity. Yet, we
still update the thread local tracking the response headers. This is
really expensive because under the hood there is a shared reference that
we synchronize on. In the case of a request processed across many shards
in a tight loop, this contention can be detrimental to performance. We
can avoid updating the thread local in these cases though, when the
response header is duplicate of one that we have already seen, or when
it's dropped on the floor. This commit addresses these performance
issues by avoiding the unnecessary set.

Return an error to the user if the put privilege api is called with
an empty body (no privileges)

Resolves: elastic#37561

ExecuteWatchResponse did not allow unknown fields. This commit fixes the
test and ConstructingObjectParser such that it does now allow unknown
fields. It also creates a new client side test for the response.

Relates elastic#36938

Some tests (e.g. testRestoreIndexWithShardsMissingInLocalGateway) were split-braining since
being switched to Zen2 because the bootstrap setting was left around when nodes got restarted
with data folders wiped.

The test in question here was starting one node (which autobootstrapped to that single node), then
another node. The first node was then shut down (after excluding it from the voting configuration),
its data folder wiped, and restarted. After restart, the node had an empty data folder yet
initial_master_nodes set to itself (i.e. same name). This made the node sometimes form a cluster of
its own, and not rejoin the existing cluster with the other node.

* Add ccr follow info api

This api returns all follower indices and per follower index
the provided parameters at put follow / resume follow time and
whether index following is paused or active.

Closes elastic#37127

* iter

* [DOCS] Edits the get follower info API

* [DOCS] Fixes link to remote cluster

* [DOCS] Clarifies descriptions for configured parameters

This adds deprecation to _type in the script contexts for ingest and update. 
This adds a DeprecationMap that wraps the ctx Map containing _type for these 
specific contexts.

Fixes ``./gradlew eclipse` failure introduced in 6d99e79

Throws an exception if hit extractor tries to retrieve unsupported
object. For example, selecting "a" from `{"a": {"b": "c"}}` now throws
an exception instead of returning null.

Relates to elastic#37364

This commit updates the contribution docs to include java11 as a requirement for building and testing Elasticsearch.

This change fixes the setup of the SSL configuration for the test
openldap realm. The configuration was missing the realm identifier so
the SSL settings being used were just the default JDK ones that do not
trust the certificate of the idp fixture.

See elastic#37591

This commit updates the file docker's entrypoint script looks for when
deciding to process the ELASTIC_PASSWORD env var. The x-pack subdir
of bin no longer exists in 7.0, where the backcompat layer for x-pack
script locations was removed.

closes elastic#37240

This change adds the unfollow action for CCR follower indices.

This is needed for the shrink action in case an index is a follower index.
This will give the follower index the opportunity to fully catch up with
the leader index, pause index following and unfollow the leader index.
After this the shrink action can safely perform the ilm shrink.

The unfollow action needs to be added to the hot phase and acts as
barrier for going to the next phase (warm or delete phases), so that
follower indices are being unfollowed properly before indices are expected
to go in read-only mode. This allows the force merge action to execute
its steps safely.

The unfollow action has three steps:
* `wait-for-indexing-complete` step: waits for the index in question
  to get the `index.lifecycle.indexing_complete` setting be set to `true`
* `wait-for-follow-shard-tasks` step: waits for all the shard follow tasks
  for the index being handled to report that the leader shard global checkpoint
  is equal to the follower shard global checkpoint.
* `pause-follower-index` step: Pauses index following, necessary to unfollow
* `close-follower-index` step: Closes the index, necessary to unfollow
* `unfollow-follower-index` step: Actually unfollows the index using 
  the CCR Unfollow API
* `open-follower-index` step: Reopens the index now that it is a normal index
* `wait-for-yellow` step: Waits for primary shards to be allocated after
  reopening the index to ensure the index is ready for the next step

In the case of the last two steps, if the index in being handled is
a regular index then the steps acts as a no-op.

Relates to elastic#34648

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Gordon Brown <gordon.brown@elastic.co>

From elastic#29453 and elastic#37285, the `include_type_name` parameter was already present and defaulted to false. This PR makes the following updates:
- Add deprecation warnings to `RestPutMappingAction`, plus tests in `RestPutMappingActionTests`.
- Add a typeless 'put mappings' method to the Java HLRC, and deprecate the old typed version. To do this cleanly, I opted to create a new `PutMappingRequest` object that differs from the existing server one.

…37483)

* ML: creating ML State write alias and pointing writes there

* Moving alias check to openJob method

* adjusting concrete index lookup for ml-state

Single bucket aggs are now supported in datafeed aggregation configurations.

* Remove obsolete deprecation checks

This also updates the old-indices check to be appropriate for the 7.x
series of releases, and leaves it as the only deprecation check in
place.

* Add toString to DeprecationIssue

* Bring filterChecks across from 6.x

* License headers

To make further refactoring of GeoGrid aggregations
easier (related: elastic#30320), splitting out these inner
class dependencies into their own files makes it
easier to map the relationship between classes

This commit optimizes some of the performance issues from using
deprecation logging:
 - we optimize encoding the deprecation value
 - we optimize formatting the deprecation string
 - we optimize away getting the current time (by using cached startup
   time)

This is related to elastic#35975. This commit adds timeout functionality to
the local session on a leader node. When a session is started, a timeout
is scheduled using a repeatable runnable. If the session is not accessed
in between two runs the session is closed. When the sssion is closed,
the repeating task is cancelled.

Additionally, this commit moves session uuid generation to the leader
cluster. And renames the PutCcrRestoreSessionRequest to
StartCcrRestoreSessionRequest to reflect that change.

Currently we add the CcrRestoreSourceService as a index event
listener. However, if ccr is disabled, this service is null and we
attempt to add a null listener throwing an exception. This commit only
adds the listener if ccr is enabled.

The subparser in get users allows for unknown fields. This commit sets
the value to true for the parser and modifies the test such that it
accurately tests it.

Relates elastic#36938

With the release of 11.0.2, the old URLs no longer work. This exposed a
few small bugs in the gradle config. One was that --no-cache was not
present in the docker build command, so it was not failing at
first. Then once only the ext.expansions was changed and the docker
build task was not, it was not executing it.

The ML subproject of xpack has a cache for the cpp artifact snapshots
which is checked on each build. The cache is outside of the build dir so
that it is not wiped on a typical clean, as the artifacts can be large
and do not change often. This commit adds a cleanCache task which will
wipe the cache dir, as over time the size of the directory can become
bloated.

Removes all sensitive settings (passwords, auth tokens, urls, etc...) for
watcher notifications accounts. These settings were deprecated (and
herein removed) in favor of their secure sibling that is set inside the
elasticsearch keystore. For example:
`xpack.notification.email.account.<id>.smtp.password`
is no longer a valid setting, and it is replaced by
`xpack.notification.email.account.<id>.smtp.secure_password`

This commit adds

* An OpenID Connect Realm definition
* Necessary OpenID Connect Realm settings to support Authorization code
 grant and Implicit grant flows
* Rest and Transport Action and Request/Response objects for initiating and
 completing the authentication flow
* Functionality for generating OIDC Authentication Request URIs Unit tests

Notably missing (to be handled in subsequent PRs):
* The actual implementation of the authentication flows
* Necessary JW{T,S,E} functionality

Relates: elastic#35339