Skip to content
This repository has been archived by the owner on Mar 31, 2024. It is now read-only.

Commit

Permalink
Unrevert "Migrations v2: don't auto-create indices + FTR/esArchiver s…
Browse files Browse the repository at this point in the history
…upport (elastic#85778)" (elastic#89992)

* Revert "Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (elastic#85778)""

This reverts commit f979580.

* Fix flaky saved objects management test elastic#89953

* If a clone target exists, wait for yellow, not green, index status

* Fix test after master merge

* Fix types

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
  • Loading branch information
rudolf and kibanamachine authored Feb 6, 2021
1 parent 826a1ec commit fd1d965
Showing 74 changed files with 709 additions and 324 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md) &gt; [createIndexAliasNotFoundError](./kibana-plugin-core-server.savedobjectserrorhelpers.createindexaliasnotfounderror.md)

## SavedObjectsErrorHelpers.createIndexAliasNotFoundError() method

<b>Signature:</b>

```typescript
static createIndexAliasNotFoundError(alias: string): DecoratedError;
```

## Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| alias | <code>string</code> | |

<b>Returns:</b>

`DecoratedError`

Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md) &gt; [decorateIndexAliasNotFoundError](./kibana-plugin-core-server.savedobjectserrorhelpers.decorateindexaliasnotfounderror.md)

## SavedObjectsErrorHelpers.decorateIndexAliasNotFoundError() method

<b>Signature:</b>

```typescript
static decorateIndexAliasNotFoundError(error: Error, alias: string): DecoratedError;
```

## Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| error | <code>Error</code> | |
| alias | <code>string</code> | |

<b>Returns:</b>

`DecoratedError`

Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsErrorHelpers](./kibana-plugin-core-server.savedobjectserrorhelpers.md) &gt; [isGeneralError](./kibana-plugin-core-server.savedobjectserrorhelpers.isgeneralerror.md)

## SavedObjectsErrorHelpers.isGeneralError() method

<b>Signature:</b>

```typescript
static isGeneralError(error: Error | DecoratedError): boolean;
```

## Parameters

| Parameter | Type | Description |
| --- | --- | --- |
| error | <code>Error &#124; DecoratedError</code> | |

<b>Returns:</b>

`boolean`

Original file line number Diff line number Diff line change
@@ -18,6 +18,7 @@ export declare class SavedObjectsErrorHelpers
| [createBadRequestError(reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.createbadrequesterror.md) | <code>static</code> | |
| [createConflictError(type, id, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.createconflicterror.md) | <code>static</code> | |
| [createGenericNotFoundError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.creategenericnotfounderror.md) | <code>static</code> | |
| [createIndexAliasNotFoundError(alias)](./kibana-plugin-core-server.savedobjectserrorhelpers.createindexaliasnotfounderror.md) | <code>static</code> | |
| [createInvalidVersionError(versionInput)](./kibana-plugin-core-server.savedobjectserrorhelpers.createinvalidversionerror.md) | <code>static</code> | |
| [createTooManyRequestsError(type, id)](./kibana-plugin-core-server.savedobjectserrorhelpers.createtoomanyrequestserror.md) | <code>static</code> | |
| [createUnsupportedTypeError(type)](./kibana-plugin-core-server.savedobjectserrorhelpers.createunsupportedtypeerror.md) | <code>static</code> | |
@@ -27,6 +28,7 @@ export declare class SavedObjectsErrorHelpers
| [decorateEsUnavailableError(error, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.decorateesunavailableerror.md) | <code>static</code> | |
| [decorateForbiddenError(error, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.decorateforbiddenerror.md) | <code>static</code> | |
| [decorateGeneralError(error, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.decorategeneralerror.md) | <code>static</code> | |
| [decorateIndexAliasNotFoundError(error, alias)](./kibana-plugin-core-server.savedobjectserrorhelpers.decorateindexaliasnotfounderror.md) | <code>static</code> | |
| [decorateNotAuthorizedError(error, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.decoratenotauthorizederror.md) | <code>static</code> | |
| [decorateRequestEntityTooLargeError(error, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.decoraterequestentitytoolargeerror.md) | <code>static</code> | |
| [decorateTooManyRequestsError(error, reason)](./kibana-plugin-core-server.savedobjectserrorhelpers.decoratetoomanyrequestserror.md) | <code>static</code> | |
@@ -35,6 +37,7 @@ export declare class SavedObjectsErrorHelpers
| [isEsCannotExecuteScriptError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isescannotexecutescripterror.md) | <code>static</code> | |
| [isEsUnavailableError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isesunavailableerror.md) | <code>static</code> | |
| [isForbiddenError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isforbiddenerror.md) | <code>static</code> | |
| [isGeneralError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isgeneralerror.md) | <code>static</code> | |
| [isInvalidVersionError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isinvalidversionerror.md) | <code>static</code> | |
| [isNotAuthorizedError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isnotauthorizederror.md) | <code>static</code> | |
| [isNotFoundError(error)](./kibana-plugin-core-server.savedobjectserrorhelpers.isnotfounderror.md) | <code>static</code> | |
3 changes: 2 additions & 1 deletion packages/kbn-es-archiver/src/actions/empty_kibana_index.ts
Original file line number Diff line number Diff line change
@@ -25,5 +25,6 @@ export async function emptyKibanaIndexAction({

await cleanKibanaIndices({ client, stats, log, kibanaPluginIds });
await migrateKibanaIndex({ client, kbnClient });
return stats;
stats.createdIndex('.kibana');
return stats.toJSON();
}
2 changes: 1 addition & 1 deletion packages/kbn-es-archiver/src/es_archiver.ts
Original file line number Diff line number Diff line change
@@ -155,7 +155,7 @@ export class EsArchiver {
* @return Promise
*/
async emptyKibanaIndex() {
await emptyKibanaIndexAction({
return await emptyKibanaIndexAction({
client: this.client,
log: this.log,
kbnClient: this.kbnClient,
8 changes: 5 additions & 3 deletions packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
Original file line number Diff line number Diff line change
@@ -82,7 +82,9 @@ export async function migrateKibanaIndex({
*/
async function fetchKibanaIndices(client: Client) {
const resp = await client.cat.indices<unknown>({ index: '.kibana*', format: 'json' });
const isKibanaIndex = (index: string) => /^\.kibana(:?_\d*)?$/.test(index);
const isKibanaIndex = (index: string) =>
/^\.kibana(:?_\d*)?$/.test(index) ||
/^\.kibana(_task_manager)?_(pre)?\d+\.\d+\.\d+/.test(index);

if (!Array.isArray(resp.body)) {
throw new Error(`expected response to be an array ${inspect(resp.body)}`);
@@ -115,7 +117,7 @@ export async function cleanKibanaIndices({
while (true) {
const resp = await client.deleteByQuery(
{
index: `.kibana`,
index: `.kibana,.kibana_task_manager`,
body: {
query: {
bool: {
@@ -129,7 +131,7 @@ export async function cleanKibanaIndices({
},
},
{
ignore: [409],
ignore: [404, 409],
}
);

19 changes: 15 additions & 4 deletions src/core/server/saved_objects/migrationsv2/actions/index.ts
Original file line number Diff line number Diff line change
@@ -150,12 +150,23 @@ export const removeWriteBlock = (
.catch(catchRetryableEsClientErrors);
};

const waitForIndexStatusGreen = (
/**
* A yellow index status means the index's primary shard is allocated and the
* index is ready for searching/indexing documents, but ES wasn't able to
* allocate the replicas. When migrations proceed with a yellow index it means
* we don't have as much data-redundancy as we could have, but waiting for
* replicas would mean that v2 migrations fail where v1 migrations would have
* succeeded. It doesn't feel like it's Kibana's job to force users to keep
* their clusters green and even if it's green when we migrate it can turn
* yellow at any point in the future. So ultimately data-redundancy is up to
* users to maintain.
*/
const waitForIndexStatusYellow = (
client: ElasticsearchClient,
index: string
): TaskEither.TaskEither<RetryableEsClientError, {}> => () => {
return client.cluster
.health({ index, wait_for_status: 'green', timeout: '30s' })
.health({ index, wait_for_status: 'yellow', timeout: '30s' })
.then(() => {
return Either.right({});
})
@@ -259,7 +270,7 @@ export const cloneIndex = (
} else {
// Otherwise, wait until the target index has a 'green' status.
return pipe(
waitForIndexStatusGreen(client, target),
waitForIndexStatusYellow(client, target),
TaskEither.map((value) => {
/** When the index status is 'green' we know that all shards were started */
return { acknowledged: true, shardsAcknowledged: true };
@@ -687,7 +698,7 @@ export const createIndex = (
} else {
// Otherwise, wait until the target index has a 'green' status.
return pipe(
waitForIndexStatusGreen(client, indexName),
waitForIndexStatusYellow(client, indexName),
TaskEither.map(() => {
/** When the index status is 'green' we know that all shards were started */
return 'create_index_succeeded';
Loading

0 comments on commit fd1d965

Please sign in to comment.