[6.x] [CCR] Follower index CRUD (#27936) (#30148)

* [CCR] Follower index CRUD (#27936)

* [CCR] Refactor redux for Auto-follow pattern detail panel (#27491)

* [CCR] Refactor redux for Auto-follow pattern detail panel

* [CCR] Small refactor

* [CCR] Change to present tense

* [CCR] Display auto-follow pattern name even if it does not exist

* [CCR] Use href to edit auto-follow pattern + remove middelware to update "pattern" query params

* [CCR] Fix navigation back bug + set 2 ids for detail and edit an auto-follow pattern

* [CCR] Replace api middleware with redux-thunk action

* [CCR] Show detail footer close button even when cluster is not valid

* [CCR] Add endpoints for fetching and creating follower indices (#27646)

* Add GET /follower_indices endpoint with deserialization logic and tests.

* Add POST /follower_indices endpoint with serialization logic and tests.

* [CCR]  Add unit tests for RemoteClusterForm, RemoteClusterList, and RemoteClusterTable (#27647)

* Use componentDidUpdate instead of getDerivedStateFromProps.

* Add unit tests for RemoteClusterForm, RemoteClusterList, and RemoteClusterTable.

* Add jest mock for eui `makeId()` utility and get deterministic aria IDs for snapshots

* Update snapshot for Remote Cluster list test

* [CCR] Follower indices table and detail panel (#27804)

* Store for follower indices

* Initial work for follower indices table and detail panel

* Fix load auto-follow stats load as middleware was removed

* [CCR] Create follower index UI form (#27864)

* Initial setup Follower Index form

* Working form without client validation

* Add client side validation for follower index

* Add client validation to check if index already exist

* Improve error message when leader index does not exist

* Remove update method for follower index

* Clear api error on field change

* Fix i18n error

* Update snapshots

* [CCR] Add pause, resume, and unfollow actions for follower indices (#28305)

* Add pause and resume follower index routes

* Add unfollow route

* Add api methods for new routes

* Adjust routes to have bulk capabilities, add corresponding actions

* Refresh list after pausing/resuming, remove items after unfollowing

* First pass at UI for pause and unfollow (and resume, but that is not visible due to ES stats response)

* Handle additional conditions needed for unfollowing leader index, add placeholder code to deduce paused status

* PR feedback

* [CCR] Advanced settings UI for follower indices (#28267)

* Add client side validation of advanced settings form
* Move form entry row to separate component
* Add server side serialization for advanced settings
* Ignore advanced settings input when that section is hidden.
  - Cache and restore input when the section is shown again.

* [CCR] Show remote cluster validation in CCR forms, and add edit follower index (#28778)

* [CCR] Advanced settings component

* Remove preview active on toggle settings

* Add client side validation of advanced settings form

* Move form entry row to separate component

* Add title to panel

* Add i18n translation of advanced settings

* Update Follower index form with toggle for advanced settings

* Add server side serialisation for advanced settings

* Make code review changes

* Fix test: mock constant dependency

* Add section to edit follower index

* Show confirm modal before updating follower index

* Add edit icon in table + update server endpoint to pause / resume

* [CCR] Show remote cluster validation in follower index form & auto-follow pattern form

* PR feedback, cleanup form sizes, add redirect to edit remote cluster

* Fix routing, remove unused code, adjust auto follow pattern edit loading error page

* Adjust error messages and make remote cluster not found edit page the same

* Fix functionality as result of merge

* Fix validation, reorder actions, fix tests, and address feedback

* PR feedback and fix validation pt 2

* Adjust remote cluster validation

* Fix i18n

* Fix api error not showing on add follower form

* [CCR]  Integrate new follower index info ES API (#29153)

* Integrate new follower index info ES API

* Collate data from follower stats and info apis when retrieving all followers and single follower
* Add follower settings info to detail panel
* Add paused/active UI state
* Surface follower default settings to UI

* Adjust tests

* Address PR feedback

* Update snapshots

* [CCR] Surface license errors in-app and refine permissions error UI. (#29228)

* Fix camelcasing bug in XPackInfo.
* Silently swallow API error when checking for index name availability.
* Fix typo in followerIndexForm fatal error.
* Add permissions check before allowing user to access the app.

* Refine wording of CCR permission denied page, to specify cluster privileges. (#29533)

* [CCR] Improve form error handling and general UX (#29419)

* Remove unnecessary eslint disable-line

* [CCR] Implement Advanced Settings design feedback (#29543)

* Use EuiSwitch to toggle advanced settings in Create Follower Index form.
* Move 'optional' from each Advanced Setting field to the section heading.
* Change Advanced Settings switch label and description to emphasize that you can customize them or use the defaults.
* Prepopulate Advanced Settings fields with default values.
* When editing a follower index, check if advanced settings have been edited and open them if so.
* Add 'Reset to default' button below advanced settings fields if their values are different than the default.
* Remove 'Default' copy from Advanced Settings descriptions.
* Simplify toggleAdvancedSettings function, add comments, and fix React console error.

* [CCR] Follower index list fixes from design feedback (#29484)

* Delete remote cluster settings before updating

* Fix detail panel z-index

* Remove default descriptor from follower index detail panel setting values

* Follower index confirm action copy adjustments

* Change z-index styling to use EUI vars

* [CCR] Improve remote clusters test coverage (#29487)

* Add Jest test for RemoteClusterForm validation state.
* Extract validation functions out of RemoteClusterForm and add unit tests.
  - Return null instead of undefined from validators.
* Add unit tests for different types of remote clusters in RemoteClusterTable.
* Add unit test for RemoteClusterList empty prompt.
* Add tests verifying behavior for row link, row delete button, and detail panel delete button.
  - Use getRouterLinkProps to assign onClick and href to edit buttons in row and detail panel.

* [CCR] Adjust spacing around descriptions in list views, link to transport port docs, etc (#29789)

* Adjust spacing around description around descriptions in list views so that it's even on top and bottom.
* Add link to transport port docs from Remote Cluster form.
* Move 'View in Index Management' link from the detail panel body into the footer.

* Re-order follower index form sections: remote cluster, leader index, follower index. (#29885)

* Fix deep-linking to follower index after creating/updating it. (#29865)

* [CCR] Copy edits (#29676)

* Use 'Resume/pause data replication' in context menu and row actions.
* Update copy of 'Update' confirm modal for a paused follower index.
* Update copy of 'Update' confirm modal for an active follower index.
* Update copy of 'Pause data replication' confirm modal.
* Update copy of 'Resume data replication' confirm modal.
* Update copy for permissions check.
* Update copy of table empty state.
* Update copy around tables.
* Update form copy.
* Update copy for RemoteClustersFormField callouts.
* Convert 'data replication' -> 'replication'.
* Update copy for Unfollow confirm modal.
* Update copy for form API error and Auto-follow Patterns table.
* Update form save button labels to be 'Create' and 'Update'.
* Move API errors to bottom of form, into same position as sync validation errors. Remove spacer from SectionError implementation.

* [CCR] Open index after unfollowing leader (#29887)

* Open index after unfollowing leader, fix some variable names

* Fix typo

* Add comment

* [CCR] IE and Screen reader accesibility (#29731)

* Fix api endpoit for auto-follow stats

* Prevent letter wrapping in IE for the Remote cluster "connection" table column

* Move inline style to CSS class to fix IE flex bug

* [CCR] Add callout to paused follower index detail panel (#30013)

* Add callout to paused follower index detail panel

* Update copy

* Skip call to ccr stats api if follower index is paused (#30027)

* [CCR] Add integration tests for follower indices (#30064)

* [CCR] Add integration tests for follower indices

* Import advanced settings value from app constants

* Disable flaky follower indices API integration tests.

src/ui/public/indices/constants/index.js

@@ -19,4 +19,9 @@
 
 import { INDEX_PATTERN_ILLEGAL_CHARACTERS_VISIBLE } from 'ui/index_patterns';
 
-export const INDEX_ILLEGAL_CHARACTERS_VISIBLE = INDEX_PATTERN_ILLEGAL_CHARACTERS_VISIBLE.concat(',');
+export const INDEX_ILLEGAL_CHARACTERS_VISIBLE = [ ...INDEX_PATTERN_ILLEGAL_CHARACTERS_VISIBLE ];
+
+// Insert the comma into the middle, so it doesn't look as if it has grammatical meaning when
+// these characters are rendered in the UI.
+const insertionIndex = Math.floor(INDEX_PATTERN_ILLEGAL_CHARACTERS_VISIBLE.length / 2);
+INDEX_ILLEGAL_CHARACTERS_VISIBLE.splice(insertionIndex, 0, ',');

x-pack/plugins/cross_cluster_replication/common/constants/base_path.js

@@ -8,3 +8,4 @@ export const BASE_PATH = '/management/elasticsearch/cross_cluster_replication';
 export const BASE_PATH_REMOTE_CLUSTERS = '/management/elasticsearch/remote_clusters';
 export const API_BASE_PATH = '/api/cross_cluster_replication';
 export const API_REMOTE_CLUSTERS_BASE_PATH = '/api/remote_clusters';
+export const API_INDEX_MANAGEMENT_BASE_PATH = '/api/index_management';

x-pack/plugins/cross_cluster_replication/common/constants/index.js

@@ -7,3 +7,4 @@
 export * from './plugin';
 export * from './base_path';
 export * from './app';
+export * from './settings';

x-pack/plugins/cross_cluster_replication/common/constants/settings.js

@@ -0,0 +1,18 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export const FOLLOWER_INDEX_ADVANCED_SETTINGS = {
+  maxReadRequestOperationCount: 5120,
+  maxOutstandingReadRequests: 12,
+  maxReadRequestSize: '32mb',
+  maxWriteRequestOperationCount: 5120,
+  maxWriteRequestSize: '9223372036854775807b',
+  maxOutstandingWriteRequests: 9,
+  maxWriteBufferCount: 2147483647,
+  maxWriteBufferSize: '512mb',
+  maxRetryDelay: '500ms',
+  readPollTimeout: '1m',
+};

x-pack/plugins/cross_cluster_replication/common/services/utils.js

@@ -15,3 +15,15 @@ export const wait = (time = 1000) => (data) => {
     setTimeout(() => resolve(data), time);
   });
 };
+
+/**
+ * Utility to remove empty fields ("") from a request body
+ */
+export const removeEmptyFields = (body) => (
+  Object.entries(body).reduce((acc, [key, value]) => {
+    if (value !== '') {
+      acc[key] = value;
+    }
+    return acc;
+  }, {})
+);

x-pack/plugins/cross_cluster_replication/fixtures/follower_index.js

@@ -0,0 +1,159 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+const Chance = require('chance'); // eslint-disable-line import/no-extraneous-dependencies
+const chance = new Chance();
+
+export const getFollowerIndexStatsMock = (
+  name = chance.string(),
+  shards = [{
+    id: chance.string(),
+    remoteCluster: chance.string(),
+    leaderIndex: chance.string(),
+    leaderGlobalCheckpoint: chance.integer(),
+    leaderMaxSequenceNum: chance.integer(),
+    followerGlobalCheckpoint: chance.integer(),
+    followerMaxSequenceNum: chance.integer(),
+    lastRequestedSequenceNum: chance.integer(),
+    outstandingReadRequestsCount: chance.integer(),
+    outstandingWriteRequestsCount: chance.integer(),
+    writeBufferOperationsCount: chance.integer(),
+    writeBufferSizeBytes: chance.integer(),
+    followerMappingVersion: chance.integer(),
+    followerSettingsVersion: chance.integer(),
+    totalReadTimeMs: chance.integer(),
+    totalReadRemoteExecTimeMs: chance.integer(),
+    successfulReadRequestCount: chance.integer(),
+    failedReadRequestsCount: chance.integer(),
+    operationsReadCount: chance.integer(),
+    bytesReadCount: chance.integer(),
+    totalWriteTimeMs: chance.integer(),
+    successfulWriteRequestsCount: chance.integer(),
+    failedWriteRequestsCount: chance.integer(),
+    operationsWrittenCount: chance.integer(),
+    readExceptions: [ chance.string() ],
+    timeSinceLastReadMs: chance.integer(),
+  }]
+) => {
+  const serializeShard = ({
+    id,
+    remoteCluster,
+    leaderIndex,
+    leaderGlobalCheckpoint,
+    leaderMaxSequenceNum,
+    followerGlobalCheckpoint,
+    followerMaxSequenceNum,
+    lastRequestedSequenceNum,
+    outstandingReadRequestsCount,
+    outstandingWriteRequestsCount,
+    writeBufferOperationsCount,
+    writeBufferSizeBytes,
+    followerMappingVersion,
+    followerSettingsVersion,
+    totalReadTimeMs,
+    totalReadRemoteExecTimeMs,
+    successfulReadRequestCount,
+    failedReadRequestsCount,
+    operationsReadCount,
+    bytesReadCount,
+    totalWriteTimeMs,
+    successfulWriteRequestsCount,
+    failedWriteRequestsCount,
+    operationsWrittenCount,
+    readExceptions,
+    timeSinceLastReadMs,
+  }) => ({
+    shard_id: id,
+    remote_cluster: remoteCluster,
+    leader_index: leaderIndex,
+    leader_global_checkpoint: leaderGlobalCheckpoint,
+    leader_max_seq_no: leaderMaxSequenceNum,
+    follower_global_checkpoint: followerGlobalCheckpoint,
+    follower_max_seq_no: followerMaxSequenceNum,
+    last_requested_seq_no: lastRequestedSequenceNum,
+    outstanding_read_requests: outstandingReadRequestsCount,
+    outstanding_write_requests: outstandingWriteRequestsCount,
+    write_buffer_operation_count: writeBufferOperationsCount,
+    write_buffer_size_in_bytes: writeBufferSizeBytes,
+    follower_mapping_version: followerMappingVersion,
+    follower_settings_version: followerSettingsVersion,
+    total_read_time_millis: totalReadTimeMs,
+    total_read_remote_exec_time_millis: totalReadRemoteExecTimeMs,
+    successful_read_requests: successfulReadRequestCount,
+    failed_read_requests: failedReadRequestsCount,
+    operations_read: operationsReadCount,
+    bytes_read: bytesReadCount,
+    total_write_time_millis: totalWriteTimeMs,
+    successful_write_requests: successfulWriteRequestsCount,
+    failed_write_requests: failedWriteRequestsCount,
+    operations_written: operationsWrittenCount,
+    read_exceptions: readExceptions,
+    time_since_last_read_millis: timeSinceLastReadMs,
+  });
+
+  return {
+    index: name,
+    shards: shards.map(serializeShard),
+  };
+};
+
+export const getFollowerIndexListStatsMock = (total = 3, names) => {
+  const list = {
+    follow_stats: {
+      indices: [],
+    },
+  };
+
+  for(let i = 0; i < total; i++) {
+    list.follow_stats.indices.push(getFollowerIndexStatsMock(names[i]));
+  }
+
+  return list;
+};
+
+export const getFollowerIndexInfoMock = (
+  name = chance.string(),
+  status = chance.string(),
+  parameters = {
+    maxReadRequestOperationCount: chance.string(),
+    maxOutstandingReadRequests: chance.string(),
+    maxReadRequestSize: chance.string(),
+    maxWriteRequestOperationCount: chance.string(),
+    maxWriteRequestSize: chance.string(),
+    maxOutstandingWriteRequests: chance.string(),
+    maxWriteBufferCount: chance.string(),
+    maxWriteBufferSize: chance.string(),
+    maxRetryDelay: chance.string(),
+    readPollTimeout: chance.string(),
+  }
+) => {
+  return {
+    follower_index: name,
+    status,
+    max_read_request_operation_count: parameters.maxReadRequestOperationCount,
+    max_outstanding_read_requests: parameters.maxOutstandingReadRequests,
+    max_read_request_size: parameters.maxReadRequestSize,
+    max_write_request_operation_count: parameters.maxWriteRequestOperationCount,
+    max_write_request_size: parameters.maxWriteRequestSize,
+    max_outstanding_write_requests: parameters.maxOutstandingWriteRequests,
+    max_write_buffer_count: parameters.maxWriteBufferCount,
+    max_write_buffer_size: parameters.maxWriteBufferSize,
+    max_retry_delay: parameters.maxRetryDelay,
+    read_poll_timeout: parameters.readPollTimeout,
+  };
+};
+
+export const getFollowerIndexListInfoMock = (total = 3) => {
+  const list = {
+    follower_indices: [],
+  };
+
+  for(let i = 0; i < total; i++) {
+    list.follower_indices.push(getFollowerIndexInfoMock());
+  }
+
+  return list;
+};

x-pack/plugins/cross_cluster_replication/fixtures/index.js

@@ -10,3 +10,10 @@ export {
 } from './auto_follow_pattern';
 
 export { esErrors } from './es_errors';
+
+export {
+  getFollowerIndexStatsMock,
+  getFollowerIndexListStatsMock,
+  getFollowerIndexInfoMock,
+  getFollowerIndexListInfoMock,
+} from './follower_index';

x-pack/plugins/cross_cluster_replication/public/app/_app.scss

@@ -5,3 +5,10 @@
 .ccrFollowerIndicesHelpText {
   transform: translateY(-3px);
 }
+
+/**
+ * 1. Prevent context menu popover appearing above confirmation modal
+ */
+.ccrFollowerIndicesDetailPanel {
+  z-index: $euiZMask - 1; /* 1 */
+}