docs(administration): add opensearch configuration documentation (rea…

…nahub#209)

docs/administration/configuration/configuring-opensearch/index.md

@@ -0,0 +1,109 @@
+# Configuring OpenSearch
+
+OpenSearch is needed for live job logs functionality, together with FluentBit. It can be deployed using [OpenSearch Helm chart](https://github.com/opensearch-project/helm-charts) as a dependency. Configuration can be passed to the chart by putting it under `opensearch` configuration field in REANA Helm chart. It should first be enabled by changing `opensearch.enabled` value to `true`.
+
+External OpenSearch service can be used instead (recommended).
+
+## Number of instances
+
+Only one OpenSearch instance is deployed by default. To add more instances, each of these instances needs to have its own `PersistentVolumeClaim` and `PersistentVolume` as each instance writes to a data directory with the same name but different contents - this will not work with `reana-shared-persistent-volume` or `reana-infrastructure-persistent-volume`. OpenSearch Helm chart uses `volumeClaimTemplates` to create a separate volume for each instance. It is possible to use `volumeClaimTemplates` with the `StorageClass` of choice (see [Multiple volumes with custom storage class](index.md#multiple-volumes-with-custom-storage-class)).
+
+Change Helm chart vlaues to configure 3 OpenSearch instances:
+
+```yaml
+opensearch:
+  <...>
+  singleNode: false
+  replicas: 3
+  <...>
+```
+
+OpenSearch allows deploying nodes of different types, e. g. `master` and `data`. Current Helm setup does not allow it, and each node has all possible roles, see [default Helm chart roles value](https://github.com/opensearch-project/helm-charts/blob/4253842c1e4d3ac6d4aee294e905c1f20469adc2/charts/opensearch/values.yaml#L16-L20).
+
+## Persistent storage
+
+There are multiple ways to configure persistent strage for OpenSearch.
+
+### Host path
+
+This is the simplest method, but not usable for production, see [Configuring storage volumes](../configuring-storage-volumes/index.md#hostpath).
+
+OpenSearch for REANA is configured with `hostPath` by default.
+
+### Shared/infrastructure volume
+
+To use shared or infrastructure volume, change `opensearch.extraVolumes` parameter in REANA `values.yaml` file:
+
+```yaml
+opensearch:
+  <...>
+  extraVolumes:
+    - name: reana-opensearch-volume
+      persistentVolumeClaim:
+        claimName: reana-shared-persistent-volume  # or reana-infrastructure-persistent-volume
+        readOnly: false
+  <...>
+```
+
+Bear in mind that this only works with one OpenSearch node. To deploy multiple nodes, see [Multiple volumes with custom storage class](index.md#multiple-volumes-with-custom-storage-class).
+
+### Multiple volumes with custom storage class
+
+To deploy [multiple OpenSearch nodes](./index.md#number-of-instances), persistence configuration provided by OpenSearch chart should be used. It employs [`volumeClaimTemplates`](https://github.com/opensearch-project/helm-charts/blob/4253842c1e4d3ac6d4aee294e905c1f20469adc2/charts/opensearch/templates/statefulset.yaml#L27-L53) to create a separate `PersistentVolume` and `PersistentVolumeClaim` for each OpenSearch instance. You need to provide the name of your `StorageClass`, which you should have created as described in [NFS](../configuring-storage-volumes/index.md#nfs) or [CephFS](../configuring-storage-volumes/index.md#cephfs) configuration docs. The name of the `StorageClass` should be `reana-shared-volume-storage-class` or `reana-infrastructure-volume-storage-class`:
+
+```yaml
+opensearch:
+  <...>
+  persistence:
+    enabled: true
+    storageClass: "reana-shared-volume-storage-class"  # or reana-infrastructure-volume-storage-class
+  <...>
+  extraVolumes: []
+  extraVolumeMounts: []
+```
+
+Additionally, you can configure persistence `size`, `annotations` and other attributes, see [docs](https://github.com/opensearch-project/helm-charts/tree/main/charts/opensearch).
+
+Alternatively, `PersistentVolume` and `PersistentVolumeClaim` can be created for each node manually. `PersistentVolumeClaim` should be named as `<clusterName|nameOverride|fullnameOverride>-<nodeGroup>-<clusterName|nameOverride|fullnameOverride>-<nodeGroup>-<N>`, where `N` is a number of an OpenSearch node. The name of each node is either `clusterName-nodeGroup-X`, or `nameOverride-nodeGroup-X` if a `nameOverride` is specified, or `fullnameOverride-X` if a `fullnameOverride` is specified. By default it is `opensearch-cluster-master-0`, `opensearch-cluster-master-1`, etc., hence `PersistentVolumeClaim` names should be `opensearch-cluster-master-opensearch-cluster-master-0`, `opensearch-cluster-master-opensearch-cluster-master-1`, etc. `volumeClaimTemplates` will pick the existing volumes by their names and will not create new ones.
+
+## TLS
+
+OpenSearch uses TLS certificates for communication between nodes (not possible to turn off if using security features, even when only one OpenSearch node is deployed) and for REST API HTTPS (can be disabled). By default both internode communication and REST API use TLS. REST API HTTPS has to be enabled in order to use Basic Authentication for users as in this case username and password in plain text are sent to the server, hence without TLS credentials risk being exposed. You can read more on TLS configuration in [Opensearch documentation](https://opensearch.org/docs/latest/security/configuration/tls).
+
+By default TLS certificates for OpenSearch nodes are generated automatically by a Helm function, similar to how it is done for [Ingress](../configuring-tls-certificates/index.md#default-self-signed-certificate). It is possible for a user to supply his own certificates, by putting them in a secret and mounting to an OpenSearch pod (in `opensearch.secretMounts`). Change `opensearch.generateCerts` to `false` and manually create a secret with TLS certificates, which should have the following structure:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: reana-opensearch-tls-secret
+  namespace: <reana opensearch namespace>
+type: kubernetes.io/tls
+data:
+  tls.crt: <base64 encoded internode public key>
+  tls.key: <base64 encoded internode private key>
+  admin.crt: <base64 encoded admin public key>
+  admin.key: <base64 encoded admin private key>
+  ca.crt: <base64 encoded root CA public key for previous certs>
+```
+
+Common name (CN) of each certificate is used in `opensearch.yml` configuration and by default should be `reana.io` (CA), `opensearch.reana.io` (internode) and `opensearch-admin.reana.io` (admin). If you generate the certificates with different common names, you should change `opensearch.yml` values `plugins.security.nodes_dn` and `plugins.security.admin_dn` accordingly.
+
+Admin TLS certificates provide superadmin permissions, and are stored in the same secret - there is no `admin` user, hence Basic Authentication credentials are not needed when making requests. Bear in mind that admin certificates are also rotated on each Helm update if default automatic TLS certificate generation is enabled.
+
+TLS certificate reload is enabled in the default configuration, which allows updating TLS certificates without restarting OpenSearch instance, see [OpenSearch docs](https://opensearch.org/docs/latest/security/configuration/tls/#hot-reloading-tls-certificates).
+
+## Authentication/authorization
+
+REANA Helm chart configures system users using configuration files in OpenSearch `config/opensearch-security` folder. You can change security configuration by overriding `opensearch.customSecurityConfig` value which creates `internal_users.yaml`, `roles.yaml` and `roles_mapping.yaml` files. This configuration is loaded only once when OpenSearch is first deployed.
+
+!!! warning
+    Subsequent changes to OpenSearch security configuration files will not take effect unless you run `securityadmin.sh` script, which can also delete users that were created via REST API. Read the [documentation on Opensearch security configuration](https://opensearch.org/docs/latest/security/configuration/security-admin/) before attempting to make changes.
+
+By default two users are configured - `reana` and `fluentbit`. Their passwords need to be prepared by first spinning up OpenSearch instance in development environment, connecting to a pod and [running `plugins/opensearch-security/tools/hash.sh` script](https://opensearch.org/docs/latest/security/configuration/yaml/#internal_usersyml):
+
+```console
+kubectl exec statefulset/opensearch-cluster-master -- ./plugins/opensearch-security/tools/hash.sh -p <somepassword>
+```
+
+The generated hashes for the passwords should be supplied to Helm with `--set opensearch.customSecurityConfig.internalUsers.reana.hash='$So$Me$pASsWOrD.HasH' --set opensearch.customSecurityConfig.internalUsers.fluentbit.hash='$So$Me$pASsWOrD.HasH'` options.