feat(helm): add live logs deployment (#827)

[jlemesh]

Closes #824

OpenSearch

Optionally deploy OpenSearch alongside REANA (enabled: false by default).

In dev environment, deploy one instance with security features off (i. e. no TLS, no authn/authz)

In prod, also only one OpenSearch node is configured. To add more nodes, each of these nodes needs to have its own PersistentVolumeClaim and PersistentVolume as each node writes to a directory with the same name, but different contents - this will not work with current reana-shared-persistent-volume or reana-infrastructure-persistent-volume. It is possible to use volumeClaimTemplates with the StorageClass of choice (configured in OpenSearch Helm chart persistence.storageClass).

Volumes configuration tested with nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner, for one and multiple nodes.

OpenSearch allows deploying nodes of different types, e. g. master and data. This configuration would not allow to do it, we need to add the second opensearch dependency, e. g. opensearch-master, and use the same Helm chart with different configuration. I think it is an overkill at this stage, as this is needed for bigger clusters.

TLS certificates for OpenSearch nodes are generated automatically by a Helm function, similar to how it is done for Ingress, but without automatic rotation on each helm upgrade. It is possible for a user to supply his own certificates, by putting them in a secret and mounting to an OpenSearch pod (in opensearch.secretMounts). Admin TLS certificates provide superadmin permissions to whoever uses them, and are stored in the same secret - there is no specific admin user.

Two users are configured - reana and fluentbit. Their passwords need to be prepared by first spinning up environment with Opensearch and running hash.sh script (described in docs):

./plugins/opensearch-security/tools/hash.sh -p somepassword

After getting the hashes for the passwords, they should be supplied to Helm with --set opensearch.customSecurityConfig.internalUsers.reana.hash='$So$Me$pASsWOrD.HasH' --set opensearch.customSecurityConfig.internalUsers.fluentbit.hash='$So$Me$pASsWOrD.HasH'.

Also enabled SSL cert reload, which allows updating SSL certs without restarting OpenSearch instance. Link to docs provided in the comments for those who will want to use it.

FluentBit

Optionally deploy FluentBit alongside REANA (enabled: false by default). Collects job and workflow logs with tail input plugin and pushes logs to OpenSearch. Allows configuring custom TLS certificates for OpenSearch connection.

reana-dev

The command breaks Helm configuration if it contains long (multiline) strings. PyYAML library first reads the YAML config file, converts multiline strings to one long string delimited with " and then wraps it while dumping, i. e.:

<...>
key: "value\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\n"\
"value\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\n"\
"value\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\nvalue\n"
<...>

If the string is used with Helm template values (tpl function), it adds whitespaces in places where \ was used, which results in malformed configuration for OpenSearch and FluentBit.

dump(width=100000) will add \ only if the string is longer than 100000 characters.

It also requires user to escape certain characters in YAML values like $ and \. PyYAML converts | strings to "" and Helm command is supplied with bad values.

To avoid errors when loading/dumping values.yaml file, skip its loading altogether as there is no need to load it explicitly as it is always loaded by Helm itself. In case releasehelm mode is used for reana-dev cluster-deploy, just use an empty dict for values and add overrides into it, values.yaml will be loaded by default by Helm.

How to test

Checkout reanahub/reana-workflow-controller#602 and reanahub/reana-job-controller#468 to retrieve live logs in REANA components. Checkout and install reanahub/reana-client#731 to watch live logs via CLI.

Dev setup

Deploy REANA as usual and run:

reana-dev run-example -c r-d-helloworld -w serial --submit-only
# forward opensearch port
curl -XGET "http://localhost:9200/fluentbit-job_log/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "kubernetes.labels.job-name.keyword": "<reana run job ID>"
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}' | jq
reana-client logs --workflow helloworld-serial-kubernetes
reana-client logs --workflow helloworld-serial-kubernetes --follow --filter step=hello1

Prod setup

Deploy REANA as usual then run:

helm upgrade reana reana/helm/reana --wait --debug --force \
  --set components.reana_workflow_controller.image=docker.io/reanahub/reana-workflow-controller \
  --set opensearch.customSecurityConfig.internalUsers.reana.hash='$2y$12$e4RWvWxRW0QO8QtnDd.70eV/fJDJ2F7uUo2ox1P07bIpRArYDSUWK' \
  --set opensearch.customSecurityConfig.internalUsers.fluentbit.hash='$2y$12$e4RWvWxRW0QO8QtnDd.70eV/fJDJ2F7uUo2ox1P07bIpRArYDSUWK' \
  --set fluent-bit.outputConfig.httpPasswd='ReanaOS1=' \
  --set opensearch.enabled=true \
  --set fluent-bit.enabled=true \
  --set components.reana_workflow_controller.environment.REANA_OPENSEARCH_ENABLED=true \
  --set components.reana_workflow_controller.environment.REANA_OPENSEARCH_PASSWORD='ReanaOS1=' \
  --set components.reana_db.image=docker.io/library/postgres:14.10
reana-dev run-example -c r-d-helloworld -w serial --submit-only
# forward opensearch port
curl -u "reana:reana" --insecure -XGET "https://localhost:9200/fluentbit-job_log/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "kubernetes.labels.job-name.keyword": "<reana run job ID>"
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}' | jq
reana-client logs --workflow helloworld-serial-kubernetes
reana-client logs --workflow helloworld-serial-kubernetes --follow --filter step=hello1

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 31.08%. Comparing base (0c0849d) to head (467be7a).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #827      +/-   ##
==========================================
+ Coverage   29.80%   31.08%   +1.27%     
==========================================
  Files          26       26              
  Lines        2486     2487       +1     
==========================================
+ Hits          741      773      +32     
+ Misses       1745     1714      -31     

Files with missing lines	Coverage Δ
reana/reana_dev/cluster.py	49.11% <100.00%> (+18.75%)	⬆️

[tiborsimko]

Works nicely 👍

I have squashed the two OpenSearch commits together, slightly rephrased the reana-dev PyPAML commit, and switched off OpenSearch/FluentBit in the developer configuration for now until we put up a fix for that /var/reana/opensearch directory ownership issue. (Just in case anyone may be trying master out there).

diff --git a/helm/configurations/values-dev.yaml b/helm/configurations/values-dev.yaml
index ecf9d7a..e97f0e8 100644
--- a/helm/configurations/values-dev.yaml
+++ b/helm/configurations/values-dev.yaml
@@ -15,7 +15,7 @@ components:
       environment:
         REANA_RUNTIME_KUBERNETES_KEEP_ALIVE_JOBS_WITH_STATUSES: failed
         REANA_OPENSEARCH_USE_SSL: false
-        REANA_OPENSEARCH_ENABLED: false
+        REANA_OPENSEARCH_ENABLED: true
     reana_workflow_engine_cwl:
       image: docker.io/reanahub/reana-workflow-engine-cwl
     reana_workflow_engine_yadage:
@@ -36,7 +36,7 @@ pgbouncer:

 # OpenSearch configuration for dev environment
 opensearch:
-  enabled: false
+  enabled: true
   tls:
     generate: false
   singleNode: true
@@ -67,7 +67,7 @@ opensearch:

 # FluentBit configuration for dev environment
 fluent-bit:
-  enabled: false
+  enabled: true
   outputConfig:
     tls: "Off"
     tlsCaFile: ""