[BUG][opensearch/security] Certificates can not be generated due to allegedly missing java

[thannaske]

Describe the bug
When trying to bootstrap a new OpenSearch cluster on Ubuntu 20.04 using the latest main version of this repository, the process is stuck during the initial deployment:

TASK [linux/opensearch : Security Plugin configuration | Generate the node & admin certificates in local] ***
fatal: [os-master-1]: FAILED! => {"changed": true, "cmd": ["/tmp/opensearch-nodecerts/tools/sgtlstool.sh", "-c", "/tmp/opensearch-nodecerts/config/tlsconfig.yml", "-ca", "-crt", "-t", "/tmp/opensearch-nodecerts/config/"], "delta": "0:00:00.005591", "end": "2023-08-07 17:22:33.094310", "msg": "non-zero return code", "rc": 127, "start": "2023-08-07 17:22:33.088719", "stderr": "/tmp/opensearch-nodecerts/tools/sgtlstool.sh: line 11: /usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java: No such file or directory", "stderr_lines": ["/tmp/opensearch-nodecerts/tools/sgtlstool.sh: line 11: /usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java: No such file or directory"], "stdout": "", "stdout_lines": []}

As you can see, the script complains about not finding the java binary. However, when connecting to the server und running the java binary manually, everything works as intended:

os-master-1~ # echo $JAVA_HOME
/usr/lib/jvm/java-1.17.0-openjdk-amd64
os-master-1~ # /usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java --version
openjdk 17.0.8 2023-07-18
OpenJDK Runtime Environment (build 17.0.8+7-Ubuntu-120.04.2)
OpenJDK 64-Bit Server VM (build 17.0.8+7-Ubuntu-120.04.2, mixed mode, sharing)

To Reproduce
Steps to reproduce the behavior:

1. Install all required servers with a minimal Ubuntu 20.04
2. Install OpenJDK 17 (JRE/JDK) and make sure the JAVA_HOME is in the environment
3. Clone the latest version of opensearch-project/ansible-playbook and execute the playbook according to the README

Expected behavior
The error shouldn't occur as Java is not only properly installed but also the JAVA_HOME variable is properly set and the java binary works as intended.

Playbook Name
opensearch/tasks/security.yaml

Host/Environment (please complete the following information):

• Ansible Version: 2.10.8
• Playbook Version: 2.9.0
• Java Version: openjdk 17.0.8 2023-07-18
• Ubuntu Version: 20.04 LTS

Additional context
Executed command:

bastion~ # /opt/opensearch-ansible (main*=) # ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=<PW> kibanaserver_password=<PW>"

Inventory:

os-master-1 ip=10.33.0.48 roles=data,master
os-master-2 ip=10.33.0.49 roles=data,master
os-master-3 ip=10.33.0.50 roles=data,master
os-data-1 ip=10.33.0.51 roles=data,ingest
os-data-2 ip=10.33.0.52 roles=data,ingest
os-data-3 ip=10.33.0.53 roles=data,ingest

# List all the nodes in the os cluster
[os-cluster]
os-master-1
os-master-2
os-master-3
os-data-1
os-data-2
os-data-3

# List all the Master eligible nodes under this group
[master]
os-master-1
os-master-2
os-master-3

[dashboards]
os-master-1

[zelinh]

[Triage] Could you provide us which commit you are using to reproduce this issue?
cc: @saravanan30erd Could you provide any help here? Thanks!

[kostyasimanov]

It's because used 'local_action' in security task to generate certificates.
Then, task with 'local_action' plays on your ansible host, not deployed server.

Java with JAVA_HOME must be on your ansible host to pass the task.

[thannaske]

Ah, that was indeed the cause for the problem. That's a bit counter-intuitive to be honest. It should be at least pointed out in the docs.

[bbarani]

@kostyasimanov @thannaske Thanks for your comment. Would you be able to raise a PR to update the documentation?