[Bug] Docker Image support running without security and PA plugin

[peterzhuamazon]

Describe the bug

#1154 issue follow up after fixing docker build issues.

https://github.com/tianleh/opensearch-build/blob/main/docker/release/config/opensearch/opensearch-docker-entrypoint.sh#L100-L131

As of now we expect docker image to have security and PA plugin install by default.
If PA failed to start then OS will also terminate.

This behavior is ok for previous releases, but having issues in 1.3.0 since we dont have branches for security and PA yet, and the bundle build pipeline is still triggering snapshots builds on a daily basis for docker images.

We can resolve this by decoupling PA and OS if PA not exists.

Thanks.

To reproduce

See above.

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[dblock]

There's no indication of why security/PA are failing, either.

> docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchstaging/opensearch:1.3.0

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-1.3.0
Detected OpenSearch Security Version: 1.3.0.0

### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../securityconfig and execute:
"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling OpenSearch Security Plugin
Killing opensearch process 105
OpenSearch exited with code 143
Performance analyzer exited with code 127

[peterzhuamazon]

There's no indication of why security/PA are failing, either.

> docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchstaging/opensearch:1.3.0

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-1.3.0
Detected OpenSearch Security Version: 1.3.0.0

### Success
### Execute this script now on all your nodes and then start all nodes
### OpenSearch Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../securityconfig and execute:
"/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/plugins/opensearch-security/securityconfig" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
Enabling OpenSearch Security Plugin
Killing opensearch process 105
OpenSearch exited with code 143
Performance analyzer exited with code 127

ulimit + sysctl

And you need to cat the actual log to see.

[peterzhuamazon]

Removing this dependency from docker by adding a flag to allow a user to give flexibility to disable PA upon required, just like how its done for security plugin.

[kalpaitch]

I'm still getting it with:

opensearchproject/opensearch-dashboards   latest    a4962689e3f6   3 weeks ago     1.09GB
opensearchproject/opensearch              latest    feebbf30f81c   3 weeks ago     1.16GB

Error:

xxx     | [2023-04-11T09:36:19,045][WARN ][stderr                   ] [xxx] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
xxx     | [2023-04-11T09:36:19,046][WARN ][stderr                   ] [xxx] **SLF4J: Defaulting to no-operation (NOP) logger implementation**
xxx     | [2023-04-11T09:36:19,046][WARN ][stderr                   ] [xxx] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
xxx     | [2023-04-11T09:36:19,054][INFO ][o.o.s.s.t.SSLConfig      ] [xxx] SSL dual mode is disabled
xxx     | [2023-04-11T09:36:19,055][WARN ][o.o.s.OpenSearchSecurityPlugin] [xxx] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.