Malcolm's default standalone configuration is to use a local OpenSearch instance in a Docker container to index and search network traffic metadata. OpenSearch can also run as a cluster with instances distributed across multiple nodes with dedicated roles such as cluster manager, data node, ingest node, etc.
As the permutations of OpenSearch cluster configurations are numerous, it is beyond Malcolm's scope to set up multi-node clusters. However, Malcolm can be configured to use a remote OpenSearch cluster rather than its own internal instance.
The OPENSEARCH_… environment variables in opensearch.env control whether Malcolm uses its own local OpenSearch instance or a remote OpenSearch instance as its primary data store. The configuration portion of Malcolm install script (./scripts/configure) can help users configure these options.
For example, to use the default standalone configuration, answer Y when prompted Should Malcolm use and maintain its own OpenSearch instance?
Or, to use a remote OpenSearch cluster:
…
Should Malcolm use and maintain its own OpenSearch instance? (Y/n): n
Enter primary remote OpenSearch connection URL (e.g., https://192.168.1.123:9200): https://192.168.1.123:9200
Require SSL certificate validation for communication with primary OpenSearch instance? (y/N): n
You must run auth_setup after install.py to store OpenSearch connection credentials.
…
Whether the primary OpenSearch instance is a locally maintained single-node instance or is a remote cluster, Malcolm can additionally be configured to forward logs to a secondary remote OpenSearch instance. The OPENSEARCH_SECONDARY_… environment variables in opensearch.env control this behavior. Configuration of a remote secondary OpenSearch instance is similar to that of a remote primary OpenSearch instance:
…
Forward Logstash logs to a secondary remote OpenSearch instance? (y/N): y
Enter secondary remote OpenSearch connection URL (e.g., https://192.168.1.123:9200): https://192.168.1.124:9200
Require SSL certificate validation for communication with secondary OpenSearch instance? (y/N): n
You must run auth_setup after install.py to store OpenSearch connection credentials.
…
In addition to setting the environment variables in opensearch.env as described above, users must provide Malcolm with credentials for it to communicate with remote OpenSearch instances. These credentials are stored in the Malcolm installation directory as .opensearch.primary.curlrc and .opensearch.secondary.curlrc for the primary and secondary OpenSearch connections, respectively, and are bind-mounted into the Docker containers that need to communicate with OpenSearch. These cURL-formatted config files can be generated for you by the auth_setup script as illustrated:
$ ./scripts/auth_setup
…
Store username/password for primary remote OpenSearch instance? (y/N): y
OpenSearch username: servicedb
servicedb password:
servicedb password (again):
Additional local accounts can be created at https://localhost/auth/ when Malcolm is running
Require SSL certificate validation for OpenSearch communication? (Y/n): n
Will Malcolm be using an existing remote primary or secondary OpenSearch instance? (y/N): y
Store username/password for secondary remote OpenSearch instance? (y/N): y
OpenSearch username: remotedb
remotedb password:
remotedb password (again):
Require SSL certificate validation for OpenSearch communication? (Y/n): n
…
These files are created with permissions such that only the user account running Malcolm can access them:
$ ls -la .opensearch.*.curlrc
-rw------- 1 user user 36 Aug 22 14:17 .opensearch.primary.curlrc
-rw------- 1 user user 35 Aug 22 14:18 .opensearch.secondary.curlrc
One caveat with Malcolm using a remote OpenSearch cluster as its primary document store is that the accounts used to access Malcolm's web interfaces, particularly OpenSearch Dashboards, are passed directly through to OpenSearch itself. For this reason, both Malcolm and the remote primary OpenSearch instance must have the same account information. The easiest way to accomplish this is to use an Active Directory/LDAP server that both Malcolm and OpenSearch use as a common authentication backend.
See the OpenSearch documentation on access control for more information.