Error log when creating a PIT search object

[f-galland]

Description

OpenSearch 2.18.0 introduced a bug on PIT creation that triggers an error log along with a stack trace.
Functionality of the search doesn't seem to be affected, though.

This was already reported in their repository:

• [BUG] No enum constant org.opensearch.action.search.SearchPhaseName.CREATE_PIT opensearch-project/OpenSearch#16750

[AlexRuiz7]

We cannot fix this one, as the warning message is raised by OpenSearch's code. We tried to catch this exception to silence it, but it didn't work. We can only wait for a fix from OpenSearch on a future release.

[AlexRuiz7]

For development, this error can be overcome by changing the OpenSearch's snapshot version to 2.17.1.

opensearch_version = System.getProperty("opensearch.version", "2.17.1-SNAPSHOT")

This change cannot be committed to the repository as currently the wazuh-indexer packages are to be built using OpenSearch 2.18.0.

OpenSearch has a pull request ready to be merged in their main branch. A backport to v2.19.0 will follow. v2.19.0 is planned or February, meaning that we'll need to keep this issue for a few months.

The logs show periodically, showing full traceback of the unhandled exception, completely clobbering the wazuh-indexer's log file, making it really hard for the development and testing of the Data Persistence MVP features.

Given the situation, we should consider temporarily moving back to OpenSearch v2.17.1, which still resides in https://github.com/wazuh/wazuh-indexer/tree/master-2.17.1.

A more moderate temporal solution would be to add custom configurations to log4j to filter out this log, by the means of these references:

• https://logging.apache.org/log4j/2.x/manual/filters.html#example-configuration-file
• https://logging.apache.org/log4j/2.x/manual/appenders/rolling-file.html
• https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/
• https://community.cloudera.com/t5/Community-Articles/How-to-Suppress-some-specific-unwanted-log-entries-from/ta-p/246715

Global filter

Add these lines at the end of /etc/wazuh-indexer/log4j2.properties to filter out every warning log.

filter.0.type = LevelMatchFilter
filter.0.level = WARN
filter.0.onMatch = DENY
filter.0.onMismatch = NEUTRAL

Restart.

This could be improved using more granular filters, for example StringMatchFilter.

Cluster level filter

Use this command to raise the log level of the org.opensearch.action.search module to ERROR, thus not logging warnings.

PUT /_cluster/settings
{
  "persistent" : {
    "logger.org.opensearch.action.search" : "ERROR"
  }
}

Restart.

Both options seem to be a valid workaround, as tested using an MVP vagrant environment.

[root@rhel-server vagrant]# tail -f /var/log/wazuh-indexer/wazuh-cluster.log
[2024-12-13T17:26:31,894][INFO ][o.o.j.s.JobScheduler     ] [node-1] Scheduling job id s0T3wJMBfKSCwQzVMo2s for index .scheduled-commands .
[2024-12-13T17:26:34,616][WARN ][stderr                   ] [node-1] SLF4J(W): No SLF4J providers were found.
[2024-12-13T17:26:34,617][WARN ][stderr                   ] [node-1] SLF4J(W): Defaulting to no-operation (NOP) logger implementation
[2024-12-13T17:26:34,617][WARN ][stderr                   ] [node-1] SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
[2024-12-13T17:26:34,617][WARN ][stderr                   ] [node-1] SLF4J(W): Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
[2024-12-13T17:26:34,617][WARN ][stderr                   ] [node-1] SLF4J(W): Ignoring binding found at [jar:file:/usr/share/wazuh-indexer/plugins/wazuh-indexer-command-manager/log4j-slf4j-impl-2.23.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[2024-12-13T17:26:34,617][WARN ][stderr                   ] [node-1] SLF4J(W): See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
[2024-12-13T17:26:34,641][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/orders]
[2024-12-13T17:26:34,674][ERROR][c.w.c.u.h.HttpRestClient ] [node-1] Execution failed javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-12-13T17:26:34,675][ERROR][c.w.c.u.h.HttpResponseCallback] [node-1] POST https://127.0.0.1:55000/orders->javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-12-13T17:26:40,403][INFO ][o.o.m.a.MLModelAutoReDeployer] [node-1] Index not found, not performing auto reloading!
[2024-12-13T17:26:40,405][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [node-1] Starting ML sync up job...
[2024-12-13T17:26:46,137][INFO ][c.w.c.u.h.HttpRestClient ] [node-1] Sending payload with id [null] to [https://127.0.0.1:55000/orders]
[2024-12-13T17:26:46,147][ERROR][c.w.c.u.h.HttpResponseCallback] [node-1] POST https://127.0.0.1:55000/orders->javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2024-12-13T17:26:46,149][ERROR][c.w.c.u.h.HttpRestClient ] [node-1] Execution failed javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Similar workarounds could be applied for development environments.