[BUG] Cluster status yellow for single node clusters

[drewmiranda-gl]

What is the bug?
A clear and concise description of the bug.

I understand this isn't a bug in the sense that the code is working as designed and an intentional choice was made. However, as far as I can find, documentation for Anomaly Detection doesn't appear to explicitly say it does not support single node clusters. IMO either the documentation should be updated to communicate this, OR there should be a configurable way to control the value of SETTING_NUMBER_OF_REPLICAS.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Start with a single-node OpenSearch cluster
2. Create an Anomaly Detection detector
3. Observe that indexes are automatically created, and include both primary and replica shards

What is the expected behavior?
A clear and concise description of what you expected to happen.

Clear guidance and requirements on if a single-node cluster is supported with Anomaly Detection, OR the ability to configure the behavior of SETTING_NUMBER_OF_REPLICAS BEFORE the indexes are created to prevent cluster health from being displayed as yellow.

I understand a workaround could be used (e.g. curl -X PUT "http://127.0.0.1:9200/.opendistro-*/_settings" -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":0}}'), however, this requires one to know such workaround exists and to implement it AFTER the indexes are created. This makes it difficult to have a single series of steps for building/configuring an opensearch cluster.

What is your host/environment?

• OS: Ubuntu Server
• Version: 22.04 LTS
• OpenSearch: 2.6 (have tested with 2.4.1, 2.5, 2.6, however it appears based on the commit history this has existed for many years)

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

index                               shard prirep state      docs   store ip         node
.opendistro-anomaly-detection-state 0     r      UNASSIGNED                         
.opendistro-anomaly-detector-jobs   0     r      UNASSIGNED                         
.opendistro-anomaly-detectors       0     r      UNASSIGNED                         

Do you have any additional context?
Add any other context about the problem.

[kaituo]

@drewmiranda-gl, AD supports single node. The problem is AD indices may lose fault tolerance if the single node goes down. We can also allow configuring the behavior of SETTING_NUMBER_OF_REPLICAS. Such configuration is for advanced users. For normal users, I am considering to have auto_expand_replicas of AD indices to be "0~1" or "0~2" so that the single cluster won't go yellow. What do you think?

[drewmiranda-gl]

Allowing configuration of SETTING_NUMBER_OF_REPLICAS would be amazing! I'm not that familiar with auto_expand_replicas, would that allow a default of having a replica shard created but only if there are sufficient nodes to support it? That would also work too.

Thanks!

[kaituo]

Allowing configuration of SETTING_NUMBER_OF_REPLICAS would be amazing! I'm not that familiar with auto_expand_replicas, would that allow a default of having a replica shard created but only if there are sufficient nodes to support it? That would also work too.

Thanks!

Yes. We will only create replica shards if there are sufficient nodes to support it. Cool. We will put this to our backlog.

[VijayanB]

@kaituo Shall we create new Feature issue and close this issue or move this issue from bug to Feature?

[kaituo]

I moved this issue from bug to Feature

[williamtrelawny]

Also wanted to note that in a brand new v2.9.0 single-node install, after configuring the Security plugin the following indices are created as well:

• .plugins-ml-config
• .opensearch-sap-pre-packaged-rules-config
• .opensearch-sap-log-types-config

[kaituo]

.plugins-ml-config comes from ml-commons plugin. .opensearch-sap indices come from security analytics plugin.

[williamtrelawny]

I understand in the OP description he was referring to AD, but the issue title refers to single node deployments in general, so I figured single-node deployments with the Security plugin enabled would be valid here as well.

Do you want me to make a new issue for this? What exactly do you mean in your response?

[kaituo]

I meant these indices are not created by AD and this repo is AD repo. Do you want to create a new issue in ml-commons and security analytics repo?