node.role for node coordinator settings doesn't match with yml file

[skorzan]

Elasticsearch Version

8.1.1

Installed Plugins

No response

Java Version

openjdk version "11.0.14.1" 2022-02-08 LTS OpenJDK Runtime Environment 18.9 (build 11.0.14.1+1-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.14.1+1-LTS, mixed mode, sharing)

OS Version

Centos 7 3.10.0-1160.59.1.el7.x86_64

Problem Description

Under creation elasticsearch node as coordinator function over docker compose we're facing problem with node roles.
Also You can find the description this case there: elastic/go-ucfg#188

But still we don't know how I can deploy in straight way coordination node. On elasticsearch community we've agreed that we need to open this case for fix it.

Any suggest how we can resolve that issue over syntax with node.roles: [] in docker compose ?

Described syntax for node.roles : [ ] doesn't match.

Steps to Reproduce

create coordinator node over docker compose

version: "3.7"

services:
  es_coordination_1:
    container_name: es_coordination_1
    image: priv_repo/elk-docker/elasticsearch/elasticsearch:8.1.0
    user: elasticsearch
    environment:
      - node.name=es_coordination_1
      - xpack.ml.enabled=false
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - xpack.security.http.ssl.key=certs/es_coordination_1/es_coordination_1.key
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.certificate=certs/es_coordination_1/es_coordination_1.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.certificate=certs/es_coordination_1/es_coordination_1.crt
      - xpack.security.transport.ssl.key=certs/es_coordination_1/es_coordination_1.key
      - xpack.license.self_generated.type=basic
      - ELASTIC_USERNAME=elastic
      - ELASTIC_PASSWORD=changeme
      - cluster.name=elk_cluster
      - discovery.seed_hosts=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_1_3,es_master_2_3,es_data_ssd_3_1_ingest,es_data_ssd_1_1,es_data_ssd_2_1,es_data_ssd_3_1,es_data_ssd_4_1,es_data_ssd_5_1,es_data_hdd_1_1,es_data_hdd_2_1,es_data_hdd_3_1,es_data_hdd_4_1,es_data_hdd_5_1,es_data_hdd_6_1,es_data_hdd_7_1,es_data_hdd_8_1,es_data_hdd_9_1,es_coordination_2,es_master_1_2,es_master_2_2,es_data_ssd_3_2_ingest,es_data_ssd_1_2,es_data_ssd_2_2,es_data_ssd_3_2,es_data_ssd_4_2,es_data_ssd_5_2,es_data_hdd_1_2,es_data_hdd_2_2,es_data_hdd_3_2,es_data_hdd_4_2,es_data_hdd_5_2,es_data_hdd_6_2,es_data_hdd_7_2,es_data_hdd_8_2,es_data_hdd_9_2,es_coordination_3,es_master_1_3,es_master_2_3,es_data_ssd_3_3_ingest,es_data_ssd_1_3,es_data_ssd_2_3,es_data_ssd_3_3,es_data_ssd_4_3,es_data_ssd_5_3,es_data_hdd_1_3,es_data_hdd_2_3,es_data_hdd_3_3,es_data_hdd_4_3,es_data_hdd_5_3,es_data_hdd_6_3,es_data_hdd_7_3,es_data_hdd_8_3,es_data_hdd_9_3
      - cluster.initial_master_nodes=es_master_1_1,es_master_2_1,es_master_1_2,es_master_2_2,es_master_1_3,es_master_2_3
      - ingest.geoip.downloader.enabled=false
      - bootstrap.memory_lock=true
      - node.roles= 

Logs (if relevant)

name node.role
es_coordination_1 cdfhimrstw
es_coordination_2 cdfhimrstw
es_coordination_3 cdfhimrstw

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

This looks like a duplicate of #65577, which was recently fixed by #85186 (for 8.2.0)

[skorzan]

Ok thanks, but one thing is interesting where these enviroments was storing inside container? (When we’re setting through docker compose file)

[haji08-ksd]

I agree this issue.

I setted es8.2.3 with docker-compose, but still get cdfhimrstw

I tried

node.roles: ""
or
node.roles:

but still can get issue

[piyushkp]

Any updates on this ? this is not working on 8.4.0 while it does work on 8.2.0

[grcevski]

Hi @piyushkp,

Yes the change is in 8.4.0 as well. Can you please explain in more detail what doesn't work in 8.4.0 while it works in 8.2.0? Are you able to supply reproduction steps for us?

Thanks

[piyushkp]

grcevski i'm trying to set node.roles via env variable through HELM chart.

    roles: []
    # For client nodes, we also need to add an empty node.roles in elasticsearch.yml
    # This is due to https://github.com/elastic/helm-charts/pull/1186#discussion_r631225687
    esConfig:
      elasticsearch.yml: |
        node.roles: []

it throws below error:

Exception in thread "main" org.elasticsearch.ElasticsearchParseException: null-valued setting found for key [node.roles] found at line number [9], column number [12] at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:768) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:743) at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:687) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1190) at org.elasticsearch.node.InternalSettingsPreparer.loadOverrides(InternalSettingsPreparer.java:143) at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:53) at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110) at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.Command.main(Command.java:50) at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

[grcevski]

Thanks @piyushkp, would it be possible for you to show us what the nodes.roles section looks like in the final elasticsearch.yml, after the helm chart finished applying? I want to double check what it generated, so we can tell if this is a bug in the helm chart or elasticsearch.

[piyushkp]

root@coordinator-0:/usr/share/elasticsearch# cat config/elasticsearch.yml node.roles: []

root@coordinator-0:/usr/share/elasticsearch# env | grep roles node.roles=

[piyushkp]

grcevski this is related issue elastic/cloud-on-k8s#3718

[piyushkp]

grcevski i was trying to upgrade from 8.2.0 to 8.4.0 and master and data nodes were upgrade successfully but coordinator failed. we want to move on to 8.4.0 but b'z of this issue we can't, can you provide a fix or way around so we can have cluster up and running ?

[skorzan]

also I've met the same issue under deploying ELK cluster on docker swarm environment through docker compose file. czw., 1 wrz 2022 o 18:26 Piyush ***@***.***> napisał(a):

…

grcevski <https://github.com/grcevski> i was trying to upgrade from 8.2.0 to 8.4.0 and master and data nodes were upgrade successfully but coordinator failed. we want to move on to 8.4.0 but b'z of this issue we can't, can you provide a fix or way around so we can have cluster up and running ? — Reply to this email directly, view it on GitHub <#85731 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKTQJVS65IW73VNZDRJUU3V4DKJPANCNFSM5SWZ5KKQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- https://pl.linkedin.com/pub/stanisław-korzan/5a/539/847

[grcevski]

Hi @piyushkp, I can't see a difference in behaviour here between 8.2 and 8.4. For example, in 8.2.4 I modified the ES configuration to set node.roles to empty, I get the exception above:

in config/elasticsearch.yml:

node.roles: ${ENV_VAR}

ENV_VAR="" bin/elasticsearch      (8.2)
warning: ignoring JAVA_HOME=/usr/local/opt/openjdk@17; using bundled JDK
warning: ignoring JAVA_HOME=/usr/local/opt/openjdk@17; using bundled JDK
Exception in thread "main" org.elasticsearch.ElasticsearchParseException: null-valued setting found for key [node.roles] found at line number [89], column number [12]
	at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:767)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:743)

ENV_VAR="[]" bin/elasticsearch starts just fine, and so does 8.4.

It looks like you might be overwriting the node.roles with null or empty in your environment variables.

[piyushkp]

grcevski its a bug in elastic helm chart. https://github.com/elastic/helm-charts

[piyushkp]

@williamrandolph any updates on this bug ?