E2E test failure: TestUpdateHTTPCertSAN/ES_nodes_topology_should_eventually_be_the_expected_one

[sebgl]

https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-master/389/testReport/github/com_elastic_cloud-on-k8s_test_e2e_es/TestUpdateHTTPCertSAN_ES_nodes_topology_should_eventually_be_the_expected_one/

=== RUN   TestUpdateHTTPCertSAN/ES_nodes_topology_should_eventually_be_the_expected_one
Retries (5m0s timeout): ...................................................................................................
    TestUpdateHTTPCertSAN/ES_nodes_topology_should_eventually_be_the_expected_one: utils.go:84: 
        	Error Trace:	utils.go:84
        	Error:      	Received unexpected error:
        	            	expected elements missing from cluster [{Name:master Config:0xc000411268 Count:1 PodTemplate:{ObjectMeta:{Name: GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[test-name:test-http-cert-san] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[]} Spec:{Volumes:[] InitContainers:[] Containers:[{Name:elasticsearch Image: Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[cpu:{i:{value:2 scale:0} d:{Dec:<nil>} s:2 Format:DecimalSI} memory:{i:{value:2147483648 scale:0} d:{Dec:<nil>} s:2Gi Format:BinarySI}] Requests:map[]} VolumeMounts:[] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath: TerminationMessagePolicy: ImagePullPolicy: SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds:<nil> ActiveDeadlineSeconds:<nil> DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken:<nil> NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:*1000,RunAsNonRoot:*true,SupplementalGroups:[],FSGroup:*1000,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority:<nil> DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:<nil> PreemptionPolicy:<nil> Overhead:map[] TopologySpreadConstraints:[]}} VolumeClaimTemplates:[{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:elasticsearch-data GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[]} Spec:{AccessModes:[ReadWriteOnce] Selector:nil Resources:{Limits:map[] Requests:map[storage:{i:{value:1073741824 scale:0} d:{Dec:<nil>} s:1Gi Format:BinarySI}]} VolumeName: StorageClassName:0xc000d01d90 VolumeMode:<nil> DataSource:nil} Status:{Phase: AccessModes:[] Capacity:map[] Conditions:[]}}]}]
        	Test:       	TestUpdateHTTPCertSAN/ES_nodes_topology_should_eventually_be_the_expected_one
{"log.level":"error","@timestamp":"2020-07-01T14:02:40.685Z","message":"stopping early","service.version":"0.0.0-00000000","service.type":"eck","ecs.version":"1.4.0","error":"test failure","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngh.neting.cc/elastic/cloud-on-k8s/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:43\ngh.neting.cc/elastic/cloud-on-k8s/test/e2e/es.TestUpdateHTTPCertSAN\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/es/certs_test.go:90\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:991"}
    --- FAIL: TestUpdateHTTPCertSAN/ES_nodes_topology_should_eventually_be_the_expected_one (300.00s)

[sebgl]

I think it could be related to the transform node role change?

[pebrc]

It looks the there is a interdependency between the data role and the transform role that I missed and that is also not documented.

When we turn off data for this test that means that GET _node/_all/settings | jq '.nodes | values[] | .roles' does no longer report transform. This seems off as the documentation mentions that dedicated transform nodes are possible (could be a bug in Elasticsearch it is on purpose but documentation is incorrect)

So basically a user specifying:

node.data: false

results in:

node.master: true
node.data: false
node.ingest: true
node.ml: true
node.transform: false

Unless:

node.data: false
node.transform: true

which results in:

node.master: true
node.data: false
node.ingest: true
node.ml: true
node.transform: true

[pebrc]

Closed with #3411