services other than -http do not get populated #2824
Comments
|
We recently added a documentation to create services for exposing different node types that could help. I followed it and the endpoints are well created: > k get svc,ep | grep nodes
service/hulk-es-coordinator-nodes ClusterIP 10.1.111.41 <none> 9200/TCP 9m58s
service/hulk-es-ingest-nodes ClusterIP 10.1.111.48 <none> 9200/TCP 9m58s
service/hulk-es-non-master-nodes ClusterIP 10.1.103.137 <none> 9200/TCP 9m58s
endpoints/hulk-es-coordinator-nodes 9m58s
endpoints/hulk-es-ingest-nodes 9m58s
endpoints/hulk-es-non-master-nodes 9m58sIf not, could you please share the YAML manifest that you used to create your Elastic Stack and the associated services? |
|
Right but this is done after the cluster is created rather than using the services that get created by the operator automatically. Essentially what you are doing just updates the existing services to add the port and targets, which are missing by default in the services created by the operator. So I guess the ask is to let the operator create a functioning service that I do not need to update at a later time (later step). As far as my manifest, sure here it is: apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: production-1
spec:
version: 7.6.1
updateStrategy:
changeBudget:
maxSurge: 1
maxUnavailable: 1
podDisruptionBudget:
spec:
disruptionsAllowed: 1
maxUnavailable: 1
selector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-1
http:
service:
spec:
clusterIP: None
tls:
selfSignedCertificate:
disabled: true
nodeSets:
- name: master
count: 3
config:
node.master: true
node.data: false
node.ingest: false
node.ml: false
node.store.allow_mmap: false
cluster.remote.connect: false
node.attr.hostname: ${DIAMANTI_NODE_NAME}
cluster.routing.allocation.awareness.attributes: hostname
indices.lifecycle.poll_interval: 1m
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 256Gi
storageClassName: high-m3
podTemplate:
metadata:
labels:
forceUpdate: 'true'
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-1
topologyKey: kubernetes.io/hostname
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms8g -Xmx8g
- name: DIAMANTI_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 16Gi
cpu: 4
limits:
memory: 16Gi
cpu: 4
- name: data-hot
count: 3
config:
node.master: false
node.data: true
node.ingest: true
node.ml: false
node.store.allow_mmap: false
cluster.remote.connect: false
node.attr.data: hot
node.attr.hostname: ${DIAMANTI_NODE_NAME}
cluster.routing.allocation.awareness.attributes: hostname
indices.lifecycle.poll_interval: 1m
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 256Gi
storageClassName: high
podTemplate:
metadata:
labels:
forceUpdate: 'true'
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-1
topologyKey: kubernetes.io/hostname
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms8g -Xmx8g
- name: DIAMANTI_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 16Gi
cpu: 8
limits:
memory: 16Gi
cpu: 8
- name: data-warm
count: 3
config:
node.master: false
node.data: true
node.ingest: true
node.ml: false
node.store.allow_mmap: false
cluster.remote.connect: false
node.attr.data: warm
node.attr.hostname: ${DIAMANTI_NODE_NAME}
cluster.routing.allocation.awareness.attributes: hostname
indices.lifecycle.poll_interval: 1m
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 256Gi
storageClassName: medium
podTemplate:
metadata:
labels:
forceUpdate: 'true'
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-1
topologyKey: kubernetes.io/hostname
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms8g -Xmx8g
- name: DIAMANTI_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 16Gi
cpu: 4
limits:
memory: 16Gi
cpu: 4
- name: data-cold
count: 3
config:
node.master: false
node.data: true
node.ingest: true
node.ml: false
node.store.allow_mmap: false
cluster.remote.connect: false
node.attr.data: cold
node.attr.hostname: ${DIAMANTI_NODE_NAME}
cluster.routing.allocation.awareness.attributes: hostname
indices.lifecycle.poll_interval: 1m
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 256Gi
storageClassName: best-effort
podTemplate:
metadata:
labels:
forceUpdate: 'true'
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-1
topologyKey: kubernetes.io/hostname
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms4g -Xmx4g
- name: DIAMANTI_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 8Gi
cpu: 1
limits:
memory: 8Gi
cpu: 1
- name: ingest
count: 3
config:
node.master: false
node.data: false
node.ingest: true
node.ml: false
node.store.allow_mmap: false
cluster.remote.connect: false
node.attr.hostname: ${DIAMANTI_NODE_NAME}
cluster.routing.allocation.awareness.attributes: hostname
indices.lifecycle.poll_interval: 1m
podTemplate:
metadata:
labels:
forceUpdate: 'true'
annotations:
#diamanti.com/endpoint0: '{"network":"v3041-nr","perfTier":"high"}'
#diamanti.com/endpoint1: '{"network":"elastic-ingest-r","perfTier":"high"}'
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-0
topologyKey: kubernetes.io/hostname
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms8g -Xmx8g
- name: DIAMANTI_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 16Gi
cpu: 8
limits:
memory: 16Gi
cpu: 8
- name: coordinator
count: 2
config:
node.master: false
node.data: false
node.ingest: true
node.ml: false
node.store.allow_mmap: false
cluster.remote.connect: false
node.attr.hostname: ${DIAMANTI_NODE_NAME}
cluster.routing.allocation.awareness.attributes: hostname
indices.lifecycle.poll_interval: 1m
podTemplate:
metadata:
labels:
forceUpdate: 'true'
annotations:
#diamanti.com/endpoint0: '{"network":"v3041-nr","perfTier":"high"}'
#diamanti.com/endpoint1: '{"network":"elastic-ingest-r","perfTier":"high"}'
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: production-0
topologyKey: kubernetes.io/hostname
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: -Xms8g -Xmx8g
- name: DIAMANTI_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
memory: 16Gi
cpu: 8
limits:
memory: 16Gi
cpu: 8 |
The docs are indicating to create new services, not updating the headless services that are created to give the sset pods a stable network identity. The docs linked have you create a ClusterIP service. You do not need to wait for the operator to create the extra service pointing to whatever combination of roles you would like, you can create the service at the same time you create your Elasticsearch resource. We would definitely be open to suggestions on how to make the docs clearer. We do say to create services (rather than update them), but maybe there's some other way we can be more explicit. |
|
Right the problem then if you are to do this and use say the self signed certificates is that requests will fail due the SNI pointing to say $name-es-ingest-nodes, $name-es-ingest-nodes.svc... etc So back to my original question, why not create proper fully functioning services when the operator creates the cluster? If its going to go trough the trouble of creating the services why not make them fully functional since it already gives you a logical breakdown of your cluster. IF from that point I need more customization and different combinations of nodes then the document you have linked makes complete sense, but if I want to rely on the built in functionality I have to do hoopty hoops to get there. |
|
So far we've been reluctant to add even more services by default as it is difficult to predict what roles people would want to target, and the workaround is fairly simple (~12 lines of yaml per role you want to target). We haven't ruled it out entirely though. I'm a bit confused by "the built in functionality" you're referencing. The current services are not intended to be load balancers and cannot be: they're solely there so we have a stable host name to address the pods. |
|
Ok i think we are sorta talking past each other :D. So if I deploy the above cluster (i included my yaml) it will produce the following services by default (aka built in functionality of the operator) NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
production-1-es-coordinator ClusterIP None <none> <none> 6h37m
production-1-es-data-cold ClusterIP None <none> <none> 6h37m
production-1-es-data-hot ClusterIP None <none> <none> 6h37m
production-1-es-data-warm ClusterIP None <none> <none> 6h37m
production-1-es-http ClusterIP None <none> 9200/TCP 6h37m
production-1-es-ingest ClusterIP None <none> <none> 6h37m
production-1-es-ingest-custom ClusterIP None <none> 9200/TCP 6h37m
production-1-es-master ClusterIP None <none> <none> 6h37mNow out of those services, if I would like to use "the built in functionality of the operator giving me an already segmented list of nodes in service form" for anything useful. Like say making an Ingress object that points to the On the other hand you have the -http service which is a catch all and does me no good as I would like to just point my traffic to the ingest nodes. I understand I can use a custom service myself, you can see that I already have a -custom in there that does exactly that. However, the operator already logically separates things for me and creates the objects I need. All I am asking is for those services to be usable for other things than internal workings of the elastic cluster AND without the need for me to create custom resources. So its not about how many lines or anything I have to produce, its more the doubling up work when work has already been done and is automatically managed for me by the operator (aka if I decide to add ML nodes the operator will create an ML service and if its a fully functional service I can then direct ML traffic as i see fit using it). This has nothing to do with using the service you provide as a LoadBalancer. I simply want to be able to create other k8s objects that reference the objects created by the operator, otherwise I am only double efforting. |
|
I went ahead and created a more targeted enhancement issue for this here: #2843 So if I misunderstood what you're looking for feel free to comment there. It seems like something we can support, though it would surprise me if it was required. Since the DNS record for the service will already resolve all the IP addresses that match the label, you should be able to specify the port of the pods you want to connect to without it being specified on the service. I was looking at the ingress docs and it wasn't clear that it required the port specified in |
Bug Report
What did you do?
Deployed a ECK cluster and the associated services
What did you expect to see?
A list of services with the appropriate endpoints for each resource service defined
What did you see instead? Under which circumstances?
Environment
Kubernetes v1.15.6 on prem
ECK version:
1.0.1
I expect that just like the -http service the rest of the services and the selectors they use match up and actually populate, right now, there are services but they are unusable as they do not actually have endpoints.
The text was updated successfully, but these errors were encountered: