Add port to headless service

[anyasabo]

We should consider adding the Elasticsearch HTTP port to the headless service we create for each statefulset. This will also create SRV records for the port, and some other resources (e.g. ingress) can use that service to target the backend pods, but do not need a load balancing ip address to point to.

It's not clear to me if we need to expose a port on the service for ingress to work. I would expect not, since all the service is doing is populating a list of endpoints. The API docs and this KEP didn't make it clear either. That said, I don't really see any downside to doing this and it definitely makes it easier to use.

This also may make it somewhat easier for users who do not need a VIP to target a specific role type (e.g. ingress nodes only) without needing to create their own service, since it can use the SRV record to find the port number as well. I don't know any clients that do this though.

Reference: #2824

[charith-elastic]

I am not sure whether this is something particularly useful for users:

• Headless services do not provide load balancing. This might not be obvious to users and they could end up with a badly configured client that sends all its requests to a single node. This could potentially overwhelm that node and result in a bad user experience.
• Ingresses seem to require a service of type NodePort or LoadBalancer (at least for GCLB): error while evaluating the ingress spec: service "default/hulk-es-default" is type "ClusterIP", expected "NodePort" or "LoadBalancer"

[sebgl]

I'm on the fence with this. It may encourage users to rely on those headless services which are not really made for routing traffic.

[anyasabo]

Just doing some googling, it seems like this ingress uses headless services (though not sure if you'd use it here): https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-for-anthos

Not sure if others can use it. It's not clear to me how the elasticsearch clients handle multiple IP addresses for one host name. Would need to look into it more.

[charith-elastic]

Just doing some googling, it seems like this ingress uses headless services (though not sure if you'd use it here): https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-for-anthos

That's a link to Anthos Ingress (they seem to be using a CR there). I think you meant to link to this instead? https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress. The example uses a NodePort service. I did actually try creating an Ingress with the headless service and got the error message mentioned above so at least GCLB does not appear to support them.

It is quite possible that some ingress controllers would simply use the service as a means to obtain the list of endpoints and wouldn't care about the service type itself. Therefore, my point about ingresses is not a strong one. It's just not a general solution we can recommend to all users though.

[kurktchiev]

Yes you are correct @charith-elastic Ingress controllers (regradless of implementation) use the service to get a list of endpoints. As @anyasabo and I discussed in my original issue around this, the problem here is that if you do kubectl get endpoints the services created by the operator do not all link to their proper endpoints. So if I go to use the service for Ingest nodes only and point it to my ingress controller, i get nada due to the fact that the ingress controller cannot find anything to actually send my traffic to, because the service itself does not return the endpoins it should try to hit. By adding the proper bits to the services created by the operator (simply adding the ports to the definition) it fixes all of this.