Allow to choose between ELB and Route53 round robin for the APIServer

[spacepluk]

Hi,
I think this would be a nice option to allow deployments of really tiny (and cheap) clusters.

[tarvip]

I would like to use 0.8.3 approach for small clusters where controller node HA is not important. Instead of creating autoscaling group and ELB for 1 controller node, just use ElasticIP for external access and host internal ip for worker nodes.

[redbaron]

Whats the point? ELB approach is generic enough to cover both small and HA setups

[tarvip]

I guess the biggest difference is that you have to pay a bit more.
ElasticIP is free as long it is attached to a host, ELB is not.

Another problem is when you don't want to open API server to the world (basically restrict access in security group to specific IPs). Then you have to create additional internal ELB for worker nodes and attach it to controller autoscaling group otherwise worker nodes cannot access API (this setup also assumes that you have recreated API certificate etc.). I think this is more generic problem, but with small clusters it means 2 ELBs in total.

[redbaron]

Another problem is when you don't want to open API server to the world

use internal ELB,or if you use external ELB add workers SG to ELB SG rules

[tarvip]

if you use external ELB add workers SG to ELB SG rules

I already tried that. It doesn't work.

[mumoshu]

Hi @tarvip, I've recently worked on adding a generic support to switch internal/external ELB for serving Kubernetes API in #284.
Would you confirm if it works for your use-case?

To make an external ELB public, set controller.loadBalancer.private to false(which is also the default value for the key):

controller:
  subnets:
  - name: private1
  - name: private2
  loadBalancer:
    private: false

And obviously, set controller.loadBalancer.private to true for an internal ELB.

[mumoshu]

Hi @spacepluk @tarvip! Thanks for your interest on kube-aws.
Also thanks as always @redbaron for your support.

I occasionally receive feature requests like this.
Personally, I'm not too opposed to add it if it does contribute to more wider acceptance of kube-aws in the world, as long as it doesn't break or affect our primary focus: "a community-driven tool to manage production-grade CoreOS Kubernetes clusters on AWS".
Could you share me more details of your use-case? Why and for what do you want a tiny kube-aws cluster?

[tarvip]

Would you confirm if it works for your use-case?

I could confirm it, I have to recreate few clusters anyway from scratch (i think they are too old for upgrade), but v0.9.3 is still (binaries are pending coreos signs).

[spacepluk]

Hi there,
In my case I would like to use kube-aws everywhere. I've deployed a couple of clusters for clients where the HA controllers are very welcome.

I also migrated my own servers (previously debian) to a very small kube-aws cluster. This was before the HA changes. It's great and super cheap with spot instances. But in this scenario, adding an ELB costs about the same as the whole cluster without bringing much value, whereas the Route53 alias would be free.

I think that this option adds a lot of flexibility and is very appealing for people like me that work under a tight budget but might want to scale up occasionally.

[cknowles]

@mumoshu this is similar to my concern about the 50% increase in price for dev cluster with dedicated etcd. I guess we could support this? In the case of etcd, my ideal is support shared/external etcd somehow.

[mumoshu]

Thanks everyone.
@spacepluk @tarvip If someone is willing to test it, I guess I can implement the round-robin thing(Sorry but my resource is very limited and must focus on the kube-aws's primary goal!
@tarvip controller.loadBalancer.perivate is added since v0.9.4-rc.1 released today. Would you mind trying?
@c-knowles Shared/external etcd is not my primary focus but I guess it may/may not be implemented in a similar form in process of e.g. #27. Do you remember that someone requested separated cfn stacks among etcd and controller nodes? If we go that route, adding the support for shared/external etcd on top of that won't be too hard.

[mumoshu]

Btw how many ip addresses of controller nodes can be associated to a single A record(for k8s api endpoint)?
If the limit is very low, also in that regard, we have to inform our users that the ideal dns round-robin for k8s api endpointfeature shouldn't be recommended for production use.

[tarvip]

controller.loadBalancer.perivate is added since v0.9.4-rc.1 released today. Would you mind trying?

I'll try to do that this week.

[spacepluk]

@mumoshu I would definitely test the round robin thing if you have time to implement it :)

I'm not sure about the limits though.

[tarvip]

@tarvip controller.loadBalancer.private is added since v0.9.4-rc.1 released today. Would you mind trying?

It seems that it doesn't cover my case. This option allows only creating public or private ELB. But I need both. The idea is that we want to restrict access to public ELB (by removing 0.0.0.0/0), but when you restrict access to public ELB then nodes cannot connect to API via public ELB any more.

[mumoshu]

@tarvip Ah, makes sense! So what you'd like to have is both:

• a public ELB or public A record for round-robin of public IPs of controller nodes
• a private ELB or private A record for round-robin private IPs of controller nodes

And you're going to manually tweak security groups for controller nodes in stack-template.json for further access control, right?

[tarvip]

And you're going to manually tweak security groups for controller nodes in stack-template.json for further access control, right?

Yes. Anyway, I guess if others are happy with current solution, then there is no need to support such case.

[mumoshu]

Honestly I've not yet considered a separation of controller LBs like that but it sounds like a good thing to do for security reasons.
I'd like to hear from others, too.

[mumoshu]

Just a quick thought without any consideration on feasibility but probably a generic solution for #281 (comment) would be something like the below in cluster.yaml?

controller:
  # Instead of this
  # loadBalancer:
  #   private: true

# Introduce this
apiEndpoints:
  # Non-empty value inside the `external` key + type `dnsRoundRobin` requires `controller.subnets[].private` to be false i.e. controller nodes are deployed to public subnets
  external:
    dnsName: k8s.external.example.com
    hostedZoneId: <id for the hosted zone external.example.com>
    securityGroups:
    - id: sg-toallowexternalaccess
    type: loadBalancer
  internal:
    dnsName: k8s.internal.example.com
    hostedZoneId: <id for the hosted zone internal.example.com>
    securityGroups:
    - id: sg-toallowinternalacess
    type: dnsRoundRobin

[mumoshu]

Are there any use-cases that requires 2 or more external endpoints and/or 2 or more internal endpoints?
If not, I'll go ahead with #281 (comment)

[mumoshu]

Probably, we'd need something like the below to consistently support both this use-case and @c-knowles's use-case #343:

controller:
  # Instead of this
  # loadBalancer:
  #   private: true

# Introduce this
apiEndpoints:
- name: stableExternalEndpoint
  # `dnsName` this will be added to CNs in the apiserver cert/cc @c-knowles
  dnsName: k8s.external.example.com 
  # You can omit this. If omitted, it is your responsibility to add controller nodes to an ELB serving `k8s.external.example.com`
  loadBalancer:
    id: id-of-existing-internet-facing-elb
- name: stableInternalEndpoint
  dnsName: k8s.internal.example.com
  loadBalancer:
    id: id-of-existing-internal-elb
# Former `externalDNSName` + `hostedZoneId` + newly added SG definitions for controller ELB
- name: versionedExternalEndpoint
  dnsName: v2.k8s.external.example.com
  loadBalancer:
    hostedZone:
      id: <id for the hosted zone external.example.com>
    securityGroups:
    - id: sg-toallowexternalaccess
# Former `externalDNSName` + `hostedZoneId` without an ELB + newly added SG definitions for controller nodes
- name: versionedInternalEndpoint
  dnsName: v2.k8s.internal.example.com
  dnsRoundRobin: # will enable a bash script in cloud-config-controller to update Route 53 record sets
    hostedZone:
      id: <id for the hosted zone internal.example.com>
    securityGroups:
    - id: sg-toallowinternalacess

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.