mult-node/aws: Add support for HA controllers.

[eliaslevy]

This

• replaces the single controller with an autoscale group
• replaces the static controller IP address and elastic controller IP address with an internal elastic load balancer and assigns it a domain name using Route 53
• clusters etcd in the controller nodes using the monstantoco/etcd-aws-cluster container.

Podmaster was already configured in the controller, so the scheduler and controller-manager were already configured to support HA.

The monstantoco/etcd-aws-cluster is used rather than etcd's discovery mechanism or a static configuration as it makes use of AWS tools to discover the members of the controllers autoscale group. That allows the container to not only of starting a new cluster, but of managing removal and addition of cluster members as the membership of the autoscale group changes, for a fully automated solution.

You probably don't want to merge this as is. On my own branch I've moved to native AWS networking. Therefore, I've made no attempt to configure etcd on worker nodes. Flannel appeared to be the only etcd user on workers, and with it gone on my branch, I've removed reference of etcd from the worker's configuration. I've also closed incoming access over the Internet and use a VPN access server to access the cluster's VPC, so the load balancer and hostname assign to it are internal in this pull, whereas you probably want to make them both internal and external if you are keeping external access to the cluster.

[bcwaldon]

Were you able to explore any other options than this? Is there no way to assign static IPs at deploy time and have a static initial-cluster option?

[eliaslevy]

I didn't care much for using the etcd discovery service. I could have gone with elastic IPs, but after reading http://engineering.monsanto.com/2015/06/12/etcd-clustering/ this seemed like a better approach to me.

[bcwaldon]

It also seems like using the ASG would be racy, when we need absolute determinism to bootstrap this cluster. How does this work that makes it safe to run in parallel on several machines?

[eliaslevy]

Given that we want to run in HA mode, then by definition it has to work in parallel on several machines. ;-) So far we've had no issues with it, but I am guessing there may be some scenarios not handled. Alas, I've not explored them. Might want to ping the Monsanto folks that wrote it. They probably have some experience running it on larger cluster for far longer and can give you a better idea about failure scenarios.

[bcwaldon]

Ok, that's good to hear. I'm wary of relying on a tool for management of etcd that I have zero experience with, and haven't even begun to understand. I'm also thinking that the controller cluster should be separated from the etcd cluster soon. Have you considered this separation?

[mgoodness]

@bcwaldon I'm using the Monsanto approach in exactly the scenario you described. The etcd ASG spins up first, using the etcd-aws-cluster container for node registration. Then the Kubernetes controller cluster is started, which also makes use of the container. The etcd ASG is passed in through an environment variable, and etcd-aws-cluster configures proxy mode. Here's the relevant section of my controller cloud_config:

    - name: etcd-peers.service
      command: start
      content: |
        [Unit]
        Description=Write peers to file (for bootstrapping cluster)
        Requires=docker.service
        After=docker.service

        [Service]
        ExecStartPre=/usr/bin/docker pull monsantoco/etcd-aws-cluster:latest
        ExecStart=/usr/bin/docker run --rm -e PROXY_ASG=${ETCD_ASG} \
          -e ETCD_CLIENT_SCHEME=https -e ETCD_PEER_SCHEME=https \
          -e ETCD_CURLOPTS="--cacert /etc/ssl/etcd/ca.pem \
            --cert /etc/ssl/etcd/client.pem \
            --key /etc/ssl/etcd/client-key.pem" \
          -v /etc/sysconfig/:/etc/sysconfig/ \
          -v /etc/ssl/etcd:/etc/ssl/etcd \
          monsantoco/etcd-aws-cluster:latest
        ExecStartPost=/usr/bin/chown -R etcd /etc/ssl/etcd
        Restart=on-failure
        RestartSec=10

It's worked very well so far. There's another project out there that is using a single Go executable to do the same kind of dynamic bootstrapping; it was inspired by the Monsanto work. Not nearly as flexible, though.

[danielwhatmuff]

Is support for HA controllers coming soon?

[aaronlevy]

I am going to close this PR as the underlying functionality has changed significantly since the time this was proposed. If this should be re-opened, please let me know