diff --git a/website/content/docs/k8s/installation/deployment-configurations/clients-outside-kubernetes.mdx b/website/content/docs/k8s/installation/deployment-configurations/clients-outside-kubernetes.mdx index d8f178258340..3c17199c3efb 100644 --- a/website/content/docs/k8s/installation/deployment-configurations/clients-outside-kubernetes.mdx +++ b/website/content/docs/k8s/installation/deployment-configurations/clients-outside-kubernetes.mdx @@ -11,8 +11,21 @@ description: >- Consul clients running on non-Kubernetes nodes can join a Consul cluster running within Kubernetes. -## Auto-join +## Networking + +Within one datacenter, Consul typically requires a fully connected +[network](/docs/architecture). This means the IPs of every client and server +agent should be routable by every other client and server agent in the +datacenter. Clients need to be able to [gossip](/docs/architecture/gossip) with +every other agent and make RPC calls to servers. Servers need to be able to +gossip with every other agent. See [Architecture](/docs/architecture) for more details. +-> **Consul Enterprise customers** may use [network +segments](/docs/enterprise/network-segments) to enable non-fully-connected +topologies. However, out-of-cluster nodes must still be able to communicate +with the server pod or host IP addresses. + +## Auto-join The recommended way to join a cluster running within Kubernetes is to use the ["k8s" cloud auto-join provider](/docs/agent/cloud-auto-join#kubernetes-k8s). @@ -29,20 +42,122 @@ started using the [official Helm chart](/docs/k8s/helm): ```shell-session $ consul agent -retry-join 'provider=k8s label_selector="app=consul,component=server"' ``` +-> **Note:** This auto-join command only connects on the default gossip port +8301, whether you are joining on the pod network or via host ports. Either a +consul server or client that is already a member of the datacenter should be +listening on this port for the external client agent to be able to use +auto-join. -By default, Consul will join the default gossip port. Pods may set an -annotation `consul.hashicorp.com/auto-join-port` to an integer value or -a named port to specify the port for the auto-join to return. This enables -different pods to have different exposed ports. +### Auto-join on the Pod network +In the default Consul Helm chart installation, Consul clients and servers are +routable only via their pod IPs for server RPCs and gossip (HTTP +API calls to Consul clients can also be made through host IPs). This means any +external client agents joining the Consul cluster running on Kubernetes would +need to be able to have connectivity to those pod IPs. -## Networking +In many hosted Kubernetes environments, you will need to explicitly configure +your hosting provider to ensure that pod IPs are routable from external VMs. +See [Azure AKS +CNI](https://docs.microsoft.com/en-us/azure/aks/concepts-network#azure-cni-advanced-networking), +[AWS EKS +CNI](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html) and +[GKE VPC-native +clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips). + +Given you have the [official Helm chart](/docs/k8s/helm) installed with the default values, do the following to join an external client agent. + + 1. Make sure the pod IPs of the clients and servers in Kubernetes are + routable from the VM and that the VM can access port 8301 (for gossip) and + port 8300 (for server RPC) on those pod IPs. + + 1. Make sure that the client and server pods running in Kubernetes can route + to the VM's advertise IP on its gossip port (default 8301). + + 2. Make sure you have the `kubeconfig` file for the Kubernetes cluster in `$HOME/.kube/config` on the external VM. + + 2. On the external VM, run: + ```bash + consul agent \ + -advertise="$ADVERTISE_IP" \ + -retry-join='provider=k8s label_selector="app=consul,component=server"' + -bind=0.0.0.0 \ + -hcl='leave_on_terminate = true' \ + -hcl='ports { grpc = 8502 }' \ + -config-dir=$CONFIG_DIR \ + -datacenter=$DATACENTER \ + -data-dir=$DATA_DIR \ + ``` + + 3. Check if the join was successful by running `consul members`. Sample output: + ```shell-session + / $ consul members + Node Address Status Type Build Protocol DC Segment + consul-consul-server-0 10.138.0.43:9301 alive server 1.9.1 2 dc1 + external-agent 10.138.0.38:8301 alive client 1.9.0 2 dc1 + gke-external-agent-default-pool-32d15192-grs4 10.138.0.43:8301 alive client 1.9.1 2 dc1 + gke-external-agent-default-pool-32d15192-otge 10.138.0.44:8301 alive client 1.9.1 2 dc1 + gke-external-agent-default-pool-32d15192-vo7k 10.138.0.42:8301 alive client 1.9.1 2 dc1 + ``` + +### Auto-join via host ports +If your external VMs can't connect to Kubernetes pod IPs, but they can connect +to the internal host IPs of the nodes in the Kubernetes cluster, you have the +option to expose the clients and server ports on the host IP instead. + + 1. Install the [official Helm chart](/docs/k8s/helm) with the following values: + ```yaml + client: + exposeGossipPorts: true # exposes client gossip ports as hostPorts + server: + exposeGossipAndRPCPorts: true # exposes the server gossip and RPC ports as hostPorts + ports: + # Configures the server gossip port + serflan: + # Note that this needs to be different than 8301, to avoid conflicting with the client gossip hostPort + port: 9301 + ``` + This will expose the client gossip ports, the server gossip ports and the server RPC port at `hostIP:hostPort`. Note that `hostIP` is the **internal** IP of the VM that the client/server pods are deployed on. + + 1. Make sure the IPs of the Kubernetes nodes are routable from the VM and + that the VM can access ports 8301 and 9301 (for gossip) and port 8300 (for + server RPC) on those node IPs. + + 1. Make sure the client and server pods running in Kubernetes can route to + the VM's advertise IP on its gossip port (default 8301). + + 3. Make sure you have the `kubeconfig` file for the Kubernetes cluster in `$HOME/.kube/config` on the external VM. + + 4. On the external VM, run (note the addition of `host_network=true` in the retry-join argument): + ```bash + consul agent \ + -advertise="$ADVERTISE_IP" \ + -retry-join='provider=k8s host_network=true label_selector="app=consul,component=server"' + -bind=0.0.0.0 \ + -hcl='leave_on_terminate = true' \ + -hcl='ports { grpc = 8502 }' \ + -config-dir=$CONFIG_DIR \ + -datacenter=$DATACENTER \ + -data-dir=$DATA_DIR \ + ``` + 3. Check if the join was successful by running `consul members`. Sample output: + ```shell-session + / $ consul members + Node Address Status Type Build Protocol DC Segment + consul-consul-server-0 10.138.0.43:9301 alive server 1.9.1 2 dc1 + external-agent 10.138.0.38:8301 alive client 1.9.0 2 dc1 + gke-external-agent-default-pool-32d15192-grs4 10.138.0.43:8301 alive client 1.9.1 2 dc1 + gke-external-agent-default-pool-32d15192-otge 10.138.0.44:8301 alive client 1.9.1 2 dc1 + gke-external-agent-default-pool-32d15192-vo7k 10.138.0.42:8301 alive client 1.9.1 2 dc1 + ``` + +## Manual join +If you are unable to use auto-join, you can also follow the instructions in +either of the auto-join sections but instead of using a `provider` key in the +`-retry-join` flag, you would need to pass the address of at least one +consul server, e.g: `-retry-join=$CONSUL_SERVER_IP:$SERVER_SERFLAN_PORT`. + +However, rather than hardcoding the IP, it's recommended to set up a DNS entry +that would resolve to the consul servers' pod IPs (if using the pod network) or +host IPs that the server pods are running on (if using host ports). -Consul typically requires a fully connected network. -Because the Consul Helm chart currently doesn't allow exposing servers' gossip ports via a `hostPort`, -nodes outside of Kubernetes joining a cluster running within Kubernetes must be able to communicate -to pod IPs via the network. Note that the auto-join provider discussed above will use pod IPs by default. --> **Consul Enterprise customers** may use -[network segments](/docs/enterprise/network-segments) to -enable non-fully-connected topologies. However, out-of-cluster nodes must still -be able to communicate with the server pod or host IP addresses.