Document how clients not on K8s can join a DC in K8s. (#9438)

website/content/docs/k8s/installation/deployment-configurations/clients-outside-kubernetes.mdx

@@ -11,8 +11,21 @@ description: >-
 
 Consul clients running on non-Kubernetes nodes can join a Consul cluster running within Kubernetes.
 
-## Auto-join
+## Networking
+
+Within one datacenter, Consul typically requires a fully connected
+[network](/docs/architecture). This means the IPs of every client and server
+agent should be routable by every other client and server agent in the
+datacenter. Clients need to be able to [gossip](/docs/architecture/gossip) with
+every other agent and make RPC calls to servers. Servers need to be able to
+gossip with every other agent. See [Architecture](/docs/architecture) for more details.
 
+-> **Consul Enterprise customers** may use [network
+segments](/docs/enterprise/network-segments) to enable non-fully-connected
+topologies. However, out-of-cluster nodes must still be able to communicate
+with the server pod or host IP addresses.
+
+## Auto-join
 The recommended way to join a cluster running within Kubernetes is to
 use the ["k8s" cloud auto-join provider](/docs/agent/cloud-auto-join#kubernetes-k8s).
 
@@ -29,20 +42,122 @@ started using the [official Helm chart](/docs/k8s/helm):
 ```shell-session
 $ consul agent -retry-join 'provider=k8s label_selector="app=consul,component=server"'
 ```
+-> **Note:** This auto-join command only connects on the default gossip port
+8301, whether you are joining on the pod network or via host ports.  Either a
+consul server or client that is already a member of the datacenter should be
+listening on this port for the external client agent to be able to use
+auto-join.
 
-By default, Consul will join the default gossip port. Pods may set an
-annotation `consul.hashicorp.com/auto-join-port` to an integer value or
-a named port to specify the port for the auto-join to return. This enables
-different pods to have different exposed ports.
+### Auto-join on the Pod network
+In the default Consul Helm chart installation, Consul clients and servers are
+routable only via their pod IPs for server RPCs and gossip (HTTP
+API calls to Consul clients can also be made through host IPs). This means any
+external client agents joining the Consul cluster running on Kubernetes would
+need to be able to have connectivity to those pod IPs.
 
-## Networking
+In many hosted Kubernetes environments, you will need to explicitly configure
+your hosting provider to ensure that pod IPs are routable from external VMs.
+See [Azure AKS
+CNI](https://docs.microsoft.com/en-us/azure/aks/concepts-network#azure-cni-advanced-networking),
+[AWS EKS
+CNI](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html) and
+[GKE VPC-native
+clusters](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips).
+
+Given you have the [official Helm chart](/docs/k8s/helm) installed with the default values, do the following to join an external client agent.
+
+  1. Make sure the pod IPs of the clients and servers in Kubernetes are
+  routable from the VM and that the VM can access port 8301 (for gossip) and
+  port 8300 (for server RPC) on those pod IPs.
+
+  1. Make sure that the client and server pods running in Kubernetes can route
+  to the VM's advertise IP on its gossip port (default 8301).
+
+  2. Make sure you have the `kubeconfig` file for the Kubernetes cluster in `$HOME/.kube/config` on the external VM.
+
+  2.  On the external VM, run:
+  ```bash
+  consul agent \
+    -advertise="$ADVERTISE_IP" \
+    -retry-join='provider=k8s label_selector="app=consul,component=server"'
+    -bind=0.0.0.0 \
+    -hcl='leave_on_terminate = true' \
+    -hcl='ports { grpc = 8502 }' \
+    -config-dir=$CONFIG_DIR \
+    -datacenter=$DATACENTER \
+    -data-dir=$DATA_DIR \
+  ```
+
+  3. Check if the join was successful by running `consul members`. Sample output:
+  ```shell-session
+  / $ consul members
+  Node                                           Address           Status  Type    Build  Protocol  DC   Segment
+  consul-consul-server-0                         10.138.0.43:9301  alive    server  1.9.1  2         dc1  <all>
+  external-agent                                 10.138.0.38:8301  alive    client  1.9.0  2         dc1  <default>
+  gke-external-agent-default-pool-32d15192-grs4  10.138.0.43:8301  alive    client  1.9.1  2         dc1  <default>
+  gke-external-agent-default-pool-32d15192-otge  10.138.0.44:8301  alive    client  1.9.1  2         dc1  <default>
+  gke-external-agent-default-pool-32d15192-vo7k  10.138.0.42:8301  alive    client  1.9.1  2         dc1  <default>
+  ```
+
+### Auto-join via host ports
+If your external VMs can't connect to Kubernetes pod IPs, but they can connect
+to the internal host IPs of the nodes in the Kubernetes cluster, you have the
+option to expose the clients and server ports on the host IP instead.
+
+  1. Install the [official Helm chart](/docs/k8s/helm) with the following values:
+  ```yaml
+  client:
+    exposeGossipPorts: true # exposes client gossip ports as hostPorts
+  server:
+    exposeGossipAndRPCPorts: true # exposes the server gossip and RPC ports as hostPorts
+    ports:
+      # Configures the server gossip port
+      serflan:
+        # Note that this needs to be different than 8301, to avoid conflicting with the client gossip hostPort
+        port: 9301
+  ```
+   This will expose the client gossip ports, the server gossip ports and the server RPC port at `hostIP:hostPort`. Note that `hostIP` is the **internal** IP of the VM that the client/server pods are deployed on.
+
+  1. Make sure the IPs of the Kubernetes nodes are routable from the VM and
+  that the VM can access ports 8301 and 9301 (for gossip) and port 8300 (for
+  server RPC) on those node IPs.
+
+  1. Make sure the client and server pods running in Kubernetes can route to
+  the VM's advertise IP on its gossip port (default 8301).
+
+  3. Make sure you have the `kubeconfig` file for the Kubernetes cluster in `$HOME/.kube/config` on the external VM.
+
+  4. On the external VM, run (note the addition of `host_network=true` in the retry-join argument):
+  ```bash
+  consul agent \
+    -advertise="$ADVERTISE_IP" \
+    -retry-join='provider=k8s host_network=true label_selector="app=consul,component=server"'
+    -bind=0.0.0.0 \
+    -hcl='leave_on_terminate = true' \
+    -hcl='ports { grpc = 8502 }' \
+    -config-dir=$CONFIG_DIR \
+    -datacenter=$DATACENTER \
+    -data-dir=$DATA_DIR \
+  ```
+  3. Check if the join was successful by running `consul members`. Sample output:
+  ```shell-session
+  / $ consul members
+  Node                                           Address           Status  Type    Build  Protocol  DC   Segment
+  consul-consul-server-0                         10.138.0.43:9301  alive    server  1.9.1  2         dc1  <all>
+  external-agent                                 10.138.0.38:8301  alive    client  1.9.0  2         dc1  <default>
+  gke-external-agent-default-pool-32d15192-grs4  10.138.0.43:8301  alive    client  1.9.1  2         dc1  <default>
+  gke-external-agent-default-pool-32d15192-otge  10.138.0.44:8301  alive    client  1.9.1  2         dc1  <default>
+  gke-external-agent-default-pool-32d15192-vo7k  10.138.0.42:8301  alive    client  1.9.1  2         dc1  <default>
+  ```
+
+## Manual join
+If you are unable to use auto-join, you can also follow the instructions in
+either of the auto-join sections but instead of using a `provider` key in the
+`-retry-join` flag, you would need to pass the address of at least one
+consul server, e.g: `-retry-join=$CONSUL_SERVER_IP:$SERVER_SERFLAN_PORT`.
+
+However, rather than hardcoding the IP, it's recommended to set up a DNS entry
+that would resolve to the consul servers' pod IPs (if using the pod network) or
+host IPs that the server pods are running on (if using host ports).
 
-Consul typically requires a fully connected network.
-Because the Consul Helm chart currently doesn't allow exposing servers' gossip ports via a `hostPort`,
-nodes outside of Kubernetes joining a cluster running within Kubernetes must be able to communicate
-to pod IPs via the network. Note that the auto-join provider discussed above will use pod IPs by default.
 
--> **Consul Enterprise customers** may use
-[network segments](/docs/enterprise/network-segments) to
-enable non-fully-connected topologies. However, out-of-cluster nodes must still
-be able to communicate with the server pod or host IP addresses.