How to setup k3s Cluster with disabled external Interfaces #252
Replies: 8 comments 20 replies
-
|
To clarify, the tool installs k3s, not rke2 :) |
Beta Was this translation helpful? Give feedback.
-
|
Successfully tested the derlinuxer script with a NAT Gateway as per https://community.hetzner.com/tutorials/how-to-set-up-nat-for-cloud-networks. I used a bastion host in the private network from where I could run the script and access the private hosts. |
Beta Was this translation helpful? Give feedback.
-
|
Nice that it's working. :-) And thank for the link to setup a clean and easy NAT GW. |
Beta Was this translation helpful? Give feedback.
-
|
Have verified this it works for me by setting up a new cluster along with the above provided nat gateway setup. Just one issue is when its a three node HA master cluster, the k8s api load balancer ip is still exposed to the public. Need to verify if the merged pr of @mgalesloot will help make the lb private and isolate all incoming connections to only the exposed application level load balancers. |
Beta Was this translation helpful? Give feedback.
-
|
Do note that the interface need not be |
Beta Was this translation helpful? Give feedback.
-
|
Try to access the k8s api via the private IP of the LB. This should work. Next you can disable public access of the hetzner LB. |
Beta Was this translation helpful? Give feedback.
-
|
Disabling the public network and manually editing kubeconfig file stops the worker nodes to be able to connect to the master and this also fails |
Beta Was this translation helpful? Give feedback.
-
|
hostname is working perfectly, an A record to the private ip of the load balancer is working perfectly as expected, thanks for your pr. I suppose setting the api_server_hostname directly to the private IP of the LB also might make it behave in exactly same manner. But I am sticking to DNS resolution for now. |
Beta Was this translation helpful? Give feedback.
-
|
Since the pfsense host is a potential single point of failure I guess this setup is not ideal for clusters (applications) that need to contact a lot of external APIs, am I right? Would it be an alternative to block all incoming connections on the nodes using the Hetzner firewall that's available on the cloud servers and let the nodes talk to external hosts directly? |
Beta Was this translation helpful? Give feedback.
-
|
@pgruener I have 1000 IPs available in my account. You can just request a raise to your account's limit for public IPs. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, wow, ok, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
A good reason for deploying a NAT gateway is when external APIs have a firewall. It is easier to request access for a single IP adress than for a range. |
Beta Was this translation helpful? Give feedback.
-
|
My thought on this (with Hetzner) is that's better to firewall than use a NAT gateway mainly because you can't assign private IPs explicitly to instances. You can get away with some hacky subnet stuff, but I digress If you were to run PF/OPNsense in HA/CARP mode, you still would have massive downtime if the NAT failed because you can't immediately failover to a different default route. You could theoretically run unicast rpf on every node for a whole subnet dedicated to NAT, but there isn't a good TF provider for opnsense/pfsense and unicast comes with it's own issues. |
Beta Was this translation helpful? Give feedback.
-
|
from a security perspective they're functionally the same, and if you aren't running port and security monitoring on every node anyway you should be. The difference in my head is that terraform and monitoring will catch misconfigured firewalls. Monitoring will already catch bad pods with heavy networking requirements. |
Beta Was this translation helpful? Give feedback.
-
|
shameless plug: See also #379 - has some additional infos on the topic |
Beta Was this translation helpful? Give feedback.
-
|
Can anyone help with testing v2 rc1? see #385 |
Beta Was this translation helpful? Give feedback.
-
|
Hey vitobotta, |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I'm currently trying to boot a cluster in the private network with version 2.0.8. But for some reason the CloudInit step takes up to 7 minutes, until then the ‘create’ command cancels. If I use the same config with a public ipv4 address, the CloudInit step runs through within a few seconds. Am I missing something? Is my configuration wrong? Is it possible to set the timeout via a variable? My config: cluster_name: test-cluster
kubeconfig_path: "~/.kube/config"
k3s_version: v1.26.4+k3s1
networking:
ssh:
port: 22
use_agent: false # set to true if your key has a passphrase
public_key_path: "~/.ssh/id_rsa.pub"
private_key_path: "~/.ssh/id_rsa"
allowed_networks:
ssh:
- 0.0.0.0/0
api:
- 0.0.0.0/0
public_network:
ipv4: false
ipv6: false
private_network:
enabled : true
subnet: 10.12.8.0/22
existing_network_name: "my-network"
disable_flannel: false # set to true if you want to install a different CNI
schedule_workloads_on_masters: false
masters_pool:
instance_type: cpx31
instance_count: 1
location: nbg1
worker_node_pools:
- name: jira-node
instance_type: ccx33
instance_count: 1
location: nbg1
post_create_commands:
- export IP=$(ip addr show enp7s0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
- >
printf "network## {config## disabled}" |
sed 's/##/:/g' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
- >
printf "network##\n version## 2\n renderer## networkd\n ethernets##\n enp7s0##\n addresses##\n - $IP/32\n routes##\n - to## default\n via## xx.xx.xx.xx\n - to## xx.xx.xx.xx/32\n scope## link\n - to## xx.xx.xx.xx/32\n scope## link\n - to## xx.xx.xx.xx/8\n via## xx.xx.xx.xx\n on-link## true\n - to## xx.xx.xx.xx/12\n via## xx.xx.xx.xx\n on-link## true\n - to## xx.xx.xx.xx/16\n via## xx.xx.xx.xx\n on-link## true\n" |
sed 's/##/:/g' > /etc/netplan/50-cloud-init.yaml
- netplan generate
- netplan applyAnd the resulting log: |
Beta Was this translation helpful? Give feedback.
-
|
Hi! From the log it seems the problem is SSH connectivity. Are you running hetzner-k3s from an instance in the same private network since the public IP is disabled? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @vitobotta, yeah when hetzner-k3s tries to access the server via ssh it is not working because the routes are not created yet. The instance is in the same VPN as my local machine and a few minutes later, when cloud init is finished, I can access the server via ssh. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I've just opened a pull request for this. As soon as I set the timeout to 10 minutes, everything works great. It would be cool if it would come in, so we could use this great tool. Thanks a lot! |
Beta Was this translation helpful? Give feedback.
-
|
sharing also my post_create_commands that worked for me (using networkd instead of netplan) and an instance as NAT gateway in the same private network |
Beta Was this translation helpful? Give feedback.
-
Some pre-condition:
existing_networkExample config for test cluster:
Some explanations:
post_create_commandsfor cloud init will create own network configuration on VM creation:directly the yaml parser crashes. So I use##and replace it the end. Didn't found anythink to mask the:(Maybe you found it ;-))Deployment grafik
Small graphic to get an overview (Network is different to example config):
Beta Was this translation helpful? Give feedback.
All reactions