This book is about creating a router for a home / office network using a Linux VM on Proxmox.
- HP t740
- Quad core CPU
- 32GB of RAM
- 1TB NVME
- Dual 10Gb NIC installed in PCIE slot.
Download the latest Proxmox VE release here.
Write the .iso image file to a suitable USB drive with dd or
UNetbootin.
During installation, you will need to plug in a monitor, keyboard, mouse, and the USB device into the host system. Boot the router host from the USB drive.
Select Install Proxmox VE (Graphical) and press Enter.
Click I agree.
The device you select will be automatically partitioned for LVM storage:
/img/proxmox/select-storage.webp
Click Next.
/img/proxmox/configure-locale.webp
Click Next.
/img/proxmox/set-password.webp
Click Next.
Configure a static IP address and hostname for the management network interface:
/img/proxmox/management-interface.webp
The management interface is used to administer the Proxmox host from
your existing network, and also used for the default vmbr0 bridge
network and provides Internet access to the proxmox host.
On the Summary page, confirm the settings you chose, then click
Install.
When the installer finishes, it will automatically reboot the host. Once fully rebooted, you should see the text console message indicating the URL to log into the management interface:
Welcome to the Proxmox Virtual Environment. Please use your web browser to configure this server - connect to https://192.168.1.2:8006
At this point you may remove the keyboard, monitor, mouse, and USB device. You will only need to connect by SSH or by the management web interface.
The Proxmox host has SSH enabled by default and so you can access it via the IP address and password you chose during install.
The next step will be to improve security by setting up your workstation SSH key and to disable authentication via password.
For more information see the chapter on SSH in the Linux Workstation book.
Copy your public SSH to the Proxmox host:
When prompted, you must enter the password you chose during install:
If successful, it will indicate how many keys it copied:
Now that you have tested that you can Login to the Proxmox host using your key, you should reconfigure SSH to disable password authentication:
If you see Permission denied (publickey), and it does not prompt you
for a password for foo, you have successfully disabled password
authentication.
Open the web console at https://X.X.X.X:8006 (replace X.X.X.X with the proxmox IP address.)
You will see a security warning in your browser because of the self-signed certificate. Your browser should have an option to proceed anyway.
At the Proxmox VE Login prompt, enter your credentials:
- username:
root - password: (the password you set during install)
- Realm: Linux PAM standard authentication (Default)
Proxmox is open-source and is completely free to download and use.
However, there may be some confusion at first, when you see the
following message in the web console stating No valid subscription:
/img/proxmox/no-valid-subscription.webp
The default installation is configured to use the Enterprise
repository, which requires a paid Enterprise license, and this also
grants you professional support. For now, you may simply click OK to
bypass the nag screen.
To get an enterprise license visit Proxmox Support.
If you don’t want to pay for an enterprise license, you may enable the
free community repository which allows you to access Proxmox’s
no-subscription version, which is ideal for individual users,
small-scale setups, or open-source enthusiasts who want the full
Proxmox experience but without professional updates and support.
Run this command to enable the community repository:
Proxmox has a multi-layer firewall, which can be enabled at the Datacenter, PVE node, and VM levels. The Datacenter firewall sets the default policy. The Node firewall will be used to protect the PVE node itself, while the VM firewalls protects individual VMs.
By default, the firewalls are disabled. Reset the firewall, add some basic rules, and enable the firewalls now:
Download and run the [proxmox_firewall.sh](https://blog.rymcg.tech/blog/proxmox/01-install/#the-script) script:
Proxmox VMs are configured by default to use bridge networking with your LAN. While great for application servers, this is undesirable when creating a router.
Download and run the [proxmox_nat.sh](https://blog.rymcg.tech/blog/proxmox/02-networking/#the-script) script:
The script outputs the current list of bridges (just vmbr0 by
default) and the menu of options.
Enter the c command to create a new bridge interface:
Enter the existing upstream bridge interface (vmbr0):
Enter the number 1 to create the new NAT interface (vmbr1):
Configure the PVE host IP address for the new interface:
Now quit the script by entering q.
Download and run the [proxmox_kvm.sh](https://blog.rymcg.tech/blog/proxmox/05-kvm-templates/#the-script) script:
Create the template:
Create the router VM with the IP address 10.10.1.2 and the mnemonic
ID 102:
You need to find the device ID of the Ethernet controller that you are going to passthrough to the router VM:
To pass the device to the VM, you only need to pass the Bus and Device
ID: 01:00. The function suffixes .0 and .1 can be ignored,
because the entire device will be passed through to the VM, including
all of these PCI functions.
Once you found the device ID (e.g., 01:00), passthrough the device
to the router VM:
Before starting the VM for the first time, take an initial snapshot:
Create a new entry for the router VM in ~/.ssh/config :
You can access the VM directly from your workstation by setting a proxy jump in your ssh config:
Replace X.X.X.X with the management IP address of your proxmox host.
Copy your workstation key to the router VM:
The first time you connect, you will need to accept the SSH fingerprint
the first time, type yes and press Enter.
[nifty-filter](https://github.com/EnigmaCurry/nifty-filter) is used as a convenient method of configuring nftables.
The Fedora cloud image the VM is booted from is automatically setup for NetworkManager. For a router, this just gets in the way of things, so you should disable NetworkManager and setup static IP addresses via systemd-networkd.
First, create a static network config for the management interface
(name eth0 by default, but it will be renamed to mgmt):
Enable systemd-networkd:
Disable NetworkManager:
Disable cloud-init:
cloud-init was useful for first boot configuration, but now that its done its job, it needs to get out of the way.
List the detected network interfaces on the router VM:
The physical dual 10G Ethernet passed into the VM is recognized as two
interfaces, enp1s0f0 and enp1s0f1, and will become the new
router’s LAN and WAN ports respectively:
Let’s rename these interfaces so they are easier to identify:
To see the changes, you will have to reboot the VM.
Once rebooted, log back in and check the interface names again:
Show the IP address of the LAN interface:
nftables (netfilter) is used to configure the firewall and routes.
nifty-filter lets you configure your whole router in one shell script,
which is written to /usr/local/sbin/router.sh:
nifty-filter expects the configuration to be passed via environment variables, which this script sets up:
| Name | Description |
|---|---|
| INTERFACE_MGMT | The name of the management network (MGMT) interface. |
| INTERFACE_LAN | The name of the local area network (LAN) interface. |
| INTERFACE_WAN | The name of the wide area network (WAN) interface. |
| SUBNET_LAN | The CIDR notation subnet mask for the LAN clients. |
| ICMP_ACCEPT_LAN | The list of allowed ICMP request types to accept from the LAN. |
| ICMP_ACCEPT_WAN | The list of allowed ICMP request types to accept from the WAN. |
| TCP_ACCEPT_LAN | The list of TCP ports on the router to allow access from the LAN clients. |
| TCP_ACCEPT_WAN | The list of TCP ports on the router to allow access from the WAN clients. |
| UDP_ACCEPT_LAN | The list of UDP ports on the router to allow access from the LAN clients. |
| UDP_ACCEPT_WAN | The list of UDP ports on the router to allow access from the WAN clients. |
| TCP_FORWARD_LAN | The list of TCP forwarding routes to allow from the LAN. |
| TCP_FORWARD_WAN | The list of TCP forwarding routes to allow from the WAN. |
| UDP_FORWARD_LAN | The list of UDP forwarding routes to allow from the LAN. |
| UDP_FORWARD_WAN | The list of UDP forwarding routes to allow from the WAN. |
DHCP is the process your LAN clients perform when they are first
connected to the network: the client asks the router to configure an
IP address for them, they recieve the address, and can now get online.
DNS is how your LAN clients can ask the router what the IP address is
for any domain name, allowing you to easily navigate the Internet by
entering names like example.com.
To configure DHCP and DNS for your LAN will require two interrelated services to run on the router:
- dnscrypt-proxy is a local caching DNS proxy which can be configured to use Cloudflare DNS over HTTP (DoH) as the upstream resolver, or any other provider you choose.
- dnsmasq is a DHCP server and DNS forwarder for small networks.
nifty-filter can also configure dnsmasq. Specify your entire config in
a shell script /usr/local/sbin/dnsmasq-lan.sh:
dnsmasq is setup to prioritize any host names defined in /etc/hosts
and to resolve these by itself (without forwarding the query to
dnscrypt). You can put any names you want in this file, whether they
are masking real domains, or even if they are completely made up.
# Example /etc/hosts file 192.168.10.2 foo foo.lan.example.com 192.168.10.3 bar boatymcboatface bar.lan.example.com
You can connect up 254 LAN clients to the physical lan interface of
the VM. For more than one connection, you’ll need to use a network
switch in between.
For most clients, you simply need to plug the cable in and they will automatically configure themselves via DHCP.
To make DHCP explicit on the client, configure systemd-network:
Verify the interface has the proper IP address for the LAN network.
Test that the client can ping the router IP address:
Test that the client can ping an Internet IP address:
Verify that the client DNS resolver is the router IP address and LAN domain name:
Test that the client can query DNS names on the Internet (this query goes through dnscrypt):
Test that the client can query DNS names from the router /etc/hosts
file (this query is handled by dnsmasq directly):
Test that you can ping an Internet server by name:
Find your public gateway IP address:
Let’s recap what has been accomplished so far:
- Proxmox has been installed to create a router VM with PCI passthrough of two physical network interfaces.
- The VM uses one interface for the LAN and the other interface for
the WAN, and they are renamed
lanandwanrespectively. - The router forwards connections from
lantowan(IP masquerade / SNAT), providing Internet access to the LAN clients. - The router forwards connections from
wanto specific routes onlan(DNAT), making select services available from the Internet. - The router provides a DHCP and DNS service to the LAN, which assists LAN clients to get online quickly with minimal client side config.
- Up to 254 LAN clients can share the same switch and router and all clients will be on the same native subnet.
This physically segmented archicture is fine, but it has some major drawbacks as well:
- Its a lot of ethernet cables.
- Unless you run more than one cable to each room, a client’s network subnet is determined by its room location. You can’t do work in the family den. You can’t print from the office.
- The number of LANs you can create is limited by the number of physical interfaces on your router.
This chapter will discuss a superiour strategy: VLANs
- VLANs allow multiple segmented networks to share a single ethernet cable (up to 4096 VLANs).
- VLANs require a “managed” network switch, which means it has an admin tool that allows you to configure each switch port to be on a different (or multiple) VLANs.
- If you put a VLAN aware switch in each room, you only need one cable going to each room, and you can setup clients in that room to access any of your VLANs.
The VLAN architecture looks like this:
To retrofit the native LAN config into a VLAN config you need to do the following:
- Rename the
laninterface totrunkwhich now will carry all of our VLAN traffic. - Create new virtual interfaces for each desired VLAN:
lan@trunk,work@trunk,apps@trunk,iot@trunk. - Configure one or more ports on the managed switch to use specific VLANs.
The existing lan config needs to be removed:
Reboot the router for the interface name change to take effect:
Wait for it to reboot, log back in, and verify the name change for
trunk:
List all of the VLAN names you need in the trunk interface config:
It is important that you protect the trunk interface of the router with a managed switch. You should not let any client connect directly to the trunk interface – you must use a managed switch as an intermediary. One of the job’s of the switch is to enforce VLAN tagging on specific switch ports. Don’t require your clients to provide their own VLAN config: configure the VLANs on the switch itself and assign each port to specific VLANs.
In this example, we will be working with a Sodola 9 port web managed switch (PDF manual here).
- Ports 1-8 operate at 2.5Gbps, and these will be dedicated to specific VLANs.
- Port 9 operates at 10Gpbs, and this will provide the trunk containing all VLANs.
These ports will be assigned as follows:
/img/router/sodola-9ports.webp
- Port 1 - MGMT is used only for the management of the Sodola
switch. Factory IP address:
192.168.2.1. - Port 2 - LAN (VLAN 10 untagged) -
192.168.10.1/24subnet. - Port 3 - LAN (VLAN 10 untagged) -
192.168.10.1/24subnet. - Port 4 - LAN (VLAN 10 untagged) -
192.168.10.1/24subnet. - Port 5 - WORK (VLAN 11 untagged) -
192.168.11.1/24subnet. - Port 6 - APPS (VLAN 12 untagged) -
192.168.12.1/24subnet. - Port 7 - APPS (VLAN 12 untagged) -
192.168.12.1/24subnet. - Port 8 - IOT (VLAN 13 untagged) -
192.168.13.1/24subnet. - Port 9 - TRUNK (all tagged VLANs @10Gbps.)
Plug a workstation client into port 1 and set a static IP address of
192.168.2.2. Open your browser to http://192.168.2.1.
Then open your browser to http://localhost:8000.
Enter the credentials when prompted:
- Username:
admin - Password:
admin.
Find the Configuration menu.
- Find the
VLANmenu.- Find the
802.1Q VLANmenu.
- Find the
Create the following VLANs:
/img/router/sodola-lan-vlan.webp
- Create VLAN 10:
- VLAN:
10 - VLAN Name:
lan - Untagged Ports: 2,3,4
- Tagged Ports: 9
- Click
Add / Modify
- VLAN:
- Create VLAN 11:
- VLAN:
11 - VLAN Name:
work - Untagged Ports: 5
- Tagged Ports: 9
- Click
Add / Modify
- VLAN:
- Create VLAN 12:
- VLAN:
12 - VLAN Name:
apps - Untagged Ports: 6,7
- Tagged Ports: 9
- Click
Add / Modify
- VLAN:
- Create VLAN 13:
- VLAN:
13 - VLAN Name:
iot - Untagged Ports: 8
- Tagged Ports: 9
- Click
Add / Modify
- VLAN:
Configure the PVID of each VLAN:
- Find the
802.1Q VIDmenu:
/img/router/sodola-lan-vlan-vid.webp
- For each port:
- Set the PVID, which sets the native VLAN for that port.
- Set the Accepted Frame Type, which sets the expected type of data to come in that port.
- For MGMT (port 1), leave this as the default
All. - For VLANs (ports 2-8..), set this to
Untag-onlyand then native traffic will become tagged for that VLAN automatically. - For TRUNK (port 9), set this to
Tag-onlyand all traffic pre-tagged will flow unmodified.
- For MGMT (port 1), leave this as the default
If you have the managed router switch in a central location in a home, you can put unmanaged switches in every room that only require access to one VLAN. For rooms that require access to more than one VLAN, it requires placing another managed switch in that room, and then configuring the port for tagged traffic only, which creates a kind of sub-trunk port between the two managed switches:
The port isolation setting can further isolate the networks between the VLAN ports. For example, ports 2,3,4 and 9 all need to talk to each other, but ports 1,5,6,7,8 should not be able to participate in those conversations. The isolation list can make this communication explicitly denied.
- Find the
Configurationmenu:- Find the
Port Isolationmenu:- Using the multi-select boxes, set the port isolation list for
each port, and then click the
Applybutton.
- Using the multi-select boxes, set the port isolation list for
each port, and then click the
- Find the
Here are the final port isolation lists of the example:
/img/router/sodola-port-isolation.webp
- Port 1 (MGMT):
- Isolation list:
1 - Since this is the management interface of the switch itself, it doesn’t need to talk to any other ports.
- Isolation list:
- Port 2 (LAN):
- Isolation list:
2-4,9 - LAN ports can talk to each other and to TRUNK.
- Isolation list:
- Port 3 (LAN):
- Isolation list:
2-4,9 - LAN ports can talk to each other and to TRUNK.
- Isolation list:
- Port 4 (LAN):
- Isolation list:
2-4,9 - LAN ports can talk to each other and to TRUNK.
- Isolation list:
- Port 5 (WORK):
- Isolation list:
5,9 - Since there’s only one WORK port, it only needs to talk to TRUNK.
- Isolation list:
- Port 6 (APPS):
- Isolation list:
6-7,9 - APPS can talk to each other and to TRUNK.
- Isolation list:
- Port 7 (APPS):
- Isolation list:
6-7,9 - APPS can talk to each other and to TRUNK.
- Isolation list:
- Port 8 (IOT):
- Isolation list:
8-9 - Since there’s only one IOT port, it only needs to talk to TRUNK.
- Isolation list:
- Port 9 (TRUNK):
- Isolation list:
2-9 - TRUNK can talk to any port except for MGMT (port 1)
- Isolation list:
After making changes on the switch you must save the config:
- Find the
Toolsmenu:- Find the
Savemenu:- Click the
Savebutton.
- Click the
- Find the
Rebootmenu:- Click the
Rebootbutton to test that the config is still applied on next boot.
- Click the
- Find the