This subdirectory contains scripts to build a small and cheap but functional physical network that's compatible with the multicast ingest platform, in that:
- it will propagate PIM joins for external sources to the ingest-rtr
- it will forward multicast traffic from outside the network to the receivers
The expectation is that if you're running the devices in this network, you'll also have the ingest-rtr from the main section of the repo attached to the brf0 interface of what I'm calling the "border router".
(NB: This border router is for travel and home/small office use, and it just uses dhcp to get an assigned upstream IP and has its default route point out that way, almost making the full local network look like a single consumer device. It's not actually configured as what you'd usually call a "border router", with BGP. FRR does support BGP and you can configure it if you want to by editing the contents of the files in /etc/frr/ appropriately.)
The devices I've used so far for the multicast ingest platform and its supporting network were the cheapest off-the-shelf microcomputers I've got lying around, suitable for traveling:
There's a few different devices I've used for different builds of this network, including:
- Zotac CI329 nano (pictured)
- Minisforum Z83-F
- GUZILA Fanless Mini PC (This worked but is tight as the border-rtr because you use both USB ports for ethernet, so you can't also use a keyboard)
You'll need 2 of these (for the border-rtr and the access-rtr) plus another for your ingest-rtr if you're using the same platform.
You'll need for your border-rtr to have 3 ethernet interfaces and your access-rtr to have 2, so for the minisforum or guzila you'll need an extra 3 Ethernet-to-USB adapters, or just 1 for the zotac since it has 2 ethernet ports built in.
Depending what kinds of receivers you'll be connecting, you'll also probably want an ethernet switch to connect them, or possibly a wi-fi router.
If you use a wi-fi router, you'll need one that does IGMP Proxying. I know the Fritz!Box and the Nighthawk include IGMP proxying support, and I also know OpenWRT with the mcproxy package installed works.
Anyway, these devices started this life as a default install of Ubuntu Server 18.04. You can stick with the defaults, or select the "docker" snap (or not--it'll be installed by the setup script on ingest-rtr).
I expect any setup suitable for FRRouting will work the same, but these instructions include all the commands and configs starting after completing this install and logging into the new machines.
After that, Free Range Routing is launched, and the config in this repo is applied on the border-rtr and the access-rtr (plus the ingest-rtr config), to bring up this very simple lab network:
Think of this as a simplified version of one of the examples in the DRIAD spec.
Normal lab machine setup:
-
add your keys and
chmod 0600 .ssh/authorized_keys, if desired; maybe visudo to adduser ALL=NOPASSWD: ALLat the end if you're crazy and/or well-isolated. -
catch up your updates (with thanks)
-
prevent your drive from filling up with old kernels over time, since updates are for some reason auto-downloaded but not auto-cleaned by default:
-
restart
sudo bash -x -e <<EOF
export DEBIAN_FRONTEND=noninteractive
export APT_LISTCHANGES_FRONTEND=none
echo 'libc6 libraries/restart-without-asking boolean true' | debconf-set-selections
apt-get update
apt-get --allow-downgrades --allow-remove-essential --allow-change-held-packages -o Dpkg::Options::="--force-confold" --force-yes -o Dpkg::Options::="--force-confdef" -fuy dist-upgrade
apt-get -y autoremove
echo 'Unattended-Upgrade::Remove-Unused-Dependencies "true";' | \
sudo tee -a /etc/apt/apt.conf.d/50unattended-upgrades
echo 'Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";' | \
sudo tee -a /etc/apt/apt.conf.d/50unattended-upgrades
echo 'Unattended-Upgrade::Automatic-Reboot-Time "07:15";' | \
sudo tee -a /etc/apt/apt.conf.d/50unattended-upgrades
echo 'Unattended-Upgrade::Automatic-Reboot "true";' | \
sudo tee -a /etc/apt/apt.conf.d/50unattended-upgrades
reboot
EOFThe basic process for each of the routers is the same, from the different directories:
border-rtr/access-rtr/
# border-rtr
git clone https://github.com/GrumpyOldTroll/multicast-ingest-platform.git
cd multicast-ingest-platform/
cd sample-network/border-rtr/
./setup.sh && ../docker-frr.sh && sudo reboot# access-rtr
git clone https://github.com/GrumpyOldTroll/multicast-ingest-platform.git
cd multicast-ingest-platform/
cd sample-network/access-rtr/
./setup.sh && ../docker-frr.sh && sudo rebootAfter a successful install, it should be possible to reboot and come back up gracefully.
For ease of config, I've got dhcp client on for ingest-rtr and access-rtr running on the upstream interface, so it still should work ok after configuring it, whether or not you're running through the border-rtr and have the topology set up, in theory.
Check the scripts for details. In both cases, it builds frr locally from source and installs it, and turns on ip forwarding.
If you're using USB-to-Ethernet adapters, you'll need them plugged for this stage (or it gives an early error for not having enough interfaces). The config will incorporate the MAC of the chosen adapter, so if you shuffle them later you'll need to update the config like the "Interface Setup" section below describes.
Extra details:
- border-rtr:
- runs dhcp client upstream
- runs dhcp server on both downstream interfaces (mainly for convenience during setup of the other 2, and could reasonably be turned off after they're up)
- runs a dns server
- runs a NAT with iptables for traffic from downstream
- access-rtr:
- runs dhcp server downstream
- uses border-rtr's dns server
If you want to build frr and run it as a service instead of using docker images for frr, you can use ../build-frr.sh instead of ../docker-frr.sh in the above installation steps. This way got superseded by the docker approach, but the older scripts to support it are still there in case anyone prefers it (especially if they need to build local patches for frr).
There's also scripts in ingest-host to bring the ingest-rtr from a default ubuntu install to an active ingest router. It's similar to the access-rtr and border-rtr, but because the docker commands need the interface name to be set (to irf0), there's a post-reboot script to run.
# ingest-rtr
git clone https://github.com/GrumpyOldTroll/multicast-ingest-platform.git
cd multicast-ingest-platform/
cd sample-network/ingest-host/
./setup.sh && sudo reboot
# then log back in, go back to ingest-host:
./setup2.shThis section is about the cryptic messages you should hopefully get about interface names and mac addresses at the end of setup.sh.
Ubuntu 18 started using interface names that depend on the hardware.
There are pros and cons to that, but with the USB-to-Ethernet adapters I'm using, I was getting names that incorporate the MAC address (e.g. "enxc8b3730f34bc"), which makes the network config files non-portable to other systems if used directly.
I'm working around this by MAC address with udev rules. I'm auto-generating the mac addresses from the output of ip link show, so things might go wrong if there's any unexpected slight mismatches.
If you want to use the config files from this setup as they stand, you MUST use the exact interface names here, with your own MAC addresses (or other selection criteria).
You can of course configure your system as you like and change the names, but you should make sure to change them in all the other config files also, or your packets are going to have a bad day.
Some claim that changing udev rules also requires `sudo update-initramfs -u' to get it to work, but that was not my experience with Ubuntu 18.04.2. YMMV, but try that if you're having trouble.
The setup.sh scripts attempt to generate these files, but you should check them to make sure the interfaces are the ones you want, and edit them if necessary.
- /etc/udev/rules.d/10-border-rtr-inames.rules
- /etc/udev/rules.d/10-access-rtr-inames.rules
The interface names to configure are:
- border-rtr:
- bup0: border-rtr upstream
- bdn0: border-rtr downstream
- brf0: border-rtr reflector
- access-rtr:
- xup0: access-rtr upstream
- xdn0: access-rtr downstream
This is an example of what one of the udev rules might look like.
Remember to use your own mac addresses.
# /etc/udev/rules.d/10-border-rtr-inames.rules
# upstream, to internet
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:e0:4c:c1:55:1e", NAME="bup0"
# downstream, to access router
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="c8:b3:73:0f:34:bc", NAME="bdn0"
# reflector, to ingest router
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="c8:b3:73:0f:2f:c4", NAME="brf0"There are other approaches besides the one I'm using for configuring interface names. It looks like systemd.link is a recommended way to do this, by creating files with names ending in ".link" in /etc/systemd/network or /usr/lib/systemd/network. However, it didn't work for me on Ubuntu 18.04.2. Perhaps this will change in the future, and if you're having trouble with the udev rules, consider trying it.
In the access-rtr, I'm using xdn0 and xup0 instead of adn0 and aup0.
I think there's a bug, presumably in docker, that has to do with the interface loading order, such that a docker container that tries to use interfaces with a lexically earlier name than 'bridge' has problems hooking up the interfaces properly.
This may need further investigation, but when I was running frr inside a docker container, it worked with xup/xdn after NOT working with aup/adn. Saw the same behavior one other time in a different context, but haven't dug in well enough to make a solid bug report.
The script also sets up netplan to use these interfaces instead of the auto-configured interface name:
These should match up with the network diagram, but if you've messed with anything, you'll want to make sure to update these.
Notes:
-
access-rtr uses a static IP and assumes connectivity through the border-rtr. border-rtr offers dhcp server through both interfaces, so it should work to just plug in the upstream connection through border-rtr while configuring access-rtr and ingest-rtr, and then if everything's hooked up right, it'll all be fine when it reboots. But sometimes there's trouble.
-
the access-rtr netplan uses the border router's recursive DNS server. This lets border-rtr inject DRIAD answers for testing setups that don't have globally deployed DNS zones.
See this example DRIAD zone file. This example is no longer loaded because now there's now global TYPE260 records for the reverse zone for 23.212.185.x, but earlier testing versions of this setup loaded this zone file in the border-rtr's bind instance, and you may want to configure similar for your own addresses.
Once it's all set up, you should be able to plug in a client downstream of the access (it should get an address automatically with dhcp), and have it do a SSM join (for instance with iperf-ssm), and see pimwatch on border-rtr react by trying to find an AMT relay, and subscribing to traffic if possible.
For example, this should work if you've got docker and my sender is still alive and properly configured (you get about 1 packet per second, and it's a good way to watch the pcaps from various points on the network):
docker run -it --rm --name rx2 grumpyoldtroll/iperf-ssm:latest --server --udp --bind 232.10.1.1 --source 23.212.185.5 --interval 1 --len 1500 --interface eth0The frr config is in /etc/frr/.
The running frr process is basically sort-of like a Cisco IOS command line with slightly different commands, if you run vtysh to connect to it:
sudo /usr/bin/vtysh
border-rtr# show ip pim neighbor
Interface Neighbor Uptime Holdtime DR Pri
bdn0 10.8.1.2 00:02:38 00:01:37 1
brf0 10.9.1.2 00:02:38 00:01:37 1And be sure to read through the user manual.