Skip to content

Environment Setup

Jump to bottom
Jack Wampler edited this page May 20, 2021 · 8 revisions

These instructions document the process to set up a development environment for running a Conjure station. The same steps can be used to set up a new production environment as well.

This setup was done on a local router so the ip addresses used are from rfc 1918.

See the image in this repo paper/figures/dev_env.png for a basic layout of this guide

ON ROUTER (openwrt):

Install some non standard openwrt packages that we are going to need

opkg update
opkg install iptables-mod-tee
opkg install ip-full

    # for debugging not absolutely necessary
opkg install tcpdump

Add new interface for subnet 10.0.1.1/24 with the router at 10.0.1.1

ip addr add 10.0.1.1/24 dev br-lan

Add static ARP route (the server has 2 ethernet connections) connecting address and MAC. This used ... neigh replace... as the address on the router had been co-opted the by another interface on the server (acting as a proxy). If the ARP route hasn't been incorrectly assigned to another interface you would use ... neigh add ....

ip neigh replace 10.0.1.2 lladdr 68:05:ca:17:c4:96 nud permanent dev br-lan
# ip neigh add 10.0.1.2 lladdr 68:05:ca:17:c4:96 nud permanent dev br-lan

NOTE: IF YOU MESS UP THE MAC ADDRESS IN THE STATIC ARP RECORD TRAFFIC WILL GO TO WHATEVER MAC YOU PUT IN. STATIC ADDRESSING ON DEVICES DOES NOT MATTER.

Next port mirror traffic to the tap (server) by adding to file /etc/firewall.user

iptables -t mangle -A PREROUTING 1 -d 192.168.1.0/24 -j TEE --gw 10.0.1.2
iptables -t mangle -A POSTROUTING 1 -s 192.168.1.0/24 -j TEE --gw 10.0.1.2

Additionally, newer versions of OpenWRT contain a rule that drops packets with WAN destinations that don't have an established conntrack entry from the WAN interface; this is problematic if the client is on the local network. This may manifest itself in the station receiving the initial SYN packet from the client, but not the subsequent ACK. This can be disabled by inserting the following into /etc/config/firewall under the zone beginning with option name 'wan':

option masq_allow_invalid '1'

The full WAN zone may look something like this:

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option output 'ACCEPT'
        option masq '1'
        option mtu_fix '1'
        option input 'REJECT'
        option forward 'REJECT'
        option masq_allow_invalid '1'

This option can also be enabled in the WebUI on OpenWRT 19.07+ via Network > Firewall > Zones > WAN Zone > Edit > Conntrack Settings > "Allow "invalid" traffic".

Reload the firewall config to apply the routing rules:

/etc/init.d/firewall reload
    # or
# service firewall restart

ON SERVER:

You should now see traffic on the interface that was designated to be the TAP. test with

sudo tcpdump -i enp2s0  host 52.44.73.6

Once Per machine

Add static address for interfaceso that ubuntu never sends DHCP requests for tap interface by appending the following to /etc/network/interfaces

auto enp2s0
iface enp2s0 inet static
  address 10.0.1.2
  netmask 255.255.255.0

Restart the networking service

sudo service networking reload

Note: if adding routing rules gives and error about non-existent table you need to add the custom routing table using . Add custom ip routing table

echo "200 custom" >> /etc/iproute2/rt_tables

Once Per reboot

Set the global rp_filter to allow packets with falsified ip addresses to pass through

sysctl -w net.ipv4.conf.all.rp_filter=0

Set up the internal tunnels to route packets from rust process_packet to the golang application. Do this for each tunnel being created (one per core being used). Note that the IPtables rules need to point to the interface NOT BEING USED AS A TAP.

    # create tun0 interface
ip tuntap del mode tun tun0
ip tuntap add mode tun tun0
ip rule add iif tun0 lookup custom
ip route add local 0.0.0.0/0 dev tun0 table custom

    # rp_filter 
sysctl -w net.ipv4.conf.tun0.rp_filter=0

    #iptables DNAT routing for packets on the tunX interface
iptables -t nat -I PREROUTING 1 -p tcp -i tun0 -j DNAT --to 192.168.1.101:41245
iptables -I INPUT 1 -i tun0 -j ACCEPT
Peer-to-Peer Transports

To support the "peer-to-peer" transport model, in which the station directly connects to a listening client, we need to do a bit of extra work in the firewalls. Because the sockets are bound to phantom (non-local) addresses, we need to force inbound traffic on the tap interface to be accepted—the first thing we need to do is set rp_filter to loose (2) on our tap interface. Although we set rp_filter=0 on all above, the kernel chooses the higher value of all and the specific interface, which means we need to manually set it on the tap interface as well. Don't worry—we'll introduce a rule later to drop all unwanted traffic after we check it for the traffic we're looking for.

sysctl -w net.ipv4.conf.<tap interface>.rp_filter=2

Next, we'll set up a custom chain in the mangle table, which will allow us to easily perform multiple actions on packets.

iptables -t mangle -N conjure-p2p-inbound

Every packet that hits this chain will be marked for routing purposes later, and will be accepted:

iptables -t mangle -A conjure-p2p-inbound -j MARK --set-mark 1
iptables -t mangle -A conjure-p2p-inbound -j ACCEPT

Next, we'll check packets on our tap interface at the mangle table's PREROUTING chain for being associated with a transparent (spoofed source) socket, which will redirect to our custom chain.

iptables -t mangle -I PREROUTING -i <tap interface> -m socket --transparent -j conjure-p2p-inbound

Although we set rp_filter to loose, we don't want to keep all of its effects—we only want to allow traffic that we're expecting, and any other traffic should still be dropped as though rp_filter was set to strict mode. To do this, we can insert another rule after the above rule (notice the index 2 on -I) to drop any packets on the tap interface that don't pass rp_filter.

iptables -t mangle -I PREROUTING 2 -i <tap interface> -m rpfilter --invert -j DROP

Finally, we'll use the mark on the packets to force them into our custom route table set up above, which interprets 0.0.0.0/0 (any address) as a local address, allowing ingest into our application.

ip rule add fwmark 1 lookup custom

That's it! Now any socket with the IP_TRANSPARENT socket option should be able to bind a phantom address and hold a conversation with a client over the internet.

Troubleshooting:

List iptables rules for inspection

#list rules for specific table
iptables --table nat --list

#shortens to
iptables -t nat -L

# [-v] verbose output (all fields) with [-n] numeric usage information
iptables -t nat -v -n -L

# clear all counters for iptables rules
iptables -Z

# clear counters for specific table
iptables -t nat -Z

# Clear counters for specific chain within table 
iptables -t nat -Z PREROUTING 

# Clear counters for specific rule
iptables -t nat -Z PREROUTING 1

List rules added to ip

ip rule list

list routing information relevant to tuns

ip route show table all | grep tun

Delete all tun0 rules in the ip rule list based on interface (iif) name. If you run the singular version it only deletes one rule.

while sudo ip rule delete iif tun0 2>/dev/null; do true; done
Clone this wiki locally