Skip to content

An IPv4 router over netmap for FreeBSD

Notifications You must be signed in to change notification settings

ArchyCho/netmap-fwd

 
 

Repository files navigation

netmap-fwd was designed to require little to no setup.

Consider the following case:

[Test1]<---->[Router]<----->[Test2]<--->[Internet]

Where:

Test1:

dwc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=80008<VLAN_MTU,LINKSTATE>
       ether 02:13:06:c3:52:48
       inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

With a default route to 10.0.0.1

Test2:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
       ether e8:40:f2:c2:98:00
       inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

There is a static route to the 10.0.0.0/24 network:
  route to: 10.0.0.0
destination: 10.0.0.0
      mask: 255.255.255.0
   gateway: 192.168.0.88
       fib: 0
 interface: em0
     flags: <UP,GATEWAY,DONE,STATIC>
recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
      0         0         0         0      1500         1         0

Router:

igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
       ether 00:08:a2:09:5b:4b
       inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
       ether 00:08:a2:09:5b:4d
       inet 192.168.0.88 netmask 0xffffff00 broadcast 192.168.0.255
       nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
       media: Ethernet autoselect (1000baseT <full-duplex>)
       status: active

With a default route to 192.168.0.10.

With this setup Test1 can reach Test2 and vice-versa. Test1 can also reach the internet via Test2.

With a working setup, like this one, if I want to replace the in-kernel routing of Router with netmap-fwd, all you have to do is:

# ./netmap-fwd igb3 igb5 &
[1] 5906
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       ether: 00:08:a2:09:5b:4b
       inet 10.0.0.1 netmask 255.255.255.0 broadcast 10.0.0.255
switching interface igb3 to netmap mode.
igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
       ether: 00:08:a2:09:5b:4d
       inet 192.168.0.88 netmask 255.255.255.0 broadcast 192.168.0.255
switching interface igb5 to netmap mode.

netmap-fwd reads the interface IPs (it only works with IPv4 at this
point) and setup everything it needs.  There is a simple CLI to check
ARP and routes:

route show
Routing tables

Internet:
Destination        Gateway            Flags        Netif
0.0.0.0/0          192.168.0.10       UGS          igb5
10.0.0.0/24                           U            igb3
10.0.0.1                              UH           igb3
192.168.0.0/24                        U            igb5
192.168.0.88                          UH           igb5
arp
? (10.0.0.2) at 02:13:06:c3:52:48 on igb3 expires 1182 seconds
? (192.168.0.10) at e8:40:f2:c2:98:00 on igb5 expires 1181 seconds
? (192.168.0.88) at 00:08:a2:09:5b:4d on igb5 permanent
? (10.0.0.1) at 00:08:a2:09:5b:4b on igb3 permanent

And that's all. You're ready to go.

It was designed this way so
	1) It is easy to compare netmap-fwd with in-kernel routing, 
	2) if you kill netmap-fwd the in-kernel routing will continue to work without any change in your configuration.

But as you can see, the are a lot of room for improvements (IPv6, bgpd daemon fib synchronization and a few other details).

netmap-fwd accepts any number of network interfaces which may also include vlans.

Using pkt-gen, on a C2518 (quad core 1.7GHz Atom) 
654.302285 main_thread [1620] 1.175 Mpps (1.176 Mpkts 602.256 Mbps in 1001150 usec) 3.56 avg_batch
655.302630 main_thread [1620] 1.175 Mpps (1.175 Mpkts 601.769 Mbps in 1000345 usec) 3.55 avg_batch
656.304142 main_thread [1620] 1.175 Mpps (1.177 Mpkts 602.478 Mbps in 1001512 usec) 3.55 avg_batch
657.304823 main_thread [1620] 999.767 Kpps (1.000 Mpkts 512.229 Mbps in 1000681 usec) 3.53 avg_batch
[…]
Received 25618109 packets 1639558976 bytes 7225406 events 64 bytes each in 21.80 seconds.
Speed: 1.175 Mpps Bandwidth: 601.568 Mbps (raw 827.156 Mbps). Average batch: 3.55 pkts

FreeBSD can't do more than 233Kpps on the same hardware (single IP flow)

A bit more data:

Device under test				Network interface	Kernel forwarding	Fastforward enabled	netmap-fwd 
C2358 (2 core, 1.7 GHz, 4 GB RAM)		Intel I354		123 kpps		217 kpps		945 kpps
C2758 (8 core, 2.4 GHz, 8 GB RAM)		Chelsio T520 10G	270 kpps		426 kpps		1.683 Mpps
Xeon-D 1540 (8 core, 2 GHz, 32 GB RAM)		Intel X552 10G		439 kpps		557 kpps		2.230 Mpps
Xeon E3-1275 (4 core, 3.5 GHz, 32 GB RAM)	Intel X520-2 10G	1.058 Mpps		1.331 Mpps		5.053 Mpp

Luiz

About

An IPv4 router over netmap for FreeBSD

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • C 98.6%
  • Other 1.4%