Ubiquiti EdgeMax configuration

[dmwyatt]

Anyone have any ideas or advice on how to configure an EdgeRouter Lite to send all internet traffic on LAN through the algo server?

Specifically, I'm looking for step-by-step instructions to enable my EdgeRouter Lite to route all internet traffic on my LAN through my algo VPN instead of the open internet.

Here's my thread on the ubiquiti forums linking to this issue.

[chriseldredge]

I just started looking at this too and found these community discussions that may be helpful:

• EdgeRouter strongSwan configuration
• Route all traffic through VPN

[ebrandell-2]

Also very interested in this.

On a side note, I wasn't even entirely sure this router would support Algo. The only thing I wasn't able to confirm after talking with Ubiquiti's support was support for ECDSA certificate keys. I did, however, find this in the release notes for version 1.8.0 of EdgeOS (it's currently on 1.9.1):

[IPsec] Add the include-ipsec-secrets option for including a custom secrets file. This can be useful for example if an ECDSA key is used. Implemented by TriJetScud

This tells me it can support it with the latest versions of EdgeOS, but once I get mine I'll test it out.

[dmwyatt]

I found this which I think is what we need, but I'm far from a networking guy so I'm not sure.

[ebrandell-2]

root@ubnt:/etc/ipsec.d/certs# ipsec up ikev2-<ALGO_IP>
initiating IKE_SA ikev2-<ALGO_IP>[8] to <ALGO_IP>
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) ]
sending packet: from <WAN_IP>[500] to <ALGO_IP>[500] (1188 bytes)
received packet: from <ALGO_IP>[500] to <WAN_IP>[500] (265 bytes)
parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
received cert request for "CN=<ALGO_IP>"
sending cert request for "CN=<ALGO_IP>"
no private key found for 'CN=router'
establishing connection 'ikev2-<ALGO_IP>' failed

This is as close as I got to getting it working. I tried starting ipsec through bash since the uqiquiti CLI seems like a PITA to configure. You can do it by using sudo bash and then running ipsec <cmd>.

Output of ipsec statusall:

root@ubnt:/etc/ipsec.d/private# ipsec statusall
Status of IKE charon daemon (strongSwan 5.2.2, Linux 3.10.20-UBNT, mips64):
  uptime: 39 minutes, since Apr 02 08:05:46 2017
  malloc: sbrk 410768, mmap 0, used 264632, free 146136
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 0
  loaded plugins: charon ldap sqlite pkcs11 aes des sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs8 pem openssl agent xcbc cmac ctr ccm gcm curl attr kernel-netlink resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap addrblock
Listening IP addresses:
  <WAN_IP>
  192.168.1.1
  192.168.2.1
Connections:
ikev2-<ALGO_IP>:  %any...<ALGO_IP>  IKEv2, dpddelay=35s
ikev2-<ALGO_IP>:   local:  [CN=router] uses public key authentication
ikev2-<ALGO_IP>:    cert:  "CN=router"
ikev2-<ALGO_IP>:   remote: [<ALGO_IP>] uses public key authentication
ikev2-<ALGO_IP>:   child:  dynamic === 0.0.0.0/0 TUNNEL, dpdaction=clear
Security Associations (0 up, 0 connecting):
  none

My directory structure for certs/keys/etc on my EdgeRouter Lite:

root@ubnt:/etc# tree -pug ipsec.d/
ipsec.d/
|-- [drwxr-xr-x root     root    ]  aacerts
|-- [drwxr-xr-x root     root    ]  acerts
|-- [drwxr-xr-x root     root    ]  cacerts
|   `-- [-rw------- root     root    ]  ca.crt
|-- [drwxr-xr-x root     root    ]  certs
|   |-- [-rw------- root     root    ]  04.pem
|   |-- [-rw------- root     root    ]  <ALGO_IP>.crt
|   `-- [-rw------- root     root    ]  router.crt
|-- [drwxr-xr-x root     root    ]  crls
|-- [drwxr-xr-x root     root    ]  ocspcerts
|-- [drwxr-x--- root     root    ]  private
|   |-- [-rw------- root     root    ]  <ALGO_IP>.key
|   |-- [-rw------- root     root    ]  router.key
|   `-- [-rw------- root     root    ]  router.p12
|-- [drwxr-xr-x root     root    ]  reqs
`-- [drwxr-xr-x root     root    ]  tunnels
    `-- [-rw-r--r-- root     root    ]  remote-access

Perhaps I'm missing some files/permissions?

Any help would be greatly appreciated.

[ebrandell-2]

Got super close last night. It tried installing a new virtual ip on my EdgeRouter, but then right after I lost all internet and LAN access. I configured all of this using bash on my EdgeRouter (not using the built-in CLI). I'm guessing the fact that it blows up my network access has something to do with NAT traversal or iptables firewall rules being missing/misconfigured. I'm using EdgeOS 1.9.1 by the way (here).

My hardware setup:

Internet <- Modem <- ERLite3 <- Asus AC68U
                      	|           |
                        V           V
                       Algo       DHCP Lan Clients

Can someone else try this and let me know?

Some resources, some of which I still have to try):

• IPSEC-Tunnel-UP-but-no-traffic
• EdgeRouter-Set-up-L2TP-over-IPsec-VPN-server
• EdgeRouter-IPSEC-VPN-site-to-site-tunnel-not-passing-traffic
• Firewall-rules-needed-to-pass-IPSEC-point-to-point-traffic

Output of ipsec up <ikev2-ip>:

root@ubnt:/etc# ipsec restart
Stopping strongSwan IPsec failed: starter is not running
Starting strongSwan 5.2.2 IPsec [starter]...
root@ubnt:/etc# ipsec up ikev2-<ALGO_VPN>
initiating IKE_SA ikev2-<ALGO_VPN>[1] to <ALGO_VPN>
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ]
sending packet: from <WAN_IP>[500] to <ALGO_VPN>[500] (1196 bytes)
received packet: from <ALGO_VPN>[500] to <WAN_IP>[500] (273 bytes)
parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ]
received cert request for "CN=<ALGO_VPN>"
sending cert request for "CN=<ALGO_VPN>"
authentication of 'CN=<USER>' (myself) with ECDSA-256 signature successful
sending end entity cert "CN=<USER>"
establishing CHILD_SA ikev2-<ALGO_VPN>
generating IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ IDr AUTH CPRQ(ADDR DNS) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(MULT_AUTH) N(EAP_ONLY) ]
splitting IKE message with length of 943 bytes into 2 fragments
generating IKE_AUTH request 1 [ EF ]
generating IKE_AUTH request 1 [ EF ]
sending packet: from <WAN_IP>[4500] to <ALGO_VPN>[4500] (544 bytes)
sending packet: from <WAN_IP>[4500] to <ALGO_VPN>[4500] (464 bytes)
received packet: from <ALGO_VPN>[4500] to <WAN_IP>[4500] (544 bytes)
parsed IKE_AUTH response 1 [ EF ]
received fragment #1 of 2, waiting for complete IKE message
received packet: from <ALGO_VPN>[4500] to <WAN_IP>[4500] (369 bytes)
parsed IKE_AUTH response 1 [ EF ]
received fragment #2 of 2, reassembling fragmented IKE message
parsed IKE_AUTH response 1 [ IDr CERT AUTH CPRP(ADDR DNS DNS DNS6 DNS6) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_6_ADDR) ]
received end entity cert "CN=<ALGO_VPN>"
  using certificate "CN=<ALGO_VPN>"
  using trusted ca certificate "CN=<ALGO_VPN>"
checking certificate status of "CN=<ALGO_VPN>"
certificate status is not available
  reached self-signed root ca with a path length of 0
authentication of '<ALGO_VPN>' with ECDSA-256 signature successful
IKE_SA ikev2-<ALGO_VPN>[1] established between <WAN_IP>[CN=<USER>]...<ALGO_VPN>[<ALGO_VPN>]
installing DNS server 8.8.8.8 to /etc/resolv.conf
installing DNS server 8.8.4.4 to /etc/resolv.conf
installing DNS server 2001:4860:4860::8888 to /etc/resolv.conf
installing DNS server 2001:4860:4860::8844 to /etc/resolv.conf
installing new virtual IP 10.19.48.1
... <<< dies here

Here's the updated directory structure:

root@ubnt:/etc# tree -pug ipsec.d/
ipsec.d/
|-- [drwxr-xr-x root     root    ]  aacerts
|-- [drwxr-xr-x root     root    ]  acerts
|-- [drwxr-xr-x root     root    ]  cacerts
|   `-- [-rw-r--r-- root     root    ]  ca.crt
|-- [drwxr-xr-x root     root    ]  certs
|   |-- [-rw------- root     root    ]  01.pem
|   |-- [-rw------- root     root    ]  02.pem
|   |-- [-rw------- root     root    ]  <ALGO_IP>.crt
|   `-- [-rw------- root     root    ]  <ALGO_IP>_<USER>.crt
|-- [drwxr-xr-x root     root    ]  crls
|-- [drwxr-xr-x root     root    ]  ecparams
|   `-- [-rw------- root     root    ]  prime256v1.pem
|-- [drwxr-xr-x root     root    ]  ocspcerts
|-- [drwxr-x--- root     root    ]  private
|   |-- [-rw------- root     root    ]  <ALGO_IP>.key
|   |-- [-rw------- root     root    ]  <ALGO_IP>_<USER>.key
|   |-- [-rw------- root     root   ]  <USER>.p12
|   `-- [-rw------- root     root    ]  cakey.pem
|-- [drwxr-xr-x root     root    ]  reqs
`-- [drwxr-xr-x root     root    ]  tunnels
    `-- [-rw-r--r-- root     root    ]  remote-access

ipsec.conf:

conn ikev2-<ALGO_IP>
    fragmentation=yes
    rekey=no
    dpdaction=clear
    keyexchange=ikev2
    compress=no
    dpddelay=35s

    ike=aes128gcm16-sha2_256-prfsha256-ecp256
    esp=aes128gcm16-sha2_256-ecp256

    right=<ALGO_IP>
    rightid=<ALGO_IP>
    rightsubnet=0.0.0.0/0
    rightauth=pubkey

    leftsourceip=%config
    leftauth=pubkey
    leftcert=<ALGO_IP>_<USER>.crt
    leftfirewall=yes
    left=%defaultroute

    auto=add

[ebrandell-2]

Thinking it has to be some sort of issue with firewall or NAT rules. :(

[dmwyatt]

I added a bounty. Check the first post in this issue. You can add to the funds by clicking the badge there.

[ebrandell-2]

Added $100 bounty.

[chriseldredge]

I was able to get strongSwan to load the certs and the ecdsa key with some adjustments. I copied ipsesc.conf and ipsec.secrets to /etc/, but in ipsec.conf I changed the leftcert which was in the form of leftcert=xxx.xxx.xxx.xxx_user.crt, to be simply leftcert=user.crt (where "user" is the username of the algo account I'm connecting as).

In ipsec.secrets I made a similar adjustment.

I also changed rightsubnet to be something more specific than 0.0.0.0/0 because I haven't figured out why this seems to lock up all network traffic on the router. I changed it to a more specific private network (172.16.0.0/16) during testing.

After ipsec restart, I can run ipsec listcerts and see this in the output:

pubkey: ECDSA 256 bits, has private key

And ipsec up <connection-name> successfully establishes a connection.

I haven't been able to ping across the tunnel or otherwise do anything useful with it, but at least it seems to authenticate correctly.

/var/log/charon.log is a good place to look if you are having trouble with certs and keys.

[chriseldredge]

Adding this iptables rule enables me to ping/ssh to a private ip address on my algo ec2 instance:

iptables -t nat -I UBNT_VPN_IPSEC_SNAT_HOOK -m policy --pol ipsec --dir out -j ACCEPT

I'll try setting rightsubnet to use a default route later and see if it avoids locking up all network traffic. Feels like I'm getting closer.

[jackivanov]

@chriseldredge @ebcodes
To avoid locking all the network traffic you should configure the routing tables properly or you can modify strongswan.conf in order to use the zero routing table:

Something like this:

charon {
 routing_table = 0
}

[ebrandell-2]

@gunph1ld @chriseldredge
This might also be relevant: https://lists.strongswan.org/pipermail/users/2015-January/007289.html

What you want is a passthrough policy with source and destination being 192.168.1.0/24.
That policy with narrower subnets will take precedence before the policy that is defined by
your "toclient" connection definition.

conn lanbypass
    leftsubnet=192.168.1.0/24
    rightsubnet=192.168.1.0/24
    type=passthrough
    auto=route

[ebrandell-2]

In my latest attempt, I tried setting this up using the EdgeOS (VyOS) CLI. I'm not touching it for tonight as it's not working (not tunneling traffic). If anyone needs more info I'm happy to provide it.

I added the suggestion from @gunph1ld from above for the /etc/strongswan.conf to allow local access.

The routing structure was a bit different this time: I didn't want to piss my roommate off fucking around with this thing all night (already spent 6 hours on it...).

While I like the idea of IPSEC and it's security, I'm about ready to move on from it if it's going to be this hard to set it up.

Anyway...

Structure:

Internet <- Modem <- Asus RT-AC68U <- ERLite
                      	                 |
                                         V
                                    Algo + DHCP Lan Client (just my macbook right now)

Here are my current configs. This was attempted using 1.9.7alpha1 of the EdgeMax router firmware for ER-Lite.

/config/config.boot:

vpn {
    ipsec {
        auto-firewall-nat-exclude enable
        esp-group ALGO {
            compression enable
            lifetime 3600
            mode tunnel
            pfs dh-group19
            proposal 1 {
                encryption aes128gcm128
                hash sha256
            }
        }
        ike-group ALGO {
            dead-peer-detection {
                action clear
                interval 35
                timeout 120
            }
            ikev2-reauth no
            key-exchange ikev2
            lifetime 28800
            proposal 1 {
                dh-group 19
                encryption aes128gcm128
                hash sha256
            }
        }
        include-ipsec-secrets /config/user-data/ipsec.secrets
        logging {
            log-level 2
            log-modes net
        }
        site-to-site {
            peer <ALGO_IP> {
                authentication {
                    id <ALGO_IP>
                    mode x509
                    x509 {
                        ca-cert-file /config/auth/algo/cacert.pem
                        cert-file /config/auth/algo/<ALGO_IP>_<USER>.crt
                        key {
                            file /config/auth/algo/<ALGO_IP>_<USER>.key
                        }
                    }
                }
                connection-type initiate
                default-esp-group ALGO
                description Algo
                dhcp-interface eth0
                ike-group ALGO
                ikev2-reauth inherit
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    protocol all
                }
            }
        }
    }
}

/etc/ipsec.conf (slightly modified to fix some issues from above):

# generated by /opt/vyatta/sbin/vpn-config.pl
config setup

conn %default
        keyexchange=ikev1

conn peer-<ALGO_IP>-tunnel-1
        # dhcp-interface=eth0
        leftsourceip=%config
        left=<LOCAL_ETH0_WAN>
        leftid="<ALGO_IP>"
        right=<ALGO_IP>
        leftprotoport=%any
        rightprotoport=%any
        ike=aes128gcm128-sha256-ecp256!
        keyexchange=ikev2
        # reauth=no
        # ikelifetime=28800s
        dpddelay=35s
        dpdtimeout=120s
        dpdaction=clear
        esp=aes128gcm128-sha256-ecp256!
        # keylife=3600s
        # rekeymargin=540s
        type=tunnel
        compress=yes
        #authby=rsasig
        #leftrsasigkey=%cert
        #rightrsasigkey=%cert
        rightca=%same
        leftauth=pubkey
        leftcert=/etc/ipsec.d/certs/<ALGO_IP>_<USER>.crt
        auto=route
        # keyingtries=%forever

        # Custom
        rekey=no
        fragmentation=yes

Directory structure for certs/keys:

root@ubnt:~# tree -pug /etc/ipsec.d/
/etc/ipsec.d/
|-- [drwxr-xr-x root     root    ]  aacerts
|-- [drwxr-xr-x root     root    ]  acerts
|-- [drwxr-xr-x root     root    ]  cacerts
|   `-- [-rw------- root     root    ]  cacert.pem
|-- [drwxr-xr-x root     root    ]  certs
|   |-- [-rw------- root     root    ]  01.pem
|   |-- [-rw------- root     root    ]  02.pem
|   |-- [-rw------- root     root    ]  <ALGO_IP>.crt
|   `-- [-rw------- root     root    ]  <ALGO_IP>_<USER>.crt
|-- [drwxr-xr-x root     root    ]  crls
|-- [drwxr-xr-x root     root    ]  ecparams
|   `-- [-rw-r--r-- root     root    ]  prime256v1.pem
|-- [drwxr-xr-x root     root    ]  ocspcerts
|-- [drwxr-x--- root     root    ]  private
|   |-- [-rw------- root     root    ]  <ALGO_IP>.key
|   |-- [-rw------- root     root    ]  <ALGO_IP>_<USER>.key
|   |-- [-rw------- root     root    ]  <USER>.p12
|   `-- [-rw------- root     root    ]  cakey.pem
|-- [drwxr-xr-x root     root    ]  reqs
`-- [drwxr-xr-x root     root    ]  tunnels
    `-- [-rw-r--r-- root     root    ]  remote-access

ipsec up <PEER>:

root@ubnt:~# ipsec up peer-<ALGO_IP>-tunnel-1
initiating IKE_SA peer-<ALGO_IP>-tunnel-1[1] to <ALGO_IP>
generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ]
sending packet: from <LOCAL_ETH0_WAN>[500] to <ALGO_IP>[500] (248 bytes)
received packet: from <ALGO_IP>[500] to <LOCAL_ETH0_WAN>[500] (273 bytes)
parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ]
local host is behind NAT, sending keep alives
received cert request for "CN=<ALGO_IP>"
sending cert request for "CN=<ALGO_IP>"
authentication of 'CN=<USER>' (myself) with ECDSA-256 signature successful
sending end entity cert "CN=<USER>"
establishing CHILD_SA peer-<ALGO_IP>-tunnel-1
generating IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ IDr AUTH CPRQ(ADDR DNS) N(IPCOMP_SUP) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(MULT_AUTH) N(EAP_ONLY) ]
splitting IKE message with length of 856 bytes into 2 fragments
generating IKE_AUTH request 1 [ EF ]
generating IKE_AUTH request 1 [ EF ]
sending packet: from <LOCAL_ETH0_WAN>[4500] to <ALGO_IP>[4500] (544 bytes)
sending packet: from <LOCAL_ETH0_WAN>[4500] to <ALGO_IP>[4500] (377 bytes)
received packet: from <ALGO_IP>[4500] to <LOCAL_ETH0_WAN>[4500] (544 bytes)
parsed IKE_AUTH response 1 [ EF ]
received fragment #1 of 2, waiting for complete IKE message
received packet: from <ALGO_IP>[4500] to <LOCAL_ETH0_WAN>[4500] (343 bytes)
parsed IKE_AUTH response 1 [ EF ]
received fragment #2 of 2, reassembling fragmented IKE message
parsed IKE_AUTH response 1 [ IDr CERT AUTH CPRP(ADDR DNS DNS) N(IPCOMP_SUP) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_6_ADDR) ]
received end entity cert "CN=<ALGO_IP>"
  using certificate "CN=<ALGO_IP>"
  using trusted ca certificate "CN=<ALGO_IP>"
checking certificate status of "CN=<ALGO_IP>"
certificate status is not available
  reached self-signed root ca with a path length of 0
authentication of '<ALGO_IP>' with ECDSA-256 signature successful
IKE_SA peer-<ALGO_IP>-tunnel-1[1] established between <LOCAL_ETH0_WAN>[CN=<USER>]...<ALGO_IP>[<ALGO_IP>]
installing DNS server 8.8.8.8 to /etc/resolv.conf
installing DNS server 8.8.4.4 to /etc/resolv.conf
installing new virtual IP 10.19.48.1
CHILD_SA peer-<ALGO_IP>-tunnel-1{1} established with SPIs c7ebb694_i cb024a8c_o and TS 10.19.48.1/32 === <ALGO_IP>/32
connection 'peer-<ALGO_IP>-tunnel-1' established successfully

ipsec statusall:

root@ubnt:~# ipsec statusall
Status of IKE charon daemon (strongSwan 5.2.2, Linux 3.10.20-UBNT, mips64):
  uptime: 8 minutes, since Apr 08 07:35:44 2017
  malloc: sbrk 410768, mmap 0, used 280864, free 129904
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 2
  loaded plugins: charon ldap sqlite pkcs11 aes des sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs8 pem openssl agent xcbc cmac ctr ccm gcm curl attr kernel-netlink resolve socket-default stroke vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap addrblock
Listening IP addresses:
  <LOCAL_ETH0_WAN>
  10.1.1.1
  10.2.1.1
Connections:
peer-<ALGO_IP>-tunnel-1:  <LOCAL_ETH0_WAN>...<ALGO_IP>  IKEv2, dpddelay=35s
peer-<ALGO_IP>-tunnel-1:   local:  [CN=<USER>] uses public key authentication
peer-<ALGO_IP>-tunnel-1:    cert:  "CN=<USER>"
peer-<ALGO_IP>-tunnel-1:   remote: [<ALGO_IP>] uses public key authentication
peer-<ALGO_IP>-tunnel-1:   child:  dynamic === dynamic TUNNEL, dpdaction=clear
Routed Connections:
peer-<ALGO_IP>-tunnel-1{1}:  ROUTED, TUNNEL
peer-<ALGO_IP>-tunnel-1{1}:   <LOCAL_ETH0_WAN>/32 === <ALGO_IP>/32
Security Associations (1 up, 0 connecting):
peer-<ALGO_IP>-tunnel-1[1]: ESTABLISHED 8 minutes ago, <LOCAL_ETH0_WAN>[CN=<USER>]...<ALGO_IP>[<ALGO_IP>]
peer-<ALGO_IP>-tunnel-1[1]: IKEv2 SPIs: ef096795d0f5bb2f_i* ea779c10e3839537_r, rekeying disabled
peer-<ALGO_IP>-tunnel-1[1]: IKE proposal: AES_GCM_16_128/PRF_HMAC_SHA2_256/ECP_256
peer-<ALGO_IP>-tunnel-1{1}:  INSTALLED, TUNNEL, ESP in UDP SPIs: c7ebb694_i cb024a8c_o, IPCOMP CPIs: 794a_i bf28_o
peer-<ALGO_IP>-tunnel-1{1}:  AES_GCM_16_128, 0 bytes_i, 0 bytes_o, rekeying disabled
peer-<ALGO_IP>-tunnel-1{1}:   10.19.48.1/32 === <ALGO_IP>/32

show vpn ipsec sa:

root@ubnt:~# show vpn ipsec sa
peer-<ALGO_IP>-tunnel-1: #1, ESTABLISHED, IKEv2, ef096795d0f5bb2f:ea779c10e3839537
  local  'CN=<USER>' @ <LOCAL_ETH0_WAN>
  remote '<ALGO_IP>' @ <ALGO_IP>
  AES_GCM_16-128/PRF_HMAC_SHA2_256/ECP_256
  established 649s ago
  peer-<ALGO_IP>-tunnel-1: #1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-128
    installed 649 ago
    in  c7ebb694/794a,      0 bytes,     0 packets
    out cb024a8c/bf28,      0 bytes,     0 packets
    local  10.19.48.1/32
    remote <ALGO_IP>/32

Screenshots:

[chriseldredge]

@gunph1ld the suggestion to set routing_table to 0 seems to have helped:

charon {
 routing_table = 0
}

When I start the VPN connection my EdgeRouter will route traffic originating from the local server through the tunnel successfully. In effect this means that DNS is being routed through VPN.

However, traffic from my LAN is being routed by default route through ISP and not getting passed through VPN tunnel.

GRE/ipsec configuration may be necessary to complete the configuration.

[kiratp]

(Edit - fixing some subnets and iptables rules to tighten things up)

I have it working end to end.

Router /config/config.boot

Algo Server ipsec.conf

config setup
    uniqueids = never # allow multiple connections per user
    charondebug="ike 2, knl 2, cfg 2, net 2, esp 2, dmn 2,  mgr 2"

conn %default
    fragmentation=yes
    rekey=no
    dpdaction=clear
    keyexchange=ikev2
    compress=yes
    dpddelay=35s

    ike=aes128gcm16-sha2_256-prfsha256-ecp256!
    esp=aes128gcm16-sha2_256-ecp256!

    left=%any
    leftauth=pubkey
    leftid=<ALGO_IP>
    leftcert=<ALGO_IP>.crt
    leftsendcert=always
    leftsubnet=0.0.0.0/0,::/0

    right=%any
    rightauth=pubkey
    rightsourceip=10.19.48.0/24,fd9d:bc11:4020::/48
    rightdns=8.8.8.8,8.8.4.4
    rightsubnet=0.0.0.0/0,::/0
conn ikev2-pubkey
    auto=add

Serverside iptables changes

sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -m policy --pol none --dir out -j MASQUERADE
sudo iptables -A FORWARD -s 10.0.0.0/16 -d 10.0.0.0/16 -j DROP
sudo iptables -A FORWARD -m conntrack --ctstate NEW -s 10.0.0.0/16  -m policy --pol ipsec --dir in -j ACCEPT

    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name VLAN103_IN {
        default-action accept
        description "Block HA to LAN"
        rule 1 {
            action drop
            description "Drop traffic to all local interfaces"
            destination {
                address 10.0.0.0/8
                group {
                }
            }
            log disable
            protocol all
            state {
                established enable
                invalid enable
                new enable
                related enable
            }
        }
    }
    name VLAN103_LOCAL {
        default-action drop
        description "Block HA to Router config"
        rule 1 {
            action accept
            description "Allow HA access to DNS"
            destination {
                port 53
            }
            log disable
            protocol tcp_udp
        }
    }
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 30 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            description IKE
            destination {
                port 500
            }
            log disable
            protocol udp
        }
        rule 30 {
            action accept
            description IKE
            destination {
                port 4500
            }
            log disable
            protocol udp
        }
        rule 40 {
            action accept
            description IPSEC/ESP
            log disable
            protocol esp
        }
        rule 50 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_OUT {
        default-action accept
        description ""
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address dhcp
        description Internet
        dhcpv6-pd {
            pd 0 {
                interface eth1 {
                }
                interface eth2 {
                }
                prefix-length 64
            }
            rapid-commit enable
        }
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
            out {
                name WAN_OUT
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 10.0.0.1/24
        description Local
        duplex auto
        ipv6 {
            dup-addr-detect-transmits 1
            router-advert {
                cur-hop-limit 64
                link-mtu 0
                managed-flag true
                max-interval 600
                other-config-flag false
                prefix ::/64 {
                    autonomous-flag true
                    on-link-flag true
                    valid-lifetime 2592000
                }
                reachable-time 0
                retrans-timer 0
                send-advert true
            }
        }
        speed auto
        vif 103 {
            address 10.0.3.1/24
            description HomeAutomation
            firewall {
                in {
                    name VLAN103_IN
                }
                local {
                    name VLAN103_LOCAL
                }
            }
        }
    }
    ethernet eth2 {
        address 10.0.1.1/24
        description "Local 2"
        duplex auto
        speed auto
    }
    loopback lo {
    }
}
port-forward {

}
protocols {
    static {
    }
}
service {
   nat {
        rule 5003 {
            description "masquerade for WAN"
            destination {
                address <ALGO_IP>
            }
            log disable
            outbound-interface eth0
            protocol all
            type masquerade
        }
    }
}
system {

}
traffic-control {
    smart-queue Upload {
        upload {
            ecn enable
            rate 13bit
        }
        wan-interface eth0
    }
}
vpn {
    ipsec {
        auto-update 3600
        auto-firewall-nat-exclude enable
        esp-group ALGO {
            compression enable
            lifetime 3600
            mode tunnel
            pfs dh-group19
            proposal 1 {
                encryption aes128gcm128
                hash sha256
            }
        }
        ike-group ALGO {
            dead-peer-detection {
                action clear
                interval 35
                timeout 120
            }
            ikev2-reauth no
            key-exchange ikev2
            lifetime 28800
            proposal 1 {
                dh-group 19
                encryption aes128gcm128
                hash sha256
            }
        }
        include-ipsec-conf /config/user-data/ipsec_home.conf
        include-ipsec-secrets /config/user-data/ipsec_home.secrets
        logging {
            log-level 2
            log-modes net
        }
    }
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:system@4:ubnt-pptp@1:ubnt-util@1:vrrp@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v1.9.1.4939093.161214.0705 */

Router home_ipsec.conf (referenced in config.boot)

conn algo
    fragmentation=yes
    rekey=no
    dpdaction=clear
    keyexchange=ikev2
    compress=no
    dpddelay=35s

    ike=aes128gcm16-sha2_256-prfsha256-ecp256
    esp=aes128gcm16-sha2_256-ecp256

    right=<ALGO_IP>
    rightid=<ALGO_IP>
    rightsubnet=0.0.0.0/0
    rightauth=pubkey

    leftsourceip=%config
    leftauth=pubkey
    leftcert=<ALGO_IP>_home.crt
    leftfirewall=yes
    left=%defaultroute
    leftsubnet=10.0.0.0/16

    auto=add

conn lanbypass
    leftsubnet=10.0.0.0/16
    rightsubnet=10.0.0.0/16
    type=passthrough
    auto=route

[ebrandell-2]

@kiratp and I spent a long time tonight setting this up and I managed to get a tunnel working on my ERL! There's still plenty to do, however, to get this working fully:

• IPV6 tunneling support
• Performance improvements with IPSEC offloading or modification of encryption/hashing mechanisms
• Ansible updates to possibly generate a config during deploy

kiratp and I will be playing around with this again next weekend, but for now, at least something works!

Hats off to @kiratp for being so patient and helpful tonight. Thank you!

[mister2d]

@ebcodes @kiratp Let me know if you need help additional help testing. I'd like to give it a shot on my hardware as well. Thanks.

[kiratp]

@mister2d - More the merrier! Just jump on the Algo support Slack. The next issue I am working through as a P1 is Hardware Offload. Something about the tunnel is breaking offload (capped at 9 Mb/s vs 30Mb/s sw-only)

[kiratp]

#473 - PR to make this easy

[ghost]

@kiratp Thanks for getting a fix in for this! I'm using your PR but not having much luck. The tunnel comes all the way up and I have DNS/routing from the EdgeRouter itself via my AlgoVPN endpoint. I also have access to local LAN resources on my 192.168.1.0/24 network. However, I cannot ping my EdgeRouter .1 from my LAN or route any traffic through the VPN from any device in 192.168.1.0/24 other than the EdgeRouter itself at .1. Below is most of my config. Any ideas? This seems rather strange. Almost as if the passthrough mode is working for everything but the .1 address.

EDGE ROUTER
IPSec Conf

conn ikev2-<ALGOVPNIP>
    fragmentation=yes
    rekey=no
    dpdaction=clear
    keyexchange=ikev2
    compress=no
    dpddelay=35s

    ike=aes128gcm16-prfsha512-ecp256!
    esp=aes128gcm16-ecp256!

    right=<ALGOVPNIP>
    rightid=<ALGOVPNIP>
    rightsubnet=0.0.0.0/0
    rightauth=pubkey

    leftsourceip=%config
    leftauth=pubkey
    leftcert=rtr.crt
    leftfirewall=yes
    left=%defaultroute
    leftsubnet=192.168.1.0/24
    auto=route
        
        
conn lanbypass
    leftsubnet=192.168.1.0/24
    rightsubnet=192.168.1.0/24
    type=passthrough
    auto=route

Edge Router XFRM Policy

src 0.0.0.0/0 dst 192.168.1.0/24 
        dir fwd priority 2979 
        tmpl src <ALGOVPNIP> dst <ISP WAN Address>
                proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 192.168.1.0/24 
        dir in priority 2979 
        tmpl src <ALGOVPNIP> dst <ISP WAN Address>
                proto esp reqid 1 mode tunnel
src 192.168.1.0/24 dst 0.0.0.0/0 
        dir out priority 2979 
        tmpl src <ISP WAN Address> dst <ALGOVPNIP>
                proto esp reqid 1 mode tunnel
src 192.168.1.0/24 dst 192.168.1.0/24 
        dir fwd priority 1347 
src 192.168.1.0/24 dst 192.168.1.0/24 
        dir in priority 1347 
src 192.168.1.0/24 dst 192.168.1.0/24 
        dir out priority 1347 
src 0.0.0.0/0 dst 0.0.0.0/0 
        socket in priority 0 
src 0.0.0.0/0 dst 0.0.0.0/0 
        socket out priority 0 
src 0.0.0.0/0 dst 0.0.0.0/0 
        socket in priority 0 
src 0.0.0.0/0 dst 0.0.0.0/0 
        socket out priority 0 
src ::/0 dst ::/0 
        socket in priority 0 
src ::/0 dst ::/0 
        socket out priority 0 
src ::/0 dst ::/0 
        socket in priority 0 
src ::/0 dst ::/0 
        socket out priority 0 

EDGEROUTER ROUTES

default via <ISP DEFAULT> dev eth0  proto zebra 
10.19.48.1 dev eth0  proto kernel  scope link 
<ISPSUBNET> dev eth0  proto kernel  scope link  src <ISPMODEMIP>
<ALGOVPNIP> via <ISP DEFAULT> dev eth0  proto zebra 
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.1 
192.168.2.0/24 dev switch0  proto kernel  scope link  src 192.168.2.1 

ALGOVPN ENDPOINT

IPSec Conf

config setup
    uniqueids=never # allow multiple connections per user
    charondebug="ike 2, knl 2, cfg 2, net 2, esp 2, dmn 2,  mgr 2"

conn %default
    fragmentation=yes
    rekey=no
    dpdaction=clear
    keyexchange=ikev2
    compress=yes
    dpddelay=35s

    ike=aes128gcm16-prfsha512-ecp256!
    esp=aes128gcm16-ecp256!

    left=%any
    leftauth=pubkey
    leftid=<ALGOVPNIP>
    leftcert=<ALGOVPNIP>.crt
    leftsendcert=always
    leftsubnet=0.0.0.0/0,::/0

    right=%any
    rightauth=pubkey
    rightsourceip=10.19.48.0/24,fd9d:bc11:4020::/48
    rightdns=172.16.0.1

    rightsubnet=0.0.0.0/0,::/0

conn ikev2-pubkey
    auto=add

ALGOVPN ROUTES

default via <HOSTINGPROVIDERDEFAULT> dev eth0 onlink 
10.10.0.0/16 dev eth0  proto kernel  scope link  src 10.10.0.5 
<HOSTINGPROVIDERRANGE> dev eth0  proto kernel  scope link  src <ALGOVPNIP>
192.168.1.0/24 via <HOSTNINGPROVIDERDEFAULT> dev eth0  proto static 

ALGOVPN XFRM Policy

src 192.168.1.0/24 dst 0.0.0.0/0 
    dir fwd priority 193856 
    tmpl src <ISP WAN Address> dst <ALGOVPNIP>
        proto esp reqid 1 mode tunnel
src 192.168.1.0/24 dst 0.0.0.0/0 
    dir in priority 193856 
    tmpl src <ISP WAN Address> dst <ALGOVPNIP>
        proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 192.168.1.0/24 
    dir out priority 193856 
    tmpl src <ALGOVPNIP> dst <ISP WAN Address>
        proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0 
    socket in priority 0 
src 0.0.0.0/0 dst 0.0.0.0/0 
    socket out priority 0 
src 0.0.0.0/0 dst 0.0.0.0/0 
    socket in priority 0 
src 0.0.0.0/0 dst 0.0.0.0/0 
    socket out priority 0 
src ::/0 dst ::/0 
    socket in priority 0 
src ::/0 dst ::/0 
    socket out priority 0 
src ::/0 dst ::/0 
    socket in priority 0 
src ::/0 dst ::/0 
    socket out priority 0

[dguido]

Closing this issue as solution referenced in #473 is good enough.

[dmwyatt]

Anyone who tried out the PR from @kiratp have any objections to paying out the bounty?

[ndfred]

@dmwyatt: do we have a good idea of how to get HW acceleration working so an ERL can at least sustain 100 MBit/s? #473 also mentions routing breaking when multiple clients connect to the VPN. Until these issues are addressed I wouldn't use my ERL with Algo VPN.

Your initial ask didn't mention any reliability or performance requirements so I think it is reasonable to pay out the boundy. How is it working out for you? Do you have it enabled and are you happy with it?

[kiratp]

@ndfred - there is a thread on the UBNt forums about the offload issues - https://community.ubnt.com/t5/EdgeMAX/IPSec-performance-issue/m-p/1946992#M162999

I've been helping folks with routing on Slack - happy to help there. The issues I've seen seem to seem to stem from manually configured routing rules (xfrm), overlapping subsets etc.

I will refactor the PR once the new modular client support is released.

Thanks for releasing the bounty folks!