Setting up a virtual router at home

This post is a bit of a mind-dump, but I thought I'd write down things I learned this past weekend while switching my home router from a physical NetGear R6260 to a virtual pfSense instance. I've also been recommended to use OPNSense by my colleagues for a nicer interface, but I'll stick with pfSense for the moment.

The reason for making the switch includes the fact that VLAN configuration on the R6260 requires a reboot on every modification, and moreover, you can't have more than one VLAN on each physical port.
 I'm not exactly sure why that would be the case either.


To start, I purchased an i340-T4 card on eBay for ~$25. Brand new, these cards are still running at $100 or more, but mine works just fine for the price. (The only thing I wasn't happy about was that it arrived sitting in bubble wrap instead of an ESD bag.)

After shutting down my host and installing the card, I didn't see the interfaces come up automatically (as expected). Instead, disabled netplan, as the OS is Ubuntu 18, and queried dmesg for the name of the interfaces.
(base) root@b350-gaming-pc:~# lspci | grep net                                                                   
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
09:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)                         
09:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
09:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
09:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)                      
(base) root@b350-gaming-pc:~# dmesg | grep -P "(05\:00\.0|09\:00\.0|09\:00\.1|09\:00\.2|09\:00\.3)"              
[    0.152895] pci 0000:05:00.0: [10ec:8168] type 00 class 0x020000
[    0.152934] pci 0000:05:00.0: reg 0x10: [io  0xf000-0xf0ff]
[    0.152969] pci 0000:05:00.0: reg 0x18: [mem 0xfd700000-0xfd700fff 64bit]
[    0.152991] pci 0000:05:00.0: reg 0x20: [mem 0xf0300000-0xf0303fff 64bit pref]
[    0.153119] pci 0000:05:00.0: supports D1 D2    
[    0.153120] pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.176101] pci 0000:09:00.0: [8086:150e] type 00 class 0x020000
[    0.176136] pci 0000:09:00.0: reg 0x10: [mem 0xfd500000-0xfd57ffff]
[    0.176172] pci 0000:09:00.0: reg 0x1c: [mem 0xfd58c000-0xfd58ffff]
[    0.176207] pci 0000:09:00.0: reg 0x30: [mem 0xfd480000-0xfd4fffff pref]
[    0.176303] pci 0000:09:00.0: PME# supported from D0 D3hot D3cold
[    0.176396] pci 0000:09:00.1: [8086:150e] type 00 class 0x020000
[    0.176429] pci 0000:09:00.1: reg 0x10: [mem 0xfd400000-0xfd47ffff]  
[    0.176464] pci 0000:09:00.1: reg 0x1c: [mem 0xfd588000-0xfd58bfff]  
[    0.176586] pci 0000:09:00.1: PME# supported from D0 D3hot D3cold                                             
[    0.176661] pci 0000:09:00.2: [8086:150e] type 00 class 0x020000                                              
[    0.176693] pci 0000:09:00.2: reg 0x10: [mem 0xfd380000-0xfd3fffff]  
[    0.176727] pci 0000:09:00.2: reg 0x1c: [mem 0xfd584000-0xfd587fff]                                           
[    0.176847] pci 0000:09:00.2: PME# supported from D0 D3hot D3cold                                             
[    0.176921] pci 0000:09:00.3: [8086:150e] type 00 class 0x020000     
[    0.176952] pci 0000:09:00.3: reg 0x10: [mem 0xfd300000-0xfd37ffff]
[    0.176987] pci 0000:09:00.3: reg 0x1c: [mem 0xfd580000-0xfd583fff]
[    0.177106] pci 0000:09:00.3: PME# supported from D0 D3hot D3cold
[    0.616401] iommu: Adding device 0000:05:00.0 to group 0
[    0.616429] iommu: Adding device 0000:09:00.0 to group 0
[    0.616441] iommu: Adding device 0000:09:00.1 to group 0
[    0.616453] iommu: Adding device 0000:09:00.2 to group 0
[    0.616466] iommu: Adding device 0000:09:00.3 to group 0
[    1.344957] r8169 0000:05:00.0 eth0: RTL8168g/8111g at 0x00000000db0017c6, 1c:1b:0d:e6:c1:27, XID 0c000800 IRQ 65
[    1.344959] r8169 0000:05:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[    1.349878] r8169 0000:05:00.0 enp5s0: renamed from eth0
[    1.796052] igb 0000:09:00.0: added PHC on eth0
[    1.796053] igb 0000:09:00.0: Intel(R) Gigabit Ethernet Network Connection
[    1.796054] igb 0000:09:00.0: eth0: (PCIe:5.0Gb/s:Width x1) 90:e2:ba:35:f9:70
[    1.796057] igb 0000:09:00.0: eth0: PBA No: Unknown
[    1.796058] igb 0000:09:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    2.291198] igb 0000:09:00.1: added PHC on eth1
[    2.291200] igb 0000:09:00.1: Intel(R) Gigabit Ethernet Network Connection
[    2.291201] igb 0000:09:00.1: eth1: (PCIe:5.0Gb/s:Width x1) 90:e2:ba:35:f9:71
[    2.291204] igb 0000:09:00.1: eth1: PBA No: Unknown
[    2.291205] igb 0000:09:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    2.753645] igb 0000:09:00.2: added PHC on eth2
[    2.753647] igb 0000:09:00.2: Intel(R) Gigabit Ethernet Network Connection
[    2.753648] igb 0000:09:00.2: eth2: (PCIe:5.0Gb/s:Width x1) 90:e2:ba:35:f9:72
[    2.753651] igb 0000:09:00.2: eth2: PBA No: Unknown
[    2.753652] igb 0000:09:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    3.246171] igb 0000:09:00.3: added PHC on eth3
[    3.246172] igb 0000:09:00.3: Intel(R) Gigabit Ethernet Network Connection
[    3.246174] igb 0000:09:00.3: eth3: (PCIe:5.0Gb/s:Width x1) 90:e2:ba:35:f9:73
[    3.246177] igb 0000:09:00.3: eth3: PBA No: Unknown
[    3.246178] igb 0000:09:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[    3.247687] igb 0000:09:00.3 enp9s0f3: renamed from eth3
[    3.260482] igb 0000:09:00.2 enp9s0f2: renamed from eth2
[    3.320310] igb 0000:09:00.1 enp9s0f1: renamed from eth1
[    3.448087] igb 0000:09:00.0 enp9s0f0: renamed from eth0
From here it's trivial to set up an interfaces file for ifupdown.
(base) root@b350-gaming-pc:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp9s0f0
iface enp9s0f0 inet manual

auto enp9s0f1
iface enp9s0f1 inet manual

auto enp9s0f2
iface enp9s0f2 inet manual

auto enp9s0f3
iface enp9s0f3 inet manual

auto vmbr0
iface vmbr0 inet static
    bridge_ports enp5s0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0
And on a networking service restart, I had four additional NICs on my machine.


Now for the fun part -- I spun up a pfSense instance on my primary host and workstation in libvirt / qemu-kvm. I tend to use virt-install on my secondary host, but this time around I just used virt-manager.

Because it's just a home setup, I didn't have to give it many resources; for instance, a hyperthread on the Ryzen 2700, and 512 MiB of RAM. Even during high data transfers, such as backups, I'll see the utilization sit around 50%.

After the installation, I assigned it three physical NICs, which showed up as macvtap interfaces under ifconfig.
(base) root@b350-gaming-pc:~# for iface in macvtap{0..2}; do ip link show "$iface"; done
14: macvtap0@enp9s0f0:  mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500
    link/ether 52:54:00:21:9a:0f brd ff:ff:ff:ff:ff:ff
15: macvtap1@enp9s0f1:  mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500
    link/ether 52:54:00:67:52:6f brd ff:ff:ff:ff:ff:ff
16: macvtap2@enp9s0f2:  mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500
    link/ether 52:54:00:e6:f6:b1 brd ff:ff:ff:ff:ff:ff
I think the trickiest part, here, was switching the physical cables on the card to see which ports / names had links and which didn't.
(base) root@b350-gaming-pc:~# for iface in enp9s0f{0..3}; do printf "%s\\n" "$iface"; ethtool "$iface" | grep detected; done
        Link detected: yes
        Link detected: yes
        Link detected: yes
        Link detected: no
From here, in virt-manager, I added the interfaces, which created the following entries in the virsh template.
    <interface type='direct'>
      <mac address='52:54:00:21:9a:0f'/>
      <source dev='enp9s0f0' mode='passthrough'/>
      <target dev='macvtap0'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    <interface type='direct'>
      <mac address='52:54:00:67:52:6f'/>
      <source dev='enp9s0f1' mode='passthrough'/>
      <target dev='macvtap1'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <interface type='direct'>
      <mac address='52:54:00:e6:f6:b1'/>
      <source dev='enp9s0f2' mode='passthrough'/>
      <target dev='macvtap2'/>
      <model type='e1000'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>

External Configuration Backups

In pfSense, you also have the ability to back up your configurations under Diagnostics > Backup & Restore, and I've written the following script to log in, download the configuration file and snapshot it in the same ZFS pool (but you can always choose another).
(base) root@b350-gaming-pc:~/cronscripts# cat 
#! /bin/bash
# Download and save configuration files from pfSense and snapshot with ZFS.

IPs=( "" "" )

. /home/brandon/.profile

    if [ -n "$(ls -A "$PATH_")" ]
        rm "$PATH_/"*

    for IP in "${IPs[@]}"
        # Get token.
        wget -T 1 -t 1 -qO- --keep-session-cookies --save-cookies \
            "$PATH_/cookies.txt" --no-check-certificate "https://$IP/diag_backup.php" \
            | grep "name='__csrf_magic'" | sed 's/.*value="\(.*\)".*/\1/' > "$PATH_/csrf.txt"

        # Log in.
        wget -T 1 -t 1 -qO- --keep-session-cookies --load-cookies "$PATH_/cookies.txt" \
            --save-cookies "$PATH_/cookies.txt" --no-check-certificate \
            --post-data "login=Login&usernamefld=$login&passwordfld=$password&__csrf_magic=$(cat "$PATH_/csrf.txt")" \
            "https://$IP/diag_backup.php"  | grep "name='__csrf_magic'" \
            | sed 's/.*value="\(.*\)".*/\1/' > "$PATH_/csrf2.txt"

        # Download XML config file.
        wget -T 5 -t 1 --keep-session-cookies --load-cookies "$PATH_/cookies.txt" \
            --no-check-certificate \
            --post-data "download=download&__csrf_magic=$(head -n 1 "$PATH_/csrf2.txt")" \
            "https://$IP/diag_backup.php" -O "$PATH_/config-$IP-$(date +%s).xml" 2>/dev/null

    # Clean up.
    if [ -n "$(ls -A "$PATH_/"*.txt)" ]
        rm "$PATH_/"*.txt

    # Snapshot the mount point with only the *.xml files.
    zfs snapshot "$dataset@$(date +%s)"

    # Now clean up the rest.
    if [ -n "$(ls -A "$PATH_")" ]
        rm "$PATH_/"*


unset login password PATH_ IPs dataset
unset -f main
(base) root@b350-gaming-pc:~/cronscripts# zfs list -t snapshot -ro name,creation,written,used homePool/home/VMs/pfSenseBackups
NAME                                         CREATION               WRITTEN   USED
homePool/home/VMs/pfSenseBackups@1560209738  Mon Jun 10 19:35 2019    1.90M  1.86M
homePool/home/VMs/pfSenseBackups@1560209823  Mon Jun 10 19:37 2019    1.86M  1.86M
homePool/home/VMs/pfSenseBackups@1560210886  Mon Jun 10 19:54 2019    1.87M  1.86M
(base) root@b350-gaming-pc:~/cronscripts# ll /homePool/home/VMs/pfSenseBackups/.zfs/snapshot/1560210886/
total 1838
drwxr-xr-x 2 root root       4 Jun 10 19:54 ./
drwxrwxrwx 2 root root       2 Jun 10 19:54 ../
-rw-r--r-- 1 root root 1993397 Jun 10 19:54 config-
-rw-r--r-- 1 root root       0 Jun 10 19:54 config-
These wget commands are in Netgate's documentation as well, so you can create your own scripts.

It may also be my unfamiliarity with how the web and login forms work, but I did have to create firewall rules (important!) to allow access on port 443 inbound to the LAN interface and switch to HTTPS for these commands to work properly. (In other words, switching to HTTP did not make for good results and I wasted a lot of time trying to get it to work.)

Final thoughts

One of the nice things about pfSense is that it has a configurable firewall on the interfaces. Because I'm trying to study networking and get better at it, this gives me a lab in which to try things (and it's free to use!).

There are a few things that I think are in odd spots in the menu, including Backup & Restore -- I would have expected this, as a new user, to be under, say, System. On the other hand, the location of Auto Config Backup under Services, however, makes more sense.

Now I just need to decide what to do with the spare NIC haha. I'll probably use it for direct communication / NFS between my two hosts by setting up a different subnet, so if I ever migrate VMs, they'll have a dedicated Gb line over which to travel.