How-to virtualize opnsense on proxmox
Busy re-doing my firewall setup & taking notes. Thought I'd turn that into a mini-tutorial, noting that pieces may be a little rest of the fkin owl. If you spot a gap...ask and I'll fill it in.
Why
The gear I've got for firewall is stupidly overkill (optane drive & 16GB RAM) for something that is 99% idle so I want to stick a couple other things on the same hardware via proxmox.
Assumed knowledge and requirements
Proxmox, linux cli, basic networking, (very basic) firewall knowledge. Nothing here is super hardcore technically as such...just unforgiving in sequence & details if unfamiliar hence notes. i.e. when it doesn't work it's not obvious where you fkd up.
Bring backups & loads of time. Frankly...assume that you'll fu.ck this up if you haven't done this before. So this is a Saturday morning project not a Sunday night one.
Before you start set up a ventoy USB stick adding both proxmox and opnsense ISO on there (plus perhaps a PDF copy of this post). You may not have internet access while doing this so come prepared...
I'm using my own IP ranges etc in post - adjust as needed. Key pieces you need to know to follow my descriptions:
vmbr0 is LAN side
vmbr1 is WAN (internet) side. (Proxmox sticks VMs on 0 by default so you want LAN side being 0)
10.32.0.1 is the opnsense VM [LAN side], 10.31.0.5 is WAN side but that doesn't really matter key part is 31 makes it separate from our main 32 LAN
10.32.0.2 is the LAN side IP of proxmox
My device has 5 ports with enp2s0 being internet facing [aka vmbr1], rest of the NICs are bridge to vmbr0
The key networking file on proxmox is at /etc/network/interfaces
Assumes upstream ISP is Eth based and just DHCPs whatever is connected. If you've got something else on ISP side you're on your own (sorry)
Virtualized FW
There are two basic ways to do this. Either pass through the NICs to the FW VM, or do two sets of bridges with one being WAN (internet) side and one LAN side. Passthrough is tricky to get working on a basic linux/bios/hardware level but solves some security issues. I'm doing the bridge approach because my gear (oddly) doesn't seem to support pass through. This guide is bridge only...so abandon ship now if you're going passthrough.
Bridge has one significant security implication. Proxmox gets the external traffic and bridges it to the firewall VM. Grand. Except Proxmox is fond of serving the proxmox management interface on ALL interfaces. So it's serving the damn management GUI before anything gets to our FW. Meaning it is potentially exposed BEFORE our fancy opnsense security gets to it. That's a problem we're trying to get around with below section on messing with the Proxmox firewall...despite setting up a opnsense firewall. [In reality the WAN side has a pvt non-routable IP so should be fine even without rules but I don't 100% trust that]
Keep this in mind when doing say incoming wireguard...you're effectively double-firewalled...and would need to pass through wireguard on both.
Security
The bulk of this guide happens BEFORE anything is connected to the internet, mostly because above "I don't trust this" issue. Just the FW device and a laptop directly connected to it.
GO
Step 1: Install proxmox & ensure you can access the GUI
On the network section do:
IP 10.32.0.2
Gateway 10.32.0.1
Set DNS to 1.1.1.1
Select a NIC that is LAN side (enp7s0 for me) - this should be whatever your laptop is plug in to
The proxmox interface is on https://10.32.0.2:8006
Step 2: Set up a second bridge in /etc/network/interfaces with WAN
auto vmbr1
iface vmbr1 inet static
address 10.31.0.5/24
bridge-ports enp2s0
bridge-stp off
bridge-fd 0
iface enp2s0 inet manual
Note how it is set to a static unroutable IP despite us aiming for DHCP upstream ISP side. That's because we want the firewall VM to grab the DHCP from ISP, not the proxmox level interface.
Restart to make the 2nd bridge show in GUI.
Step 3: Set up a opnsense VM
Copy over ISO off USB stick like so:
mkdir /media/usbstick
fdisk -l [[[to get the /dev/ location for the stick]]]
mount /dev/sda1 /media/usbstick
cp /media/usbstick/OPNSense.sio /var/lib/vz/template/iso/
If the iso is in that location then proxmox GUI should pick it up.
Set up a VM that has 12GB, 5120mb RAM and 3 cores. Going lower on space & ram works but has consequences downstream on how zfs plays out so if you can do 12 & 5 or more.
Before starting VM add the second bridge (vmbr1). Ideally add the LAN side during VM creation so that the naming 0 & 1 line up consistently. Memorize the MACs on both (sorta...just which is which).
Step 4: Install opnsense
Where it asks you for a login, use installer as user and opnsense as password
Key point here is you need to line up what opnsense thinks is LAN side and WAN side...with the right MAC per above...which in turn maps to the right bridge...which in turn maps to a physical port.
Finish install, change root pass & unmount the ISO. Reboot.
Step 5: Initial config of opnsense
Log into console of opnsense (the proxmox built in one). Option 2 - set interface IP. Configure LAN side. This part is a little confusing. We DONT want it to use DHCP as in where it gets it's IP, but we want it to SERVE DHCP...so that other LAN devices can get an IP from it. So in this menu the first time it asks you about DHCP the answer is no, the second time the answer is yes. Set IP to 10.32.0.1 and 24 as subnet bit count. Enable DHCP server. Start clients at 10.32.0.51 and end at 10.32.0.254. That way we've got ~50 IPs for statics. Change GUI to http. Reset to defaults - yes.
...now try to access GUI on http://10.32.0.1
On the GUI you should have an initial setup. Set hostname, domain, and DNS. Go with a public one for now e.g. 1.1.1.1 Leave the WAN side as DHCP - no need to change anything here.
Step 6: Firewall
Go back to proxmox & navigate to datacenter on left then to firewall. This stuff cascades so the rules you set on datacenter level apply to the device and to VMs below.
The firewall rules use first match principle, starting with zero i.e. top of list. So in general we want a pattern like so
[0] - Allow SSH on LAN side
[1] - Disallow SSH on everything
So lan side incoming SSH hits rule 0 matches that and gets approved, while everything else fails rule 0 then goes to rule 1 and gets dropped.
Direction: In
Action: Accept
Interface: vmbr0 [[[[this is LAN side]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick
Direction: In
Action: Drop
Interface: [[[[leave empty]]]]
Protocol: TCP
Dest Port: 8006
Enable: Tick
Do same for 22 TCP and ICMP (protocol, no port). I'd suggest messing with ICMP to experiment and get a better grasp of what blocks when.
Triple check that you've got this right. Next step is enable firewall...and if you fkd this (8006 specifically) up then you're locked out and will need a physically screen & keyboard to sort it out.
You should also block these ports on vmbr1 (internet/wan) for good measure:
TCP 111, UDP 111, UDP 323, TCP 25, TCP 3128. No idea what they all do...but proxmox is listening on them & I don't want them open internet facing. I used command "lsof -i -P -n" to work out what ports are being listened to.
Step 7: Connect device to internet
If all went well then a device plugged into LAN side can now access the internet. First thing you want to do is head over to
https://www.yougetsignal.com/tools/open-ports/
And confirm they can't see your port 8006
Trouble shooting
1) You'll need to set a static IP in the right range on your laptop when connecting directly to a device. So when you can't reach the interfaces (proxmox or opnsense) checking that you're on the right net is helpful. Also windows is really shtty on this so sometimes you need to disable and reenable the network adapter to make it pick up things right. Try that before assuming something else is broken. You may also need to restart the laptop entirely...windows tends to get confused easily when rapidly switching networks.
2) Very likely that you'll f up DNS somewhere along the way...so when trying to work out whether a path through the firewall works try to use IPs like 1.1.1.1 with ping rather than google.com.
3) If genuinely stuck do ping device by device. i.e. my laptop is connected to a proxmox device...can i ping that by IP. Yes, ok can I ping opnsense by IP. Yes, can I ping 1.1.1.1. Yes. Can I ping google.com.
Devices
You should point all devices at the opnsense instance for both gateway and DNS [10.32.0.1 for me]. Never point a device at the pihole IP directly...that would work yes but it'll get you into trouble later in unexpected & hard to troubleshoot ways. You want the devices asking opnsense for DNS and opnsense in turn asking the pihole/adguardhome, and that asking 1.1.1.1 in turn. It's under Unbound in opnsense...section called Override...that's where the pihole goes. Also I suggest using Adguard home...it's superior to pihole.
YAY & Thoughts
Once it all sorta works, export the config in opnsense and save a copy of the proxmox /etc/network/interfaces - opnsense has a really nice "one config backup file everything is in it" config thing going...use it.
If anyone finds holes in this post please do tell...I'm running this live so if I'm wrong about something I'd like to find out from you rather than an evil hacker
Full /etc/network/interfaces for completeness
root@firewall:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface enp2s0 inet manual
iface enp3s0 inet manual
iface enp4s0 inet manual
iface enp5s0 inet manual
iface enp6s0 inet manual
iface enp7s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 10.32.0.2/24
gateway 10.32.0.1
bridge-ports enp7s0 enp6s0 enp5s0 enp4s0 enp3s0
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet static
address 10.31.0.5/24
bridge-ports enp2s0
bridge-stp off
bridge-fd 0
And firewall view:
Comments
It's a lot easier to just roll your own KVM than deal with this shit.
My pronouns are like/subscribe.
Should I expect the next LES Talk article to be from you on "How to roll your own KVM" ?
Please include how to get IPv6 working in the VMs as that is where I always get stuck.
I am currently traveling in mostly remote areas until sometime in April 2024. Consequently DM's sent to me will go unanswered during this time.
For staff assistance or support issues please use the helpdesk ticket system at https://support.lowendspirit.com/index.php?a=add
Hey, do you want to post it on hostinghowto.com (with you as Author)?
C1V Hosting: Low cost Italian Cloud & Data Center Solutions 🚀 | Contact us for special offers. | Our deals on Telegram
Thanks for this article, it will definitely add to my to do/test things.
That really depends on your distribution. As long as you've got it set up on the host you can either throw them your own/128 or who just make them a subsection of the /64. That's kind of iffy.
My pronouns are like/subscribe.
hmm... very stable for network thus far but spotted a problem. Proxmox host itself doesn't have internet connectivity. It's trying to hit 10.31.05/vmbr1 for gateway which ISP is rightly going to NOPE.
4th line from bottom of original post
Needs to be removed. And same under step 2. @FrankZ - could you fix original post pls. Can't edit it anymore
Consider it Creative Commons licensed. Yolo it as you please. I think you may get an SEO hit for posting it second though since LES will have been indexed first...maybe not sure....my SEO understanding is limited.
Maybe. I really like using proxmox because I can deploy LXC/VMs against it using terraform/ansible. So I try to standardize against that...configs I can git & IaC. My entire homelab is essentially ansible lego bricks that I just re-combine in whatever I need rn.
Plus in this particular instance dedicated FW appliance would make more sense but I just fn can't given...
Would kill my soul just running a fw off that
Yes, thank you for this. It will be fun to tinker with it.
VPS providers to check out:
I tried Opnsense but I've got a 10Gbps internet connection and couldn't get 10Gbps throughput no matter what I tweaked. It was maxing out at around 3.8Gbps even after enabling offloading, disabling Spectre and Meltdown mitigations, and tweaking a bunch of tweakables.
I tried OpenWrt instead, and could easily hit 10Gbps NAT throughput with less than 10% CPU usage. I'd definitely recommend it as an alternative - it's not just for routers as they have an x86 build these days. I have an Intel X540-T2 which is a very old 10Gbps Intel NIC with two ports. I had no issue passing it through to the VM using Proxmox.
Daniel15 | https://d.sb/. List of all my VPSes: https://d.sb/servers
dnstools.ws - DNS lookups, pings, and traceroutes from 30 locations worldwide.
Would you care to contemplate the bottleneck on the OPNsense side? It's a completely different stack of course, but both should be thoroughly optimized by now.
More than double the throughput is quite a difference, even taking into account the minimal footprint of OpenWRT (I expect your router to run on > 128 MB of RAM, would it not?)
As far as I know, something about packet routing or NAT or something is still single-threaded on BSD. When I was testing, it was hitting 100% usage of a single core. On OpenWrt, it was using all cores evenly.
I was able to hit 8Gbps with a single connection using iperf3 when using OpenWrt. You can do the same with VyOS which is also Linux-based. That's essentially impossible on FreeBSD/OpenBSD, unless you have very high-end hardware with fast single-core performance (and even then, it's commonly accepted that you need multiple connections to reach good speeds with BSD)
Minimum requirements for OpenWrt for routers are 8MB disk/flash and 64MB RAM. Out of the box, the x86 version I was testing used around 10MB disk space and less than 50MB RAM. It's very nice seeing lightweight, unbloated software still exists. It does have a lot of packages you can install though.
It's not just more than double the throughput; it's also much less CPU usage
Daniel15 | https://d.sb/. List of all my VPSes: https://d.sb/servers
dnstools.ws - DNS lookups, pings, and traceroutes from 30 locations worldwide.
That explains. I just reverted the machine running my bare metal OPNsense to the heavier one, because the CPU did not support VTx (Atom D525) and I preferred to have it virtualized. Switching to OpenWRT would not only have enabled Linux VMs, but have allowed me to use containers, saving a ton of resources.
Thanks for the detailed reply!