PfSense Dual 10Gbps NIC Pass Through in VM

[visamp]

I am running TrueNAS-SCALE-22.12.3.2 in a Dell 7920 Workstation with Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and 250 GB of Samsung ECC memory (2666 hz). I have a single 10Gbps NIC setup as the main access to the TrueNAS.

I am trying to setup PfSense in a VM on my TrueNAS passing through another NIC dual 10 Gbps PCI-E NIC X540 Controller I have. The NIC shows below in the screenshot as enp245s0f0 and enp245s0f1 respectively. However, I only have the option to select one and not both of them as I set it up. But I do want to pass both enp245s0f0 and enp245s0f1 through to the PfSense VM (one as WAN, the other as main LAN). But as I don't have the option to pass both through, I'm not sure if I just need to pass one (ie, the one that will serve as the WAN) and later do something or not to pass the other through.

My question is this: do I need only pass through one of the two ports on the dual 10 Gbps PCI-E NIC X540 Controller? If so, do I need to do anthing to the second post VM setup to make it serve as a part of the originating LAN port for my network?

I also want to understand if anyone has a recommendation for using VirtIO or Intel e82585 in this particular setup (screenshot below). Is it better to use one over the other?

Just trying not to brick this so many times as I learn. Thanks in advance.

 

[Heracles]

I am trying to setup PfSense in a VM on my TrueNAS

Not a good idea...

The hypervisor in TrueNAS is a type 2. To virtualize something like pfSense, you would need a type 1 hypervisor (like ESXi).

A type 2 hypervisor only virtualize the local hardware a VM is running from (CPU, RAM, ...). When you need to virtualize a firewall like pfSense, you must also virtualize its network environment and to do that, you need a type 1 hypervisor.

 

[sfatula]

I wouldn't ignore Heracles, but you should be able to pass the second NIC after the VM is created via the devices screen.

 

[visamp]

Thank you both! I'm going to give it a try because if I break it, I'll destroy the VM and get back on eBay to look for another machine for my firewall to run PfSense.

I wanted to give it a try because this YouTuber (https://youtu.be/qDs-MumRElE?si=4Lt-frQrN_d1_o7y) showed that it could work, and I wanted to avoid yet another device running in my office.

It would be cool if TrueNas would allow you to pass through more than one device at the same time, though.

Any thoughts on VirtIO or Intel e82585 in this experiment?

 

[Heracles]

You can go for an appliance like an SG-1100 as a pfSense solution instead of a full fledge server.

 

[sfatula]

You can go for an appliance like an SG-1100 as a pfSense solution instead of a full fledge server.

I get what he's saying though, he wants to avoid yet another device, even a smaller cheaper one. I would too if it were possible.

 

[MrGuvernment]

Thank you both! I'm going to give it a try because if I break it, I'll destroy the VM and get back on eBay to look for another machine for my firewall to run PfSense.

I wanted to give it a try because this YouTuber (https://youtu.be/qDs-MumRElE?si=4Lt-frQrN_d1_o7y) showed that it could work, and I wanted to avoid yet another device running in my office.

It would be cool if TrueNas would allow you to pass through more than one device at the same time, though.

Any thoughts on VirtIO or Intel e82585 in this experiment?

Also consider any time you take down truenas, reboot for an update, anything else, you take down your internet. Personally, and from a security standpoint, your firewall should be a separate device.

PFSense can run on a small SFF tower from HP or Dell, and tuck away in a corner. I run Pfsense on an HP SFF with a i5-6500 and a dual 10Gbps SFP+ NIC in it and runs smooth as butter with a 1Gbsps/1Gbps Fiber from my ISP, and since it is on its own, takes little power, I can work on any other system and not have the wife complain if Inet drops...

 

[MrGuvernment]

I get what he's saying though, he wants to avoid yet another device, even a smaller cheaper one. I would too if it were possible.

Certainly, less devices to worry about is nice, but, also consider the hassle when Inet is down for your house / office, where ever, because you needed to do something to your truenas server or the VM flaked out.

Again personally, TrueNAS box should be just that, a NAS, not a one stop shop for everything, sure, consolidation is nice, but all your eggs in one basket just sounds like a problem waiting to happen, whether security based or stability.

TrueNAS system should never be directly connect to the internet, even if it is via a VM running on it...

 

[Patrick M. Hausen]

If you cannot pass-through more than one PCIe device, you can of course create two bridge interfaces and attach two virtual NICs of your VM to these. I achieved near Gigabit speed on an Atom based platform with TrueNAS CORE and OPNsense that way. 10 G is probably not possible. In the VM you almost always want VirtIO.

 

[sfatula]

Certainly, less devices to worry about is nice, but, also consider the hassle when Inet is down for your house / office, where ever, because you needed to do something to your truenas server or the VM flaked out.

Again personally, TrueNAS box should be just that, a NAS, not a one stop shop for everything, sure, consolidation is nice, but all your eggs in one basket just sounds like a problem waiting to happen, whether security based or stability.

TrueNAS system should never be directly connect to the internet, even if it is via a VM running on it...

I understand and somewhat agree, but that tradeoff is up to him. What you or I pick may not be what he picks. For me, no internet would be very minor in nature, no big deal at all. I reboot my truenas once every couple months. I would simply do it at night when no one is using it. May be different for you. Scale has been rock solid for me. As directly connected to the internet, I wouldn't do that either. But I wouldn't bother with pfsense either. Everyones use and desires are different.

 

[visamp]

Also consider any time you take down truenas, reboot for an update, anything else, you take down your internet. Personally, and from a security standpoint, your firewall should be a separate device.

This I understand in part. I don't mind losing access to my internet if things come down for a bit. Right now I am using a Netgear router. The main reason I want to do this is twofold:

1. Within my pihole dashboards (I have two running recursively in two Le Potatos), I had to block netgear.com and routerlogin.com (or .net, I'm not sure which one it is right now) because my router was constantly pinging Netgear relentlessly despite having disabled their forced upon me security suite and changing the time fetching server. I submitted a request to Netgear support about this, and did get a response that they were looking into it... then I did not hear back from them again. Meanwhile my netgear router is pinging a blackhole on my system... I can only assume it was checking for updates, but I don't really need hundreds of checks per day for that.
2. I want to install and run within PfSense wireguard so that I can easily remote into the system. I know that there are other ways to install and setup Wireguard, etc., but this way seems to make sense to me as I eventually want to open this up to my small business to share files with a few other people, etc.

The part I understand in the comment is the loss of network connectivity. That I can live with as I don't see it being something I would do during my normal business day. The second part is the comment I hear about security, but I admit that I don't understand fully. I have heard that it is better to have separate devices for security, but I don't really understand why. If I install PfSense on a second device and connect that device via ethernet to my home network, then I am in reality creating a single system, no? I don't know though - I probably have a blind spot from which I would welcome being disabused.

you can of course create two bridge interfaces and attach two virtual NICs of your VM to these.

This I do not know how to do, but I will investigate. I will likely try this without doing it this way and just adding the second port within PfSense once I install on what I choose to be the main port of the dual port NIC.

10 G is probably not possible.

Is this in response to my proposed setup or just in general virtualizing PfSense?

In the VM you almost always want VirtIO

Because I've been so bold as to ask questions like this already, I ask why you suggest this course of action over the alternative (Intel e82585 in my case)?

HP SFF with a i5-6500 and a dual 10Gbps SFP+ NIC

And if this experiment breaks, I have already been searching through eBay for options. I want something with 10Gbps NICs because Comcast is telling me that by Christmas this year they will have expanded to 10Gbps up and down. If they do, I'll be glad. If not, at least I'll have my next entry ready to go in my diary.

I would too if it were possible.

I'll let you know how I break it all when I give it a try. Maybe it is possible; the guy on YouTube had it working, but I don't know if that remained the case or whatnot. Either way, I am just amazed at this entire software and the ecosystem of TrueNAS, etc.

 

[NickF]

The best way to do this is to pass the PCI-E device (the NIC) in its entirety through to the VM.

You can theoretically run pfSense with VirtIO NICs tho. Don't let the nay-sayers stop you. I've ran pfSense as my guest wireless captive portal which had hundreds of thousands of clients trying to connect every day....with VMXNET3. (Long story). But of course, with virtualized hardware in this particular application, YMMV.

Depending on your performance requirements, either solution may be valid.

Also the mention of KVM on SCALE being capable of doing this because it's "Type 2" is hogwash. KVM is one of the most mature and actively used/maintained Hypervisors out there. https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine.
It's also not 2002, and the Type-1 vs Type-2 distinction is not as black and white as it used to be anyway...Welcome to hyper converged land.

This concept is actually on my to-do list for a project write up. Also, see the SCALE Virtualization resources in my signature. It will help. The problem you originally posted is just simply that you can only add one NIC in the wizard. Click on devices and you can add the second NIC. Or click on Devices, and pass through the entire PCI-E device.

View attachment 69887

But there are of course still other caveats. This one is the big one:

The part I understand in the comment is the loss of network connectivity. That I can live with as I don't see it being something I would do during my normal business day

The bit about the security implications may be valid, depending on your deployment model. Details matter here. But in general, I disagree that a firewall has to be a physical appliance. I have run virtualized pfsense boxes on ESXI for years. I have run "official" Palo Alto VMs on ESXI for years. Also semi related, I have run load balancers virtualized for years also...

I am not an authority figure on anything, but I think I know a thing or two having run a large organization's back-end infrastructure for 40+ sites and over 25,000 users. So my musings here come from experience.

 

[sfatula]

I've never been a huge fan of one device per function, certainly not at home. Too many OSes, things to keep up with, more complexity the way I see it, more vendors to keep up with, firmwares, etc. etc. I get the idea and there are some downsides to multifunction machines, but, I think advantages outweigh those.

Let us know how it goes! Fun experiment.

 

[NickF]

Personally at home I have uncoupled my pfsense box from my hypervisor. I actually did that before switching from ESXI to SCALE. So for me, I went from 2 boxes, a hypervisor and a nas to once again 2 boxes, except a hyperconverged hypervisor (SCALE) and a router.

In any case, the only reason I personally did that was because my wife would get mad when the internet goes down. Now she only gets mad when Plex goes down. Small nerd husband victories. Trying to plan maintenance windows at home would be hilarious...she already thinks I'm nuts. So I'll take what I can get.

Professionally I virtualized firewalls all the time. But mostly in those cases, they were not my egress to the internet and instead only as a nice separate sandbox for ancillary services and people I didn't trust. For similar reasons as to my reasons at home, I probably wouldn't virtualize a firewall as an egress point to the internet in my professional life. Too many variables, not enough benefit for the usecase.

For similar use cases I also used router-on-a-stick...just a re-used old SFF desktop with a single network interface and a couple of VLANs. I'm sure that will make someone here cringe...but for that use case (and the utter lack of budget) it was perfect.

Hope that anecdote helps. How you use a thing matters, and sometimes it's okay to be scrappy and color outside the lines. The usecase here, I personally would not do it. But its for alot of non-technical reasons like the WAF (Wife-Acceptance-Factor, my sensei taught me that LOL).

But I would still say, if this is as much about your personal journey to learning IT stuff...go ahead and virtualize it. Learn by breaking things (and maybe someday probably figuring out how to fix them eventually) It's the homelab way. And in my experience, it's "THE WAY"

Reddit - Dive into anything

www.reddit.com

 

[ChrisRJ]

I run Pfsense on an HP SFF with a i5-6500 and a dual 10Gbps SFP+ NIC in it and runs smooth as butter with a 1Gbsps/1Gbps Fiber from my ISP, and since it is on its own, takes little power, I can work on any other system and not have the wife complain if Inet drops...

Could you please share the exact model?

 

[Patrick M. Hausen]

Because I've been so bold as to ask questions like this already, I ask why you suggest this course of action over the alternative (Intel e82585 in my case)?

Because for virtualized network interfaces VirtIO has the least overhead and hence performance penalty. I only tested this setup - virtualized network - with physical 1 G on the TrueNAS CORE host and I achieved 700-800 M/s of throughput.

Now if you use PCIe pass through, the VM will see the hardware device and there is no virtualization in place at all. So 10 G speed will be possible, but depending on the rest of your system of course.

HTH,
Patrick

 

[NickF]

Let us know what you end up doing

 

[MrGuvernment]

...........................


The bit about the security implications may be valid, depending on your deployment model. Details matter here. But in general, I disagree that a firewall has to be a physical appliance. I have run virtualized pfsense boxes on ESXI for years. I have run "official" Palo Alto VMs on ESXI for years. Also semi related, I have run load balancers virtualized for years also...

I am not an authority figure on anything, but I think I know a thing or two having run a large organization's back-end infrastructure for 40+ sites and over 25,000 users. So my musings here come from experience.

Certainly on ESXi, a proper hypervisor, but TrueNAS is a NAS OS at heart, with layers on top, those layers add extra layers and complexity and exploitability and layers many of us may not see or understand. When it comes to exploits or getting hacked, it is not a matter of if, but when. I know most people figure they don't have anything important in their home networks or think "but why would someone hack me" not realizing it is a bot army out there scanning 24/7 the world looking for an exploitable system to show up on the internet and not until they do get hit, then have the light bulb moment of "Oh crap....I didnt have that backed up anywhere else"

I block in and out on my PfSense purely for numbers to se (need to get my SIEM back up one day to get real details) but this is over just 10 days in/out

Inbound from WAN over 2 mins

I know I can be over the top, but a NAS often has things on it most people would not want to lose....so not having that system with a direct link to the internet in such a way, to me, just helps me sleep at night.

 

[NickF]

I think we may be hitting an impasse soon.. but I'll bite.

Certainly on ESXi, a proper hypervisor, but TrueNAS is a NAS OS at heart, with layers on top, those layers add extra layers and complexity and exploitability and layers many of us may not see or understand.

I do hear your point here...But I think you are making some improper assumptions.

• KVM, the hypervisor in SCALE, is a highly mature offering using in the Enterprise and the Cloud...soooo I don't understand?
• I think that you'll find that is the opposite of the direction TrueNAS SCALE is going. The developers have shown that they are very much taking security seriously...as evidenced by loads of new official documentation in that regard.
• I think you will also find that the developers have shifted focus to removing packages to decrease the security footprint, while maintaining features through apps in k3s. This is especially apparent in the recent Cobia beta.
  
  If you think of SCALE more like Proxmox with a storage-focused UI (and better a much better one in general imo), rather than as some nebulus blob you don't undersyand....I think you might change your tune. This isn't a far-fetched correlation. They are both based on Debian, they both use KVM, the both have the ability to host containers (LXC vs k3, but containers), they both have ZFS support and they are both offered in "Production Environments" and "Homelabs" The big difference here is Proxmox can do more stuff, but it's also alot more fiddly. Whereas, as you rightly pointed out, TrueNAS is designed as an appliance so all those crazy bells and whistles are not available.
  
  FWIW you can run "Entperise" level firewall/networking products on KVM. If they didn't think it was secure enough to offer that in their supportability matrix, they certainly would not. And those folks driving those decisions are much smarter than you or I. Example: https://docs.paloaltonetworks.com/v...ployment/set-up-the-vm-series-firewall-on-kvm

When it comes to exploits or getting hacked, it is not a matter of if, but when. I know most people figure they don't have anything important in their home networks or think "but why would someone hack me" not realizing it is a bot army out there scanning 24/7 the world looking for an exploitable system to show up on the internet and not until they do get hit, then have the light bulb moment of "Oh crap....I didnt have that backed up anywhere else"

I do not and have not disputed this claim? If anything, my rather lengthy posts should illustrate that I know and understand this. In each of those examples I am demonstrating the use of firewalls inside of firewalls. I am a huge advocate for generally taking network segmentation to a level far beyond just VLANs or other basic things. Security in layers, and all. If you'd like, we can sidebar on the benefits of PacketFence...the other pf...(Open source Cisco ISE)

I block in and out on my PfSense purely for numbers to se (need to get my SIEM back up one day to get real details) but this is over just 10 days in/out

Sure. We are in agreement. Although, I think far more is to be said about Suricata than pfBlocker...(or better still, even ZenArmor if you want to get crazy). But for the record...you can run pfBlocker on a pfSense firewall...regardless of whether it's on bare-metal or in a VM. Your appeal to your general sense of the importance of security is good, but it does not have anything to do with the question at hand.

---

Also, FWIW on the "TYPE 1" VS "TYPE 2" bit. KVM is literally listed as a "Type 1" hypervisor.

I'm sure we can have some semantical arguments on whether or not KVM running on a host OS like TrueNAS or even Hyper-V running on a phat install of Windows Server can even truely be considered "Type-1" hypervisors. We can also make semantical arguments that the source I cited there is invalid. In either case, lets consider some potential arguments there. We can even make the argument that the above screenshot shows that KVM is also a Type-2 hypervisor.

If you are willing to make the above argument, wouldn't you also have to conclude that anything that calls itself "hyperconverged" should not be definitionally considered TYPE-1 hypervisors either? And I don't think anyone is claiming that point. Seeing as how VMWare VSAN exists and everything.

Given the inconsistencies... We can infer a bunch. Rather than try to re-define KVM as a "True" Type-1 hypervisor...(the very fact that I have to put that in air-quotes there to convey this argument sorta proves my point)...let's focus on what matters. We should not be defining and basing our assumptions of the quality of a hypervisor based exclusively on the "Type" of the hypervisor. It's not really relevant any longer. If it were, "hyperconverged" would not exist...it kinda needs a base OS :)

In any case, imma keep doing me. Feel free to keep doing you. Here's my SCALE box. Dooing pretty well as a hypervisor for my deployment. If you consider not seeing any storage processes really eeking out too much CPU. Consider also, nearly all of my processes consuming CPU cycles are from KVM. This does not look like an example of a bad hypervisor to my eye.

@visamp We've obviously gotten into the weeds here a bit. But this is definitely absolutely all relevant to the spirit of the questions you are asking...even if not direct answers.

 

[MrGuvernment]

No impasse, not at all, and always happy to learn, or be corrected if I am thinking wrong, or have wrong info too or just being too narrow in my thinking. If we are not always learning, we are slowly dying!

KVM I am very familiar with, is core also using KVM? or only Scale?

TrueNAS I do not doubt the security of in or it's self, considering it is an enterprise level product and has paid options, so sure they work very hard to keep it as secure as possible.

Perhaps it is the old fashion IT guy in me (even though I am only early 40's) but having my NAS direct to the internet in any form...just scares me!

I actually feel like we are more alike vs an impasse, like you noted, segmentation, and seeing what I have seen over the years, and what you see companies have once breached wondering "What the hell, basic segmentation would of saved you!" type things, I tend to let that carry over to even home users who just want simplicity in the end.