BUILD Virtualized FreeNAS 9.3, pfSense, Plex in an ESXi 6.0 Server (E3-1241 3.5GHz, 32GB RAM)

[George777]

I benefited heavily from the many threads in this forum, so I wanted to share back my successful build. My goal was to run FreeNAS w/ Plex and pfSense as my home router safely together in a virtualized environment, and wanted room to grow. I read up a lot on virtualizing FreeNAS before diving into this build, but in the end it worked out and seems to perform well. In a simple stress test, I had six streams transcoding on my local network. I already had two WD Red drives, and I added four other drives (2 SeaGate 3TB, 2 WD Red 4TB) with the thought of varying the wear out time. I'm running these six drives in one pool as RAID-Z2, and I have 10.5TB of space. While not necessary, I upgraded all the fans to Noctua to reduce the noise from the stock fans. The base server without the UPS, networking gear, or hard disks was $1,082.

Hardware​

Case: LIAN LI PC-Q25B Black Aluminum Mini-ITX Tower Computer Case ($120 @ NewEgg)
Motherboard: ASRock E3C224D4I-14S Extended mini ITX Server Motherboard ($230 @ NewEgg)

• SAS Controller from MoBo: LSI 2308 (Flashed to IT Mode, Drivers v20)

CPU: Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz ($229 @ MicroCenter)
Memory (32GB): Kingston 8GB 1600MHz DDR3L ECC CL11 DIMM 1.35V KVR16LE11/8KF ($59 * 4 @ Amazon)
CPU Cooler: Noctua Low-Profile Quiet CPU Cooler NH-L9I ($40 @ Amazon)
Power: SeaSonic G Series SSR-550RM 550W 80 PLUS GOLD ($80 @ NewEgg)
Case Fans:

• Noctua NF-F12 PWM Cooling Fan ($20 @ Amazon)
• Noctua 140mm Premium Quiet Quality Fan NF-A14 PWM ($23 @ Amazon)

ESXi Boot Drive: SanDisk Cruzer Fit 32GB USB 2.0 Low-Profile Flash Drive SDCZ33-032G-B35 ($10 @ Amazon)
ESXi VM Datastore: Kingston HyperX Savage 240GB SSD SATA 3 2.5 SSD SHSS37A/240G ($94 @ Amazon)
UPS: CyberPower PFC Sinewave Series Mini-Tower 850VA UPS CP850PFCLCD ($111 @ Amazon)
Disks

• WD Red 4TB NAS Desktop Hard Disk Drive 3.5 Inch - WD40EFRX ($150 * 2)
• Seagate 3TB NAS HDD - ST3000VN000 ($105 * 2)
• WD Red 3TB NAS Desktop Hard Disk Drive (Previously Owned * 2)

Networking
Switch: NETGEAR GSS108E 8-Port Switch GSS108E-100NAS ($59 @ Amazon)
Wireless Access Point: Ubiquiti UBUAPACLRUS UAP-AC-LR-US 802.11ac ($108 @ B&H Photo)

 

[George777]

Software​

The server is running ESXi 6.0 as a host. One network port is tied to pfSense as the WAN port, the other is tied to the Switch to support the LAN and all the virtual machines. I also have an Ubiquiti WAP on the Switch to support my wireless devices (phones, printer, desktops, laptops).

4 Virtual Machines:
pfSense has 1 socket with 2 cores, and 4GB of RAM becaused I wanted the flexibility to run Snort and an OpenVPN client.
FreeNAS has 1 socket with 2 cores, and 24GB of RAM. It is directly accessing all six drives that are connected via the LSI 2308 passed through via VT-d. I flashed the LSI 2308 to IT mode using the information on ASRock's website. This VM is very much geared towards RAM to support the ARC. I configured this VM using this tutorial mostly: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/ I originally tried to implement the Storage network as described in the tutorial but found Plex was much slower than running on hardware. I dropped that virtual network switch and now Plex runs perfectly.
Plex has 2 socket with 2 cores (4 vCPUs), and 2GB of RAM. I wanted Plex in it's own virtual machine (CentOS 7) because more features are available to Plex when it is on Linux and not FreeBSD. This VM is very much geared towards CPU. This VM basically mounts FreeNAS shares via NFS. I tried to move most of the Plex cache to FreeNAS shares as well via symlinks.
UPS-VM has 1 socket with 1 core and 512MB of RAM. I'm running CyberPower PowerPanel Business Edition Virtual Machine (CentOS 6) to safely shut down the system. This VM barely consumes 30mhz and 100mb of RAM. I configured this VM using this tutorial mostly: https://tinkertry.com/configure-automated-shutdown-homelab-datacenter-15-minutes

 

[George777]

Reserved for photos if there is any interest

 

[destrekor]

Thank you! Your setup is very similar to what I am intending to run.

I'm thinking of going with a Xeon D build, but I had been very curious about the ability to passthrough an on-board LSI HBA, and in general flashing it. Glad to hear that is possible, so I'm assuming it should be equally possible on a Supermicro Xeon D board that has it.

It'll have less per-core performance but I am also not worried about needing to transcode 6 streams at once, and if I go with a 6 or 8 core model then multi-threaded work will benefit greatly.

Almost everything else is exactly how I planned to do things, save for the fact I'll likely be going with Sophos XG instead of pfSense. And can you expand on the differences between a separate Linux/Plex VM as opposed to what is capable with the Plex Jail on FreeNAS?


Oh, and can you explain your reasoning for the vCPU configuration? When you say 1 core with 2 sockets, what is the end result? Are you saying 1 physical core so that's 2 vCPU spread across 2 sockets? Or is it literally 1 vCPU with 2 sockets? If the latter, what is the point? And is there better performance with 2 vCPU across 2 sockets as opposed to 2 vCPU on 1 socket?

 

[George777]

I actually wanted to do this build with the Xeon D 1540's but the build prices out more than I wanted to spend (that and the MoBo I wanted hasn't been "in stock" since being announced a year ago.) I think it should definitely be possible as long as the LSI v20 drivers are there. I was actually leaning towards one of these on-board LSI controllers because I prefer the integrated build and not having to rely on finding a reliable SAS card (possible knock-off) on eBay.

I certainly have no plans to transcode 6 streams at once (more than likely 2 on average), but as I started the stress test, it just became how far can I push this build. I ran out of devices/clients before I stressed the build and saw any buffering.

Re: Plex FreeBSD port, this is a good link: https://forums.plex.tv/discussion/1...linux-version-are-missing-on-the-freebsd-port Right now the main difference is the lack of Plex Premium Music. My thinking here was that Plex doesn't need to run on the same host because transcoding is the bottleneck, so I'd rather use an OS that they demonstrated stays up to date with new features. The fact that there is already a noticeable difference between Linux and FreeBSD versions of Plex, made me want to go to a better supported configuration. I also noticed that the Plex Plugin for FreeNAS was far behind the current version of the FreeBSD download, so I think my decision was more focused on staying current in terms of Plex features with my Plex Pass, and minimal work to keep it up to date. Overall, I went with CentOS 7, and now I can just do:

Code:

wget https://downloads.plex.tv/plex-media-server/plexmediaserver.rpm
yum localupdate plexmediaserver.rpm
systemctl start plexmediaserver
systemctl status plexmediaserver

Regarding sockets/cores, I had a mistake in my verbage above that I fixed (transposed cores/sockets). I found this page to be helpful in that ultimately it doesn't really matter: http://www.virtadmin.com/sockets-vs-cores-vmware/

 

[majorjake]

This looks like a very similar build to what I'm planning.

I wouldn't have considered a dedicated Plex VM, that's a gread idea.

Are you backing up the Datastore drive? Why not mirror it? I'm new to ESXi so forgive the question if it's a bit green.

 

[sivasunken]

I am planning to do a similar build. I want to use the e5 2670 v1 processor... Is it a good choice and what is a good choice for a mother board? I want to be able to do at least 3 streams of x265 simultaneously...
Thanks in advance

 

[George777]

This looks like a very similar build to what I'm planning.

I wouldn't have considered a dedicated Plex VM, that's a gread idea.

Are you backing up the Datastore drive? Why not mirror it? I'm new to ESXi so forgive the question if it's a bit green.

I could have mirrored that drive. Since I only care about the logical files, I back them up manually occasionally. This gives me more backups for the megabyte. Since a mirrored drive would mirror the empty space.

 

[George777]

I am planning to do a similar build. I want to use the e5 2670 v1 processor... Is it a good choice and what is a good choice for a mother board? I want to be able to do at least 3 streams of x265 simultaneously...
Thanks in advance

That processor should be fine. I compared it with mine and it is slightly underpowered compared to the one I used. I am able to do 3 streams, so I think you should be able to as well. For a motherboard, I chose one that had an integrated LSI SAS controller. I found this was cheaper than buying a brand-name part, and I didn't trust the knock-offs on ebay with my data.
Here is the link comparing processors: http://cpuboss.com/cpus/Intel-Xeon-E5-2670-vs-Intel-Xeon-E3-1241-v3

 

[sivasunken]

Bas

That processor should be fine. I compared it with mine and it is slightly underpowered compared to the one I used. I am able to do 3 streams, so I think you should be able to as well. For a motherboard, I chose one that had an integrated LSI SAS controller. I found this was cheaper than buying a brand-name part, and I didn't trust the knock-offs on ebay with my data.
Here is the link comparing processors: http://cpuboss.com/cpus/Intel-Xeon-E5-2670-vs-Intel-Xeon-E3-1241-v3

Based on your suggestion, I have come with this parts list.
https://ca.pcpartpicker.com/list/YGVqBP
Also I will be adding a 5 hot swap bay module. I'd like to know your opinion on this build

 

[George777]

Based on your suggestion, I have come with this parts list.
https://ca.pcpartpicker.com/list/YGVqBP
Also I will be adding a 5 hot swap bay module. I'd like to know your opinion on this build

I think if you (or anyone) is planning on diving into an ESXi FreeNAS build, you really need to read up on HBA controllers. It is a necessary component to the build for ESXi to pass through the hard drives to the VM. I started with this link and kept reading from there: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

 

[sivasunken]

I think if you (or anyone) is planning on diving into an ESXi FreeNAS build, you really need to read up on HBA controllers. It is a necessary component to the build for ESXi to pass through the hard drives to the VM. I started with this link and kept reading from there: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

I looked through some links reg esxi and freenas and looks like i do need a HBA controller. I think if i replace the motherboard with SuperMicro X11SSH-CTF, I should be able to use to implement this build. But one question is would this controller (3008) version work for my SATA drives that i wish to add to my build?

 

[George777]

I looked through some links reg esxi and freenas and looks like i do need a HBA controller. I think if i replace the motherboard with SuperMicro X11SSH-CTF, I should be able to use to implement this build. But one question is would this controller (3008) version work for my SATA drives that i wish to add to my build?

The RAM you picked supports ECC, your CPU supports VT-d, and it looks like the 3008 can be flashed to IT mode. It should work. One thing I'd look for is other people who have used this controller in this forum (for flashing and firmware instructions), but it seems like it should work. Good luck!

 

[sivasunken]

For the switch, will any type of switch work? I am thinking of using Netgear gs108e or gs108-400nas. Is there a specific type of switch to easy connection/setup with the access point and pfsense?

 

[George777]

For the switch, will any type of switch work? I am thinking of using Netgear gs108e or gs108-400nas. Is there a specific type of switch to easy connection/setup with the access point and pfsense?

I think either would work. One feature I wish I thought of was a switch with Power over Ethernet (PoE). The Ubiquiti WAP uses PoE and comes with a power adapter, but if your switch supports it, that is one less cord.

 

[sivasunken]

Did you connect the esxi data store to the sata ports directly or through the sas interface?

 

[sivasunken]

Software​

The server is running ESXi 6.0 as a host. One network port is tied to pfSense as the WAN port, the other is tied to the Switch to support the LAN and all the virtual machines. I also have an Ubiquiti WAP on the Switch to support my wireless devices (phones, printer, desktops, laptops).

4 Virtual Machines:
pfSense has 1 socket with 2 cores, and 4GB of RAM becaused I wanted the flexibility to run Snort and an OpenVPN client.
FreeNAS has 1 socket with 2 cores, and 24GB of RAM. It is directly accessing all six drives that are connected via the LSI 2308 passed through via VT-d. I flashed the LSI 2308 to IT mode using the information on ASRock's website. This VM is very much geared towards RAM to support the ARC. I configured this VM using this tutorial mostly: https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/ I originally tried to implement the Storage network as described in the tutorial but found Plex was much slower than running on hardware. I dropped that virtual network switch and now Plex runs perfectly.
Plex has 2 socket with 2 cores (4 vCPUs), and 2GB of RAM. I wanted Plex in it's own virtual machine (CentOS 7) because more features are available to Plex when it is on Linux and not FreeBSD. This VM is very much geared towards CPU. This VM basically mounts FreeNAS shares via NFS. I tried to move most of the Plex cache to FreeNAS shares as well via symlinks.
UPS-VM has 1 socket with 1 core and 512MB of RAM. I'm running CyberPower PowerPanel Business Edition Virtual Machine (CentOS 6) to safely shut down the system. This VM barely consumes 30mhz and 100mb of RAM. I configured this VM using this tutorial mostly: https://tinkertry.com/configure-automated-shutdown-homelab-datacenter-15-minutes

How did you setup the pfsense vm? I cannot seem to connect the networks or access the webgui. could you provide some tutorial.

 

[ViciousXUSMC]

Hope this is not too much of a necro, about the only thread I found when searching for "esxi, freenas, pfsense" I have a setup at home that uses a Dell R710 running Freenas 11, inside are 3 jails 2x Transmission Jail and 1x Plex Jail. 1 VM inside Ubuntu server running Home Assistant and Ubiqiti Unify services (Also using Ubiqiti AP like you are)

I then have an old Dell desktop that I added a 4 port Intel NIC to that is running PFSense.

Now that I am more comfortable with all these devices and there configuration, I was sort of thinking to consolidate everything to the R710 as it would save me space, power, and also let me utilize more of the server. Right now I almost never use much of the servers power (other than RAM of course)

2x Xeons, 64gb RAM, and 6x 8TB WD RED in Raid Z2 on the backplane, 2x 250GB SSD on the SATA ports, and Freenas is installed on two thumb drives running as a mirror.

Just seems logical to virtulize this, earlier this year I blew up my PFsense box doing an update, I was able to recover via console and some lucky googling, but the idea of VM snapshots to recover anything especially during updates sounds amazing, the idea of taking my jails and vm's out of freenas and into ESXi where they can be managed better and letting freenas do pure storage/shares also sounds appealing.

So just chiming in to ask how the build went, and if I decide to break my setup to rebuild it, wondering what would be in store for me, hope there is a way to "move" everything without a full rebuild/reconfigure.

 

[George777]

I'm still really happy with this setup. I've upgraded to a new server but kept the same basic configuration. I moved a lot of the functionality in to VLANs now. I've appreciated the ability to snapshot often and roll back when things break down. I still use each products' internal backup functionality if it exists (pfSense, FreeNAS, NextCloud). Overall, no major surprises. I tried using VT-d to pass through the NICs and also virtualizing the network. I've found having the network virtualized is better without any noticeable bandwidth penalty, but gives me the ability to monitor traffic better and lock down promiscuous mode.

 

[George777]

How did you setup the pfsense vm? I cannot seem to connect the networks or access the webgui. could you provide some tutorial.

Sorry, I missed this earlier. I think the key for me was using static IPs in a subnet I didn't plan to use. I also used the console. Once I had everything configured properly, I switched over to DHCP in the correct range.