ESXi/FreeNAS server build help

sippyCUP

Cadet
Joined
Jun 9, 2019
Messages
3
Hey all, new to the forum and very impressed by some of the threads I've read so far. This weekend I've probably spent 8 hours bouncing around some IP surveillance camera forums, Reddit, and various other spots before I found this forum. My goal is to architect a solution for running Blue Iris surveillance software with up to about 12 IP cameras (2-4mp), while also providing Plex (probably no more than one 1080p transcode at once), and a backup server for the house. I think FreeNAS fits into this for providing the pool for Plex and backup. Also want to leave myself some headroom for one more Linux VM for future use. If I could do it in one box using ESXi, that would be great from a thermal and energy use perspective. Noise/size/beauty doesn't matter, it will be in a closet. I have perused these applicable threads and taken notes:
https://www.ixsystems.com/community...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/
https://www.ixsystems.com/community/threads/freenas-virtualized-on-esxi-or-vms-in-freenas.58410/
https://www.ixsystems.com/community/threads/esxi-question.69241/
https://www.reddit.com/r/PleX/comments/4scaws/nas_vs_htpc/
https://www.ixsystems.com/community...r-passthrough-from-esxi-6-5-to-freenas.61304/

Another thing to mention is I intend to keep the Windows 10 Blue Iris hardware and setup totally isolated from the rest of the server and network for security reasons. So VLAN, no access to the FreeNAS media/backup volume from the Win10 VM, etc.

Overall I'm very comfortable building PC's, been using some flavor of Ubuntu for my main PC for more than a decade, experience with Virtualbox VMs, built a pfSense box recently, and I do random freelance computer work for small businesses a few times a year. But I'm not a networking or NAS guru, and this would be my first romp with ESXi, Plex, and FreeNAS. I'm going to provide a little background below, but then I'll follow that up with my proposed software and rough build plan and questions.

Blue Iris runs only on Windows and it will probably be the most CPU and I/O intensive application on a 24/7 basis. A lot of folks on the IP surveillance side claim you should only run a dedicated i5/i7 box for this, but others have had success recently with newer consumer hardware (such as an i7 6700) virtualizing inside ESXi. Regardless, this part of the equation gives me the most uncertainty because it's a steady CPU and IOPS load unlike how I will be using the FreeNAS share (backups and Plex usage). Here's a thread with a guy that does Blue Iris in a VM in ESXi but isn't hosting NAS on that box: http://www.blueirissoftware.com/forum/viewtopic.php?f=2&t=2639

Server requirements:
1. Support about 40-50 mbps of 24/7 IP cam recording (used this calculator:https://ipcamtalk.com/tools/calculators/hdd/). This will probably take about 30-40% load on an i7 6700 without Quick Sync hardware acceleration.
2. For Plex, must be able to support one transcode stream simultaneously. One Direct Play stream (home theater, probably Nvidia Shield client) and one transcode (tablet) simulteaneously would be ideal though!
3. Serve as a backup spot for 1-2 PCs. The FreeNAS pool should be able to support this no problem.

My thoughts on hardware/software setup (please critique!)

Modern consumer hardware (but open to like integrated Xeon mobos, ebay'ed servers, etc. if most effective):
-i7 6700 or similar
-16-32 GB ram
-Haven't actually decided how much space I want in the main FreeNAS pool, but let's assumed 4 x 8 TB. Trying to decide between going with internal SATA controller or separate HBA card for the FreeNAS pool, either way I know I need to pass-through ESXi to FreeNAS.
-SSD(s) for VMs and ESXi, probably separate one for a SLOG too. If I don't pass the SATA controller, these could be SATA, otherwise I guess M.2 NVME
-Probably a separate single hard drive for Blue Iris, something like a WD Purple 8-12 TB. There are logistical considerations on where I hook this up (* see below).
-I guess I need a UPS too

*If the single WD Purple went on a passed-through SATA mobo controller used for the 4 x 8 TB pool, now it's also in FreeNAS. I guess I could make it its own volume, and it's basically isolated at the FreeNAS level from the other volume. Then I would do virtual networking via ESXi to allow the Win10 VM to write to that single disk volume. If I wanted to leave the WD Purple out of FreeNAS and connect it direct to the Win10 VM via ESXi, I would need to either buy a separate PCIe HBA or figure something else out! Open to suggestions.

Overall I think the separate WD Purple mitigates some of the IOPS concerns of the steady-state Blue Iris load, plus should increase mean time to failure for the media FreeNAS volume (fewer writes!). The controller stuff is what I need advice on though.

Software Setup
-ESXi
-FreeNAS VM (could jail Plex)
-Linux VM: Would need if I didn't like Plex in a FreeNAS jail, but also might want this to run OpenVPN, etc.
-Win10 VM: Blue Iris
-Details: Passthru of a controller necessary for FreeNAS media/backup pool. Intel Quick Sync hardware decoding passthrough very desired for Blue Iris and/or Plex BUT I think only one VM can get passthrough at a time. I would chose Blue Iris for this (http://www.blueirissoftware.com/forum/viewtopic.php?t=4295).

Like I said, looking for critique/suggestions and I appreciate any feedback!
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I have perused these applicable threads and taken notes:
I think you may have missed these very informative threads:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

Virtually FreeNAS ... an alternative for those seeking virtualization
https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

FreeNAS 9.10 on VMware ESXi 6.0 Guide
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

Also, if you are considering a SLOG device, you probably need to read these:

The ZFS ZIL and SLOG Demystified
https://www.ixsystems.com/blog/zfs-zil-and-slog-demystified/

Some insights into SLOG/ZIL with ZFS on FreeNAS
https://www.ixsystems.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Testing the benefits of SLOG using a RAM disk!
https://www.ixsystems.com/community/threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

SLOG benchmarking and finding the best SLOG
https://www.ixsystems.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/

How to relocate swap to an SSD or other partition - [not needed in v 11.1 and newer systems]
https://www.ixsystems.com/community/resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

Why iSCSI often requires more resources for the same result (block storage)
https://www.ixsystems.com/community...res-more-resources-for-the-same-result.28178/

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage (iSCSI)
https://www.ixsystems.com/community...and-why-we-use-mirrors-for-block-storage.112/
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Intel Quick Sync hardware decoding passthrough very desired for Blue Iris
I am no video encoding pro, and time to do the research is limited, so I will just ask, how much difference does this make?
I ask this because my suggestion for hardware would be a used server. Retired systems from server farms can be had for very reasonable prices and would be more than capable of doing the things you are interested in, with the exception that they probably don't have Quick Sync. It is entirely possible to get a server that has two CPUs with eight to ten cores each (twenty real cores plus hyper-threading). If more CPU horsepower would make up for not having Quick Sync, it might be less costly to purchase a used server than to buy a more modern system that includes Quick Sync and you could get a server board that already has 64GB to 128GB of RAM installed with the ability (depending on the exact board) to go to 1.5TB of RAM. The hardware that is available is really quite good.
 

sippyCUP

Cadet
Joined
Jun 9, 2019
Messages
3
I am no video encoding pro, and time to do the research is limited, so I will just ask, how much difference does this make?
I ask this because my suggestion for hardware would be a used server. Retired systems from server farms can be had for very reasonable prices and would be more than capable of doing the things you are interested in, with the exception that they probably don't have Quick Sync. It is entirely possible to get a server that has two CPUs with eight to ten cores each (twenty real cores plus hyper-threading). If more CPU horsepower would make up for not having Quick Sync, it might be less costly to purchase a used server than to buy a more modern system that includes Quick Sync and you could get a server board that already has 64GB to 128GB of RAM installed with the ability (depending on the exact board) to go to 1.5TB of RAM. The hardware that is available is really quite good.

Difficult to ascertain! Some responses in this thread indicate up to a 100% reduction in CPU cycles: https://www.reddit.com/r/BlueIris/comments/b9wnu8/how_much_does_quick_sync_really_help/

However cursory review of this aggregated performance data doesn't really reveal that trend to my eye:
https://biupdatehelper.hopto.org/default.html#stats

Basically I can probably expected a 20-30% reduction in CPU cycles from Blue Iris with Quicksync.

I am definitely open to the Xeon server hardware, even without Quicksync. However, many of these servers are power hogs. I would have to compare annual power consumption of a modern i7 against the affordable server harware, then make a value judgement from there, also taking into account the likely lower cost and higher feature set of the servers.
 

sippyCUP

Cadet
Joined
Jun 9, 2019
Messages
3
Also thanks for the expanded reading list, I will definitely check out those threads!
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I am definitely open to the Xeon server hardware, even without Quicksync. However, many of these servers are power hogs. I would have to compare annual power consumption of a modern i7 against the affordable server harware, then make a value judgement from there, also taking into account the likely lower cost and higher feature set of the servers.
Other people have done that, you can probably search the forum and find some threads. Basically, the one I remember most, the cost of electricity would take about six years to add up to the cost of buying a new system instead of buying a used system.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
I am definitely open to the Xeon server hardware, even without Quicksync. However, many of these servers are power hogs. I would have to compare annual power consumption of a modern i7 against the affordable server harware, then make a value judgement from there, also taking into account the likely lower cost and higher feature set of the servers.
Another option to consider would be to have 2 separate systems -- 1 for Blue Iris and another for FreeNAS+Plex etc.

The advantages for doing this would be:
  1. your IP Camera machine is completely isolated even at the physical layer
  2. Having a machine isolated at the physical layer, i.e. separate RJ45 connection and possibly a separate switch would allow you to put the surveillance box in a DMZ in your pfSense box***. This allows you to truly separate your surveillance box from your LAN with proper configuration. This can possibly allow you to access your surveillance machine from the internet (in case you are traveling) and want to check up on your house
  3. It can even be a bit less powerful than an i7 -- since it will only be running 1 software + OS
  4. your FreeNAS machine can be low power (as compared to an i7 etc) for FreeNAS and Plex

*** Having a separate physical port, switch etc is not a mandatory requirement for putting a machine in a DMZ, but having them all separate actually follows the DMZ principles of separation of the 2 networks.


I have a FreeNAS box with 5 jails --(emby, couchpotato, transmission, radarr, syncthing) -- running on a measly Pentium G3400 with 16GB of RAM and I can easily get 1 transcode on the media server which is what you are also aiming for.

You might have to obviously compare the total electricity cost of those 2 machines compared to one i7 based virtual machine, if that is important for you. But you might be surprised with how much a Pentium can handle. I like to use my machines to "close to the max" capabilities (~80%-85%) instead of building a super computer and using only 5% of it's capabilities
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I am definitely open to the Xeon server hardware, even without Quicksync. However, many of these servers are power hogs. I would have to compare annual power consumption of a modern i7 against the affordable server harware, then make a value judgement from there, also taking into account the likely lower cost and higher feature set of the servers.

TBH I think most I-7 Vers a x9/x10 Dual Cpu set ups use less then my 8700K with a 970Gtx, 3x 2Tb Raid 0, 10GB/s Nic & 3x SSD System uses more then my X9 system with 18 disks and 5 SSD 2 Xeon 6 cores 12 Threads 2x plus 10GB/s Nic.

But also you are comparing apples to oranges, My X9 system is at 200-350 Watts, My I-7 System peaks at around 550 Watts idle it stands around 250w so you are trying to compare some thing that has very much two different uses.

Both systems have Platinum power supply's. My server has dual supply's not so much that it needs them just for redundancy.
 
Last edited:
Top