Switching to a Virtualized freeNAS build

[ChrisReeve]

Hei

I am currently running the following setup:

Motherboard: Supermicro X9SRL-F
CPU: E5-2650 v2 (8C/16T)
RAM: 64GB Hynix 1600MHz (4x16GB)
NIC: Intel X540-T2 10Gb RJ45
Drives: 9x4TB WD Red

I have freeNAS directly installed on this server, but I have just started getting into virtualization.

With my usecase of the server, the E5-2650 v2 is overkill. I rarely see more than three simultaneous stream, and 90% is direct play anyways. I would therefore like to start hosting other kinds of servers from my machine. I did some testing, and some reading, and it looks like VMs are the way to go.

My plan for the future is as follows:
- Sell my current drives, and upgrade to 10x10TB WD Reds.
- Buy two HBAs for passthrough with VMs. This will leave me with 16 SATA-ports for the NAS-portion of my box. More than enough, as my case only fits 11 3.5" HDDs.
- Do a complete reinstall of my server, with ESXi free as my barebone hypervisor, and run (at least) two VMs.
+ One running freeNAS/Plex, assigning it 4 cores/8 threads. This should be more than enough for my use.
+ One running some other OS (still completely open, I have read good things about CentOS, but I might just even go with Windows Server), to host my other stuff. By other stuff, I am thining about things like a TeamSpeak server, Terraria Server, CS:GO server, Garrys Mod Server, etc. Important note: It does NOT have to be able to run all of these concurrently. We are a group of about 5-10 people who will use these servers, and not all at the same time. Minecraft and TeamSpeak might run 24/7, with the others being able to shut down when they're not needed.

This leaves me with a few questions.

1. How does passthrough work with 10GbE NICs? I am required to keep my speeds. Today, I see read/writes of 800/500MB/s to/from my server, depending on ARC hit ratio etc. It is unacceptable to return to 1GbE speeds for my NAS. Will I be able to keep these network speeds on my local network? One solution, I guess, could be to just do a HW passthrough of the 10GbE NIC to freeNAS, and use one of the onboard 1GbE connections to my game/voice server.

2. SSDs/drives. I am planning on buying a couple of SSDs to run the VMs from. I might not need to do this for my freeNAS, I could just run it off a USB stick, but for the other ones, it is a requirement. What is the best way to do this? two drives in RAID1? (2x500GB drives should be enough to cover my needs)

3. Are there any other concerns I should be thinking about at this stage? Any reasons this would not work at all?

 

[John Doe]

hi,
I did the same a couple of month ago and I am very happy with virtualized freenas on esxi

for point 2) I can recommend to use mirrored ssds in freenas and to create a nfs storage that will be passed back to esxi to host the vms.
that gives you the advantage to rollback a snapshot and the redundancy, as esxi does not provide raid (or else) functionalities.

3) my usecase varies from yours but one point I would make sure, to work with docker containers on debian or centOS to host e.g. nextcloud, plex etc. all the small stuff which does not require a lot resources in a single VM.

esxi allocation of resources to the vms is not easy. but you can find a few benchmarks. sometimes it is not the best to throw as many cores (threads) to a vm, even though they are not used!
in my case I have better results with 3 threads on freenas instead of 4 or more. same with plex, in my case it is most efficient to just use one thread.

I have no experience with 10gbit nics, so no comment from my side.

take a bit of time to plan the networks and the virtual switches. probably you want to have something like a DMZ for the game servers, a vpn network, a good firewall and a network which is only accessible within esxi (e.g. freenas to esxi for nfs storage)