new all in one server [HW picking review]

[John Doe]

Hi folks,

my current server (signature) might need to be replaced with something new, something fresh!

would you mind to share your thoughts:

CPU:
Intel Xenon Silver 4310
12 core, 3.3GHz

Motherboard:
Supermicro X12SPi-TF

Usecase:

Basic use case is to store data, fast VMs, fast iSCSI. Together with the VMs it shall be an all in one box

ESXi Hypervisor shall host:

• Freenas
• pfSense
• bunch of VMs

Pool Config:
6x 10 TB spinning rust passed thru to Truenas -> can onboards controller be passed thru to truenas?
HDD shall be smb storage as Raid Z2

Asus hyper m.2 equipped with 4x 2tb NVMe SSD -> passed thru to truenas
mirrored 2x2tb shall be iscsi
mirrored 2x 2tb shall be fast NFS for VMs

Priority points (1 low - 10 max)
10 - Data integrity
8 - Power consumption
7- Networking speed
6 - Cost of procurement (budget for CPU, RAM, Motherboard is ~2k USD, the less the better)
5 - Cpu speed (usually it is close to idle)

Questions:

1. Will 10G network be maxed out with the 3 pool configurations?
2. onboard mini sas controller able to pass thru?
3. Are more suitable CPU/ Motherboard configs arround?
4. in case I go crazy with a mellanox 100GBE card and set iSCSI target to my desktop PC to boot directly from Truenas, what speed can I expect from a mirrored 2x2TB NVME?

open discussion highly appreciated!

 

[Mannekino]

Did you end up buying an X12 board? If so, what model did you pick?

 

[Whattteva]

Your setup is strangely similar to what I have (in sig). Slight difference in CPU (Silver 4210T), hypervisor (Proxmox), and VM's (OPNsense, TrueNAS CORE, FreeBSD 13.1-RELEASE).

I can confirm that it rocks.

1. I don't have a 10G switch to verify throughput unfortunately.
2. Did I miss this? I don't think the board has a mini SAS controller.

 

[WI_Hedgehog]

I've come to understand running VMs on a FreeNAS box is like using a small business's server as a workstation.
(that is, it exponentially increases the chances of something going wrong)

Am I wrong on this?

 

[Whattteva]

I've come to understand running VMs on a FreeNAS box is like using a small business's server as a workstation.
(that is, it exponentially increases the chances of something going wrong)

Am I wrong on this?

Hmm, yes and no, I suppose. There are other solutions better than TrueNAS for running VM's (UI for TrueNAS VM management is kinda' clunky).

That being said, TrueNAS is still a fine hypervisor if you don't need anything fancy like PCIe passthrough, in my opinion.
I myself ran a Windows VM on TrueNAS CORE for a couple of years and didn't notice anything wrong with it.

 

[John Doe]

I did not upgrade my system, still with

Xeon E5-1620v4 3,5GHz 2011-3
Supermicro X10SRA-F
4x 16GB Samsung DDR4-2133 reg. ECC Ram
LSI Megaraid 9207-8i
Seasonic X650 ATX PSU
6x 10tb WD red RaidZ2
2x Samsung Evo 860 1tb (Jails Mirrored)
ESXi 6.7 with FN 11.1, pfSense, Debian (nextcloud, plex), Windows 10
256gb SSD (boot)

but installed x540 T2 10gbit (towards ISP modem) and a mellanox connectx-2 (for LAN)

unfortunately there is a flaw, I cannot motivate the mellanox card to go up to 10 gbit with pfsense VM. That would require SR-IOV (in my setup with virtualized pfsense), since some feature flags when running freebsd in firewall mode are actually limit bandwith.

feature flag enabled -> iperf maxing out 10gbit but proper internet speed test is very low (mbit range)
feature flag disabled -> iperf about 3.7 gbit but proper internet speed test is higher (eventhou not above 7gbits)

this seems only applicable for freebsd use as firewall. no impact on truenas core with feature flag enabled.

I managed to flash the mellanox card to technically enable the SR-IOV feature and ESXi is indicating availability but I cannot assign it. -> stuck with reboot to enable, even after reboot.

lessons learned:
when using virtualized freebsd firewall use NIC with SR-IOV or have a setup with dedicated, PCI passedthru NIC to archive 10gbits


regarding my Q

Questions:

1. Will 10G network be maxed out with the 3 pool configurations?
2. onboard mini sas controller able to pass thru?
3. Are more suitable CPU/ Motherboard configs arround?
4. in case I go crazy with a mellanox 100GBE card and set iSCSI target to my desktop PC to boot directly from Truenas, what speed can I expect from a mirrored 2x2TB NVME?

1: I am quite sure the bottleneck is now my pool config (from Truenas to PC: 613MB/s From PC to Truenas: 219MB/s)
2: no answer
3: no answer
4: no answer