Virtualization and NAS needed - ESXi/TrueNAS/Proxmox/QEME/KVM - run what?

[trueJack]

Hi All,

Which one or combination of the following will give me the best of VM and NAS?
ESXi/TrueNAS/Proxmox/QEME/KVM

I wanted a NAS. I also want VMs. I am purchasing a Supermicro motherboard X10SRW-F with a Xeon processor and 2x16GB DIMM ECC. The motherboard has 10 SATA onboard split across 2 or 3 controllers. I am not sure which of these ports will actually be available, or whether I will be able to pass through these onboard controllers.

I will probably buy at least 1 SSD boot drive and 6 drives for storage.

Reliability is important, but I do not plan to put any data on this system that is not backed up elsewhere or that I can download again. So just to say, I do not mind a bit of risk or experimentation to get a better solution in the end.

Recommendation? What should I install on bare metal? And what should I install on top of that? Opinions?

Thank you,

J

 

[jgreco]

Well, best choice from an engineering standpoint is to run TrueNAS on the bare metal and then run VM's on that. This gives you the best NAS performance and least amount of unexpected virtualization complications. Core support bhyve based VM's and Scale supports KVM. The main downside is that the user interfaces are less complete than a dedicated hypervisor product.

Otherwise, you're stuck running the NAS under a hypervisor, for which ESXi is still the only one known to be reliable. Some people have used Proxmox successfully, but others have had problems. Either way, be sure to read

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[fleg]

Looking for the best one is neverending story.
I`v been using Proxmox over 10 years without critical problems (mostly up to 5 nodes in cluster with tens VMSs) at my customers and FreeNAS/TrueNAS as storages for backup, NAS and this is (for me) the best combination.
But I never tried FreeNAS/TrueNAS with VMs,.
I can recomend you using only SSDs if you can. HDD are obsolete:o).

 

[jgreco]

Looking for the best one is neverending story.
I`v been using Proxmox over 10 years without critical problems (mostly up to 5 nodes in cluster with tens VMSs) at my customers and FreeNAS/TrueNAS as storages for backup, NAS and this is (for me) the best combination.
But I never tried FreeNAS/TrueNAS with VMs,.
I can recomend you using only SSDs if you can. HDD are obsolete:o).

Well, that's nice, but Proxmox doesn't do filesharing, and the OP seems to be looking for a hyperconverged solution, not two separate systems.

I admire your wealth if you think HDD's are obsolete. I purchased a pool of 12 x 14TB's (168TB) for $2280 a year ago Black Friday. Today's pricing from Samsung on the 4TB Evo is $449, so an array of 42 4TB SSD's would be $18,858, or about a 900% price premium. Too rich for me.

 

[fleg]

Well, that's nice, but Proxmox doesn't do filesharing, and the OP seems to be looking for a hyperconverged solution, not two separate systems.

Proxmox can do filesharing and I like FreeNAS/TrueNAS because I`was using it a long time before I started using Proxmox. I have FreeNAS at home too e.g.
SSD
If you need to move a huge amount of data over network so you need a speed, not capacity. HDDs are bootle neck in the 10Gb networks or in 1Gb network if you are using swarm little files (so typical for CIFS or NFS).
Common today`s requirments for NAS are SSD array with 2 stripe nvme disks as cache.
I can get 5-6Gb/s from this NAS.

I agree with you if you prefer capacity over speed, but my priority is speed.

 

[jgreco]

Proxmox can do filesharing

(bushy eyebrows raise)

Could you please tell me where I enable SMB on Proxmox, 'cause I sure as hell don't see it. Could be that early onset senility though.

HDDs are bootle neck in the 10Gb networks

HDD's may be the bottleneck in YOUR 10GbE networks. That's a matter of design, not the use of HDD's. That's like saying that NVMe PCIe 3 SSD's are the bottleneck in a 100GbE network. You can make it true if you want, but you can also make it false.

 

[trueJack]

I would enjoy SSDs, but they are a bit expensive for what I have in mind. I may get two 1TB SSDs for stuff that needs speed, not sure. The rest will be on slow cheaper disks. This is for a small home server and I am cheap, so I would prefer a converged solution.

I thought about Proxmox and a simple Linux VM file-server on it, but I am not sure how reliable the Proxmox ZFS is. Now I am leaning towards Truenas Core with LXC / Bhyve.

Having said this, I do very much like ESXi, but I want ZFS too. If I make my VMs not auto-start, could I consider ESXi on a single disk with a TrueNAS VM on that disk, with pass-through controllers to that VM and then share storage for other VMs back to ESXi ? Seems .. finicky and weird, but I cannot see any obvious glaring logical errors or pitfalls or ways it would definitely not work.

 

[trueJack]

By the way, re-reading what I wrote I realize that it looks like Truenas Core has no LXC support. I would have edited the post, but I do not see an option.

 

[fleg]

Could you please tell me where I enable SMB on Proxmox, 'cause I sure as hell don't see it. Could be that early onset senility though.

Proxmox is Debian and Debian can run Samba;o).
Proxmox can create or support many disk`s type(storage): ZFS, Directory, LVM (thin, thick), NFS, CIFS... and many of them can be shared over network.
Try Datacenter-Storage-add-CIFS e.g.
You can use Proxmox Backup Server too.

HDD's may be the bottleneck in YOUR 10GbE networks.

Not may be but hey are. They are bottleneck in 1Gb network.
Make 2 identical VM. One machine runnig from HDD, one from SSD. Share the same contents with big and small files and you have to see and feel the differencies
I still have many HDDs in my servers, but every production server must be on SSD no matter if I have 1Gb or 10Gb/s.

 

[Patrick M. Hausen]

TN CORE has got VMs (bhyve) and containers (jails). Some think jails are superior to LXC, while admittedly bhyve laggs a bit behind other hypervisor options. When it works, it proved to be stable and fast for us. Coming late to the party the team just decided to skip many legacy hardware emulation options and features like MBR/BIOS boot ...

 

[neb50]

I have been using FreeNAS and now TrueNAS Core on top of ESXi 6.7 for the last few years as a hyper converged server. I used a couple of guides off here along with Spearfoot's scripts and guides. May not be the best solution for businesses but works for home use.

 

[jgreco]

Proxmox is Debian and Debian can run Samba;o).

So in the same way that ESXi can do filesharing, i.e. if you hack on the appliance. It's not a feature of the appliance. By your logic, one could argue that any computer can do anything. Proxmox can directly run your firewall, TrueNAS can directly handle your DHCP and NAT needs, etc.

Not may be but hey are. They are bottleneck in 1Gb network.
Make 2 identical VM. One machine runnig from HDD, one from SSD. Share the same contents with big and small files and you have to see and feel the differencies

I think you have to carefully qualify that statement in so many ways that it is useless.

Back in the 2000's, when 10Gig was new and 4U 24 drive storage servers were just first being built by companies like mine, we absolutely had customers with 10Gig networks who were hitting the limit. The 10G limit. Those familiar with the history of USENET might recognize Newshosting and Highwinds from those days, and that was back when HDD speeds were rather lower than the peak 250MBytes/sec you can get today.

If your 1G network is bottlenecked by a HDD, engineer better.

From the trenches, - JG

 

[trueJack]

I have been using FreeNAS and now TrueNAS Core on top of ESXi 6.7 for the last few years as a hyper converged server. I used a couple of guides off here along with Spearfoot's scripts and guides. May not be the best solution for businesses but works for home use.

Thank you. I may try this as one of my first options. I am not afraid of a bit of tinkering. I do want to prevent data loss. I do not think ESXi + TrueNAS VM would have any inherent risks to the data and I really enjoy ESX.

 

[trueJack]

TN CORE has got VMs (bhyve) and containers (jails). Some think jails are superior to LXC, while admittedly bhyve laggs a bit behind other hypervisor options. When it works, it proved to be stable and fast for us. Coming late to the party the team just decided to skip many legacy hardware emulation options and features like MBR/BIOS boot ...

Thanks! Jails are like Linux chroots right? I have used chroot before and I found it very useful.

 

[trueJack]

Proxmox can do filesharing and I like FreeNAS/TrueNAS because I`was using it a long time before I started using Proxmox. I have FreeNAS at home too e.g.
SSD
If you need to move a huge amount of data over network so you need a speed, not capacity. HDDs are bootle neck in the 10Gb networks or in 1Gb network if you are using swarm little files (so typical for CIFS or NFS).
Common today`s requirments for NAS are SSD array with 2 stripe nvme disks as cache.
I can get 5-6Gb/s from this NAS.

I agree with you if you prefer capacity over speed, but my priority is speed.

I am not up to 2.5/10Gbps in my home network yet, but it is something worth considering for the future (just for fun and tinkering).

I am interested in using ZFS caching. Why 2 striped NVMEs ? Would one NVME/SSD not do quite well already? Does ZFS cache have to be redundant?

 

[fleg]

Why 2 striped NVMEs ?

Sorry for misunderstanding...there is HA machine so because 2xnvme (2 controllers in one case).

 

[jgreco]

Thanks! Jails are like Linux chroots right? I have used chroot before and I found it very useful.

No, chroot is only one part of it. You also get your own IP (or potentially even a virtual IP stack), a restricted process view, etc.