New build as FreeNAS ESXi guest

kmklv

Cadet
Joined
Sep 16, 2019
Messages
9
Hello, I wanted to ask for opinions related to optimizing my build. When I started ordering parts, I was mainly thinking as FreeNAS on bare metal but after realizing there's no way within FreeNAS to make blu-ray backups, and not being sure about other possible shortcomings (such as running nodered, mosquitto etc.) I thought it could be a better idea to do this under ESXi. I'm aware that this isn't a recommended build, but after reading some of the posts by @jgreco, I figured it may not be a bad idea as long as it's done right.

First the hardware:

Motherboard: Supermicro X11SCA-F
Processor: Intel Xeon E2176G
Memory: 32 gb Supermicro tested hynix
Network: Chelsio T520-SO-CR

Storage:
2 x 8tb WD-Red
2 x 512gb Samsung 860 EVO SSD (intended for other vm/jail etc. install)
2 x Crucial BX 256gb SSD (originally intended for FreeNAS O/S Install)
1 x Intel Optane 16GB (intended as SLOG)
1 x Intel 256 m2 SSD

And to be able to pass the SATA controller to the ESXi host, I ordered this card: SAS9211-8I 8PORT

I have two main questions:

1- What would be the best way to utilize the O/S install space? I intended for Intel Optane to be used as SLOG, but if I'm unable to do a PCI passthrough on it, then will it not be utilized properly under FreeNAS? Or is that too small of a space for SLOG? I've read through forums, but many similar subjects seem to be pretty outdated.

2- If this is configured properly, and the storage drives are accessible without ESXi through methods mentioned by @jgreco on sticky threads, any other reasons that I need to be cautious of to avoid virtual FreeNAS? One of my main issues is lack of bluray burning support in FreeNAS and therefore me wanting to run it side by side with something like Debian or CentOS. But I haven't seen any threads about virtualization with HBA passthrough issues that are within the last couple years, so if I can check community's pulse on this subject, I would really appreciate it.

Or, if there's a new way to burn blurays within FreeNAS through use of jails or virtualbox, please mention that and I could probably avoid ESXi. Thank you for reading this through!
 

joeinaz

Contributor
Joined
Mar 17, 2016
Messages
188
I have successfully run FreeNAS under ESXi to backup to my LTO tape. The passthrough worked fine. My next effort will be to try to do the same thing with the virtualization within FreeNAS. I have a Bluray disk so I might see if that can passthrough as well..
 

kmklv

Cadet
Joined
Sep 16, 2019
Messages
9
Thanks joeinaz, forums seem to make a big deal about not running FreeNAS as VM and I really wanted to get some opinions from a technical perspective what the big deal is, as long as HBA passthrough is working and FreeNAS can access drives directly.
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,974
forums seem to make a big deal about not running FreeNAS as VM

It's not a big deal if you know what you are doing when it comes to managing a hypervisor such as esxi. It's just not recommended for the average user.
 

joeinaz

Contributor
Joined
Mar 17, 2016
Messages
188
"It's not a big deal if you know what you are doing when it comes to managing a hypervisor such as esxi." Well said.

I am certified in VMware (VCP) and it was a challenge for me to get everything just right. I would never recommend this solution as a first time build. Ideally, you should be comfortable in both VMware and FreeNAS. Also VMware raises the hardware requirements in the area of CPU and memory especially if you intend to run lots of VMs.

If you still want to set up FreeNAS under ESXi, the forum is where you should turn for help....
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Thanks joeinaz, forums seem to make a big deal about not running FreeNAS as VM and I really wanted to get some opinions from a technical perspective what the big deal is, as long as HBA passthrough is working and FreeNAS can access drives directly.

You're building a house of cards (ZFS) inside another house of cards (ESXi). It *can* be done if your hardware/software/firmware are 100% (which you need to validate), and you are sufficiently familiar with ESXi virtualization. If you are the kind of person who throws it all together and starts loading data on the thing the next day, it's not really advisable. By way of comparison, we build stuff here professionally and will still test for a minimum of a month after burn-in, because some of the potential flaws do not cause instant purple screen.

At one point I helped one of the other moderators try to build a hyperconverged platform on a prosumer-brand board that looked like it should have been fine for the purpose, but it kept crapping out every day or two. It just reinforces that this stuff isn't easy even if you think you know what you're doing, and having been doing this longer than probably anyone else in the forums, it's probably wise to take the advice I offer: "proceed cautiously, expect failure is possible, test, test, test, test."
 

kmklv

Cadet
Joined
Sep 16, 2019
Messages
9
Thanks jgreco, I'm ok with failure, except for total data loss. I learned RDM is not the way to go reading some of the prior posts, but I've been unable to find any horror stories related to ESXi pci passthrough causing similar data loss. Any horror stories I should be reading on forums that I can learn from, or at least be aware of? I'm assuming a worst case scenario of 'things don't work well, or keep crashing, I use a freenas USB stick and rescue my data'. Is that too optimistic?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Please refer to

https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

You are basically more-or-less right that you need a backup plan that allows you to use a USB stick to recover.

Note that this article was written in ~2013. In general, but do not rely on this, my opinion is that a lot of the boards that predated Sandy Bridge were total PCIe passthru train wrecks. I've heard of some problems with Supermicro X9 but not a lot, and for the infrequent time I've heard of X10 or X11 series issues, it seems to be user error rather than hardware/firmware/software issues. So recent Supermicro gear is expected to be (at least capable of) non-drama.

Regardless, you are setting up a house of cards inside another house of cards. Check your recovery option actually works as you expect.

I've been running FreeNAS virtualized since ESXi 4.1 and the first iX releases of FreeNAS. This does not mean it's been without some drama or occasional issues. I wrote the various virtualization stickies not because it was _IM_possible, but because we were seeing so many people appear on the forums, some of whom wanted to do this on incompatible platforms, some of whom had never seen an ESXi install before, etc. People were losing data, people were doing dangerous things, etc.

For heaven's sake listen to the guy who said:

I am certified in VMware (VCP) and it was a challenge for me to get everything just right. I would never recommend this solution as a first time build. Ideally, you should be comfortable in both VMware and FreeNAS.

This is spot on. When you can read through my stickies and say "oh, yeah, that makes sense" and "oh but I know better than that" and all that, you are probably suitably positioned to be successful. If you are using this as a learning experience and you DON'T know all that stuff, be aware that you can shipwreck and have a bad experience. If you are doing this for fun and losing your data is all right, THAT'S FINE TOO. I enthusiastically support hands-on education, it's the best way to learn, and numerous people here have been through that process and will hopefully appear to answer questions.
 
Top