FreeNAS on VMWare. Yep.

[lukeren]

I've been running FreeNAS for quite a while, but recently upgraded my hardware to a SuperMicro X9DAi with an E5-2650 and 52GB RAM (it's a mix of sticks I'm looking to sort out).
Since I'm basically the only user of the system, it seemed a bit of a waste to dedicate all that to FreeNAS. I'd like to run a few VMs too, but the bhyve hypervisor wasn't really doing it for me.
I read the following posts before starting out and decided to go ahead despite all the warnings ;)

http://www.freenas.org/blog/yes-you-can-virtualize-freenas/
https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/
https://forums.freenas.org/index.php?threads/my-dream-system-i-think.41072/ (this one was a beast, but @joeschmuck had a lot of good info there)

I created a VM with 8 vCPUs, 24GB RAM and a 10GB disk. Installed FreeNAS and checked it was booting with no issues.
Then I passed the HBA through and checked dmesg to see if it was actually working, it was.

At that point all that remained was to actually connect the drives, I did and imported my config from a backup.
After a few reboots my system was running again, but my jails didn't get any network connectivity. @jgreco had the answer in this thread though.
It didn't work for me after enabling "Allow forged transmits", I had to also enable "Allow promiscuous mode". This may be down to me being impatient, I don't know how long it takes for the changes to apply.

What I noticed after this all was set up, was that it appeared the server didn't draw as much power under ESXi as it did under FreeNAS. It may just be me seeing things though.

I'd like to disable the scrubs on the boot volume, since it's not directly connected to the VM, should I do that? Also, how do I do that? :)

 

[kdragon75]

What I noticed after this all was set up, was that it appeared the server didn't draw as much power under ESXi as it did under FreeNAS

FreeNAS is definitely not tuned for low power consumption. There is a lot you can do to fix that and I should but I'm lazy...

As for the boot scrubs, I have not seen a setting anywhere.

 

[joeschmuck]

Yes, I too found that FreeNAS is not tuned for power savings, ESXi is a great product and I suspect most of the power savings is due to how ESXi alocates the CPU rescourses to the VM.

I created a VM with 8 vCPUs, 24GB RAM and a 10GB disk.

You could probably drop the vCPUs to 3 or 4 and be just fine. If you plan to run VMs on FreeNAS other than simple jails such as Plex then I'd try to steer you to use ESXi first if possible. If you really like FreeBSD then run a FreeBSD VM under ESXi. If you have some really specific reason to run a VM inside FreeNAS then you could always increase your vCPUs if needed.

I'd like to disable the scrubs on the boot volume, since it's not directly connected to the VM, should I do that?

I would not disable scrubs on the boot volume. The purpose of the scrub is to verify the data integrity and in ESXi you have a 10GB single .vmdk file that acts as the entire disk. I'd could make an argument that it is more critical to run a scrub becasue it's now a very large single file, of course I could also argue that if you maintain a copy of your configuration file then you can restore the boot drive super easy, like you did when you created your FreeNAS VM. I don't know how to disable it, I wouldn't think it would be too difficult if you searched for it but if it's a hack then I wouldn't do it. The scrub should be no issue if the vmdk file is on a spinning rust hard drive and only a very minor wear issue if on a SSD.

this one was a beast, but @joeschmuck had a lot of good info there

Yea, it is a bit long but there is a lot of info and we were all giving each other advice and we would get off topic a few times (to say the least), I was recieving the information very well. My system has changed I think 2 times since that thread completed and I'm set for a few more years, I hope.

If you have any questions, even though I'm not online as much these days, please ask. If I can offer you some advice or just answer some questions then I'm more than happy to do so.

And there were a lot of warnings against putting FreeNAS on a VM and there was a very good reason, being diciplined enough to start up the VMs and shut them down properly and use pass-through for your storage media. Simple things like this will trip up a person and then the data is at risk. So lets say you are now using ESXi and like it. If you create a FreeNAS VM and share data, then you create a Windows VM and you have one of the FreeNAS shares mounted, well you need to power on and ensure the FreeNAS VM is running before the Windows VM is running, and shutdown the Windows VM before shutting down the FreeNAS VM. Let's make it more complicated, now you have an iSCSI share in FreeNAS of say 1TB, now you create a new ESXi VM on that iSCSI share. You MUST shut down that new VM that resides on the iSCSI share or it's possible to corrupt the iSCSI share data if FreeNAS powers down first.

So my point here is to ensure you manage your machines properly and ESXi does support power on and off features and time delays. Use them! Trust me, it will bite you when you least expect it. And if you have a FreNAS software upgrade, expect a reboot and thus your iSCSI share to go away for a while. Your other VMs which rely on it may suffer.

I hope I didn't say anything you didn't already know but if I did, just be aware that it's easy to inflict harm by accident.

 

[lukeren]

I might be able to drop the amount of vCPU's, but I'm running a plex jail, so I might need them for transcoding.
In time, I plan to move plex to its own Linux VM, since the updates on FreeBSD isn't as regular as I'd like.

Regarding scrubs, the blogpost on freenas.org said to disable scrubs if the disk isn't connected to a controller that is passed through, so that's why I asked. I'll leave it as is though, I'm running nightly backups of the config anyway and it's a breeze to get it back up and running.

You are making some good points regarding iSCSI and sharing between VMs, which are very good to have written down and probably mention every time they're relevant. I believe my discipline regarding that is OK though, I hope ;)
If not, it'll bite me in the rear and I might have learned a lesson. It's not much else than a lab/playground. If it dies, it'll suck, but I won't be that bothered about it.

 

[joeschmuck]

I might be able to drop the amount of vCPU's, but I'm running a plex jail, so I might need them for transcoding.

I thought there was some study done and it was determined that three cpus were more than enough, that was before bhyve. Too manty things are single threaded so having more cores just doesn't help. But it's only a suggestion to drop your number of vCPUs, you don't need to and it will not do any harm. I just like to minimize the resourses where practical.

Regarding scrubs, the blogpost on freenas.org said to disable scrubs if the disk isn't connected to a controller that is passed through, so that's why I asked.

Actually I'm sure you did read that but also it is not recommended to run FreeNAS using drive that are not pass-through. I wouldn't be surprised if your reference was mainly conserned about your data drives but in my current opinion, I would not disable a scrub on your boot device, even if it is a vmdk file.

I believe my discipline regarding that is OK though, I hope ;)

Yea, this can get the best of us. I can't tell you how many times I dropped my VMs just becasue I did somethign to the FreeNAS VM. I've been fortunate to not have lost data but I'm doing much better these days however I'm sure screw up again, just give me 6 months.

 

[lukeren]

It depends on the content. The plex transcoder loves cores, it multithreads very well. When I move it to a dedicated VM, I'll probably scale FreeNAS back to 2 cores, it really isn't doing much work.
I totally agree about the dedication of resources. If it's not needed, it shouldn't be there.

 

[Stux]

You might find this handy
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[kdragon75]

I plan to move plex to its own Linux VM, since the updates on FreeBSD isn't as regular as I'd like.

Just build a jail from scratch and use pkg install plexmediaserver just don't quote me on the package name

The plex transcoder loves cores, it multithreads very well.

Still worth testing. If you plan run more VMs than cores, its essential to keep per VM core count to a minimum to do the job needed. It's not as bad as the 4.1 days but it will still kill performance even if the cores are idle most of the time.

Just want to clarify that last point. Any time you have more vCPUs TOTAL than physical core count, its vital to keep your per VM core count to the minimum needed to do the job.

 

[danb35]

use pkg install plexmediaserver

...which still doesn't update that frequently--installing via ports will get you a more recent version. But even so, the FreeBSD version of Plex is missing features that are available on other OSs. If OP's working in a virtualized environment anyway (as he is), I'd definitely agree with putting Plex in a Linux VM rather than in a jail.

 

[kdragon75]

...which still doesn't update that frequently--installing via ports will get you a more recent version. But even so, the FreeBSD version of Plex is missing features that are available on other OSs. If OP's working in a virtualized environment anyway (as he is), I'd definitely agree with putting Plex in a Linux VM rather than in a jail.

Yeah I cant argue too much with that. I just like the low overhead of jails ;)

 

[lukeren]

You might find this handy
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Very much so! I love the level of detail. I'm not going to need half of what you've set up, but since there's a very detailed explanation of it all, I might tinker with it anyway :)

...which still doesn't update that frequently--installing via ports will get you a more recent version. But even so, the FreeBSD version of Plex is missing features that are available on other OSs. If OP's working in a virtualized environment anyway (as he is), I'd definitely agree with putting Plex in a Linux VM rather than in a jail.

Exactly. My plex has a perpetual banner at the top that says "update available". It's a small thing, but it annoys me :P

Yeah I can't argue too much with that. I just like the low overhead of jails ;)

As do I! It's awesome, but I've found that putting those tasks I've got running in jails into Linux VMs apparently gets the job done with less powerdraw. Go figure, but unless my meter knows when it's running in a jail or not, that's my take away.