Running FreeNAS as a VM (off USB or SSD)

[Johev]

Dear Community,

It's been a while since I've posted, however before I start my new project I would like to have some feedback on what I'm about to do.

My System is:
Supermicro X10SL7-F
E3-1265L v3
32 GB ECC RAM

I would like to run FreeNas as a VM passing-through the LSI 2308 to it. On the LSI I would have all 8 SATA ports populated with 4TB NAS HDDs.

My question is if I could run FreeNas of an SSD or USB, which would not be passed to FreeNas?

The other OSes are:
Windows 10
Tvheadend with one PCI passed-though for satellite reception card.

The Hypervisor will be ESXI, which I read is fully supported by FreeNAS.

Many thanks in advance for your input.

 

[melloa]

I would like to run FreeNas as a VM

My solution.

My question is if I could run FreeNas of an SSD or USB

As a VM inside your datastore would be my suggestion.

Tvheadend with one PCI passed-though for satellite reception card.

c00l!

The Hypervisor will be ESXI, which I read is fully supported by FreeNAS.

See my signature for my system. It is also a SM, running FN on a VM, along with several other VMs. Passing through two HBAs for it, one for DSM, and one for the "LAB".

Second server has similar approach with FreeBSD as the VM controlling my volume.

 

[kdragon75]

LSI 2308

That's in IT mode right? I have to ask.

My question is if I could run FreeNas of an SSD or USB, which would not be passed to FreeNas?

USB, Only as another passthrough device but you still need somewhere (a datastore) to save the VM files. SSD, would have to be formatted as a VMFS datastore, from here the VM files (.vmx, .nvram .vmdk etc) would be saved there. This includes the FreeNAS boot disk. @Stux has done a build like this and mirrored the boot drive (vmdk) with passthrough USB drive so he could boot the FreeNAS as a bare metal server if needed. Quite clever.

The Hypervisor will be ESXI, which I read is fully supported by FreeNAS.

Nope. It works and can work well but its not "supported". ESXi fully supports FreeNAS (FreeBSD) as a guest but that's about it. While you can share the FreeNAS storage back to ESXi and it works well it is also not "supported".
Dont forget that FreeNAS REQUIRES 8GB of RAM and that PCI passthrough requires a full memory reservation so you will only have at most 21GB of RAM for your VMs (32 - 8 for FreeNAS - 1GB for overhead and ESXi). This is just an estimate.

 

[kdragon75]

If your VMs are al windows, you can save a fair amount of RAM with transparent page sharing. Look into enabling TPS across all VMs And be sure to install VMware Tools!

 

[kdragon75]

Just looked up your CPU, that not a lot of cores for ESXi, FreeNAS, and two Windows VMS. Start with a single vCPU for all of your VMs including FreeNAS. Monitor performance and add vCPS ONLY if you can prove that a single core is the bottleneck. DO NOT give any VM 4 cores. with virtualization and virtual CPUs, less is more. Just try giving them all 4 vCPUs and look at your CPU ready numbers on ESXi!

 

[Johev]

First of all I would like to thank you all for your input. I only need 3 VM's currently. Freenas which will have about 16GB of RAM assigned to it, Windows with 8GB (or 4GB) and the rest will go to TVheadend (not really sure how RAM hungry that VM will be).

@kdragon75 The LSI is already flashed with IT firmware (not sure if it's still the v16, so if you know which version is the best, please let me know). :)

So back to my question:
Would installing FreeNAS on a SSD connected to a SATA port which is not part of the LSI, and therefore has not passed-through to VM FreeNAS, be possible? Would it be risky?

Many thanks once again.

 

[kdragon75]

Would installing FreeNAS on a SSD connected to a SATA port which is not part of the LSI, and therefore has not passed-through to VM FreeNAS, be possible? Would it be risky?

YOu would have to format the SSD as a datastore for ESXi. It's not that risky but as always, make backups.

 

[joeschmuck]

Would installing FreeNAS on a SSD connected to a SATA port which is not part of the LSI, and therefore has not passed-through to VM FreeNAS, be possible? Would it be risky?

How large is the SSD? My system uses the SSD as the ESXi boot device and also holds several VMs such as FreeNAS. This is perfectly fine and works great. If you really wanted to pass though the SSD to the FreeNAS VM, well it could be done via RDM but I'd advice against it if this were the only purpose. Also using a SSD as the ESXi boot device really helps speed up software updates in which a USB Flash drive installed ESXi can take a long time to perform updates.

Tvheadend with one PCI passed-though for satellite reception card.

I'd like to hear how that goes as well. Not that I'd do it myself but it's good to have knowledge of a pass or fail so we can guide people in the future.

 

[Johev]

How large is the SSD? My system uses the SSD as the ESXi boot device and also holds several VMs such as FreeNAS. This is perfectly fine and works great. If you really wanted to pass though the SSD to the FreeNAS VM, well it could be done via RDM but I'd advice against it if this were the only purpose. Also using a SSD as the ESXi boot device really helps speed up software updates in which a USB Flash drive installed ESXi can take a long time to perform updates.

I have two 480GB Intel S3500 and I was thinking of either running ESXi off a USB from the internal motherboard port or running everything of the SSD. I bought them because of the "power-loss data protection" and because they were on sale :).

How would you advise to proceed with using the SSD? Should I partition it (one for ESXI), another for FreeNas and a third for any other OS? If yes, how large should I create the ESXI and FreeNAS partitions? I know that FreeNAS needs at least 8GB, however I would not want to cut it too close.

I'd like to hear how that goes as well. Not that I'd do it myself but it's good to have knowledge of a pass or fail so we can guide people in the future.

I'm using a Digital Devices Cine S2 V7A Sat receiver card and hopefully, as soon as I'm able to pass-it-through to the Tvheadend VM, it should replace my Vu+ DUO2 enigma receiver. This way I would be able to use any Kodi device or smart TV (with some kind of IP TV software) to watch my satellite channels directly. Or at least that's the dream :)

 

[kdragon75]

How would you advise to proceed with using the SSD? Should I partition it (one for ESXI), another for FreeNas and a third for any other OS?

That's not how esxi works.
https://pubs.vmware.com/vsphere-50/...ware.wssdk.pg.doc_50/PG_Ch8_Storage.10.3.html
Your SSDs would be considered DAS SCSI storage. ESXi is WAY more complex than most people here give it credit for.

 

[kdragon75]

When setting up the tvheadend VM, please read http://docs.tvheadend.org/basic_requirements/#system-requirements and follow the minimum numbers. Even if transcoding, start with one vCPU. The same for the Windows VM. Only add a second if you need it and the CPU ready is low.

 

[joeschmuck]

How would you advise to proceed with using the SSD? Should I partition it (one for ESXI), another for FreeNas and a third for any other OS? If yes, how large should I create the ESXI and FreeNAS partitions? I know that FreeNAS needs at least 8GB, however I would not want to cut it too close.

Please don't take this the wrong way but you are asking some pretty basic questions in which I feel you should be able to find on your own. I highly recommend that you read some ESXi docs and even visit the VMWare forums and read those. Keep in mind that this is a FreeNAS forum. I have a very long thread in the off topic section where several of us discussed ESXi and the folks here were very helpful to me however I also did a lot of reading on the VMware forums site and just general ESXi internet searches. And I understand that it's difficult to know what to read when you don't know what to read or ask.

I would highly recommend that you just build an ESXi machine and then create several VMs and given that you are new to this, it will likely take you several months to get a good feel for it if you are working on it for a few days a week. Yo will have a lot of questions on why something doesn't work, the VMWare forums will help. Create a Windoze 9or you favorite OS) VM, destroy it, copy your vmdk file(s) and import a previously created VM. Work on setting up your VM Network Swithes which can take a while to understand what is happening. Also figure out how to update VMWare ESXi software to the next version/bug fixes. Do not take on ESXi and FreeNAS at the same time, it could drive you crazy. Most importantly, do not commit data to your system, meaning do not store data on your system and expect to never delete it. This step makes it difficult to play with the system. In FreeNAS you should also see how much fun it is to replace a failing hard drive so you can swap out a drive to see how resilvering works. Destroy FreeNAS pools and recreate them. You may not like the first pool configuration.

So I hope my comments are well recieved, I really am trying to help you and part of that is to just get out there and read some docs. @kdragon75 gave you a great publication reference.

 

[Johev]

@joeschmuck Thank you for your input. I do not mind at all honest and constructive feedback to my questions, and fortunately that's what I've mostly got whenever I came to this forum for assistance.

As you have pointed, my experience with ESXI is very limited, as some of my questions clearly show. This said my experience with FreeNAS is now almost 2 years of 24/7 running. My machine has been very stable (never froze, just had to reboot for updates) and that's one of the reasons I prefer to turn it into a VM by running ESXI, instead of going with an unraid solution.

I have basically read all the official posts on using FreeNAS in a VM environment and that the only way to do it reliably is to give FreeNAS a bare bone access to the discs by passing through the LSI in IT mode. Therefore even if I screw up something with ESXI, I should still be able to boot up FreeNAS with a separate USB and have it fully functioning with all my data unattached.

I hoped that with FreeNAS 11 versions I would be able to passthrough some hardware and like that completely avoid the need for ESXI, however unfortunately that's still not the case.

I will however take all the advice given, and first build a working Windows and Linux VMs under ESXI, configure the networking and only when that's working will I go and create a FreeNAS VM and port all my data to it. :)

 

[joeschmuck]

I have basically read all the official posts on using FreeNAS in a VM environment and that the only way to do it reliably is to give FreeNAS a bare bone access to the discs by passing through the LSI in IT mode.

This was the original view point for FreeNAS and I think we are at a stage where ESXi can very reliably run FreeNAS. The main thing here is managing ESXi properly, proper configuration, and proper hardware.

I hoped that with FreeNAS 11 versions I would be able to pass-through some hardware and like that completely avoid the need for ESXI, however unfortunately that's still not the case.

My personal opinion is to never use FreeNAS as a hypervisor. I think it is fine for some small applications but when you start talking about full blown operating systems or other fairly advanced setups, keep in mind that bhyve is still fairly new to the hypervisor game.

I will however take all the advice given, and first build a working Windows and Linux VMs under ESXI, configure the networking and only when that's working will I go and create a FreeNAS VM and port all my data to it. :)

As for porting your data to it... If you already have a pool then you could just use it as-is. But if you are like many of us, when we are given an opportunity to rebuild our pools, quite often we do. So in ESXi you have two options to pass your hard drives, the best practice it to pass the entire hard drive controller through but sometimes that is not an option with your specific hardware and you are forced to use RDM to pass though individual hard drives. I use both methods actually and they both work well, RDM just requires a little more setup. And if you desire you can pull those hard drives and stick them in another machine, boot up FreeNAS on a USB Flash drive and have all your data available. The only sticky part is you would need to reconfigure your NIC for FreeNAS, otherwise it's a piece of cake.

Again, play with ESXi and a few VMs to get the feel of it before moving your FreeNAS setup into it. Feel free to ask questions, we don't mind answering them but if they are ESXi specific then maybe an ESXi forum would be better for you.

Hey, don't forget about and UPS and having it configured to automatically shutdown your VM in the proper order and then turn off the computer. It's not as straight forward as we all would like but you should still strive to get it operational.

I try to provide constructive feedback and I really hope I have. We don't need people that just make comments and don't provide solutions or recommendations.

Best of luck to you.

 

[Johev]

This was the original view point for FreeNAS and I think we are at a stage where ESXi can very reliably run FreeNAS. The main thing here is managing ESXi properly, proper configuration, and proper hardware.

Good to know, just a year and a half ago the feedback from this forum would be filled with purists, that would preach that FreeNAS could only be used as a stand alone box.

My personal opinion is to never use FreeNAS as a hypervisor. I think it is fine for some small applications but when you start talking about full blown operating systems or other fairly advanced setups, keep in mind that bhyve is still fairly new to the hypervisor game.

As I said, that would have been the dream. It's like people who are able to use Proxmox with ZFS and therefore avoid the need to run FreeNAS. I know that FreeNAS is much more than just the above solution, however it's just to make a point.

As for porting your data to it... If you already have a pool then you could just use it as-is. But if you are like many of us, when we are given an opportunity to rebuild our pools, quite often we do. So in ESXi you have two options to pass your hard drives, the best practice it to pass the entire hard drive controller through but sometimes that is not an option with your specific hardware and you are forced to use RDM to pass though individual hard drives. I use both methods actually and they both work well, RDM just requires a little more setup. And if you desire you can pull those hard drives and stick them in another machine, boot up FreeNAS on a USB Flash drive and have all your data available. The only sticky part is you would need to reconfigure your NIC for FreeNAS, otherwise it's a piece of cake.

The main reason why I'll have to migrate the data is that currently I have 3 RAIDZ 4TB HDDs, and now I bought further discs and I want to reuse the discs in a different setup (7 or 8 discs in RAIDZ2).

Hey, don't forget about and UPS and having it configured to automatically shutdown your VM in the proper order and then turn off the computer. It's not as straight forward as we all would like but you should still strive to get it operational.

That's my next purchase. I got Intel SSDs with power protection in order to mitigate any damage due to a power failure, however an UPS would be a much better solution.

I try to provide constructive feedback and I really hope I have. We don't need people that just make comments and don't provide solutions or recommendations. Best of luck to you.

What can I say @joeschmuck, your constructive feedback is very useful. I didn't even know about RDM, I thought that I could only be able to use the ports managed by the LSI 2308 on my Motherboard. :)

 

[joeschmuck]

I didn't even know about RDM

RDM requires a little more work to configure but it's an option if you need it. Most folks here, even the ones who would support FreeNAS on ESXi will still frown on using it but for me as long as you are aware there more to configure and it's not really straight forward, then I don't see an issue with it. I'm not the only one using RDM, there are a few of us here using it.