SOLVED FreeNAS as a VM?

[simon_lefisch]

Hello everyone,

New to the forum and FreeNAS, but have been running Linux (CentOS) for a couple years now and loving it. Will actually be switching to Debain Stable due to new packages being available (ie; newer KVM features), but that's off-topic. Anyway, I have some questions about setting up FreeNAS as a VM.

I did find 2 threads about virtualizing FreeNAS (thread 1 and thread 2). The first says not to do it in a production environment while the other says it would be ok for home use, which is what I will be doing. I wanted to get peoples opinions about setting it up as both a file server (for transferring files between PCs and serving as storage for a Plex server VM) and a Time Machine backup server for my home. Obviously having dedicated hardware would be best, however I already have my main server tower and a video encoding machine that run for encoding my movies to Plex, so not having anymore visible hardware would be preferable. Can anyone give me some tips for setting up FreeNAS in KVM on my CentOS 7 box? I do realize that the qemu-kvm version in CentOS 7 is older (v1.5.3 if I'm not mistaken) which is why I am looking into switching to Debian 10 stable...newer version of KVM (v2/v3) with pass-through feature if I recall correctly. And from the reading of those 2 threads, it sounds like that's what needs to be done for FreeNAS to work properly/efficiently. Now, I currently have a VM of FreeNAS running and I set it up as a Time Machine backup server for testing purposes....I wanted to see how well it worked and that I was able to restore file/etc. So far it is working great and I would love to implement it....which is why I am here. So please, any info or insight anyone can provide would be great so I can make the best decision of which route to take....VM or hardware. Thank you for your time and help

 

[zeebee]

There's also this if you haven't read it already. https://www.ixsystems.com/blog/yes-you-can-virtualize-freenas/

Like a number of people here, I run FreeNAS in a VM under ESXi, and I freakin love it. I only did this build six months ago, but both FreeNAS and ESXi have been rock solid in that time. FreeNAS is only used for providing storage (via SMB shares) and cloud backup (via Backblaze B2). Everything else I do in Ubuntu Server, Windows Server and Windows 10 VMs.

This was my first FreeNAS build, and I followed a lot of advice I found on these forums. After doing about two months of testing and burn in, I setup FreeNAS bare-metal and created my pool, ensured it was all working, and then installed ESXi and passed the HBA through to FreeNAS. This way I know I can just boot directly to FreeNAS should anything go wrong with ESXi. My FreeNAS config is backed up regularly, so the VM itself is pretty much throw-away.

I'd highly recommend virtualizing. Let FreeNAS do just what it's really good at (storing data), and do the rest somewhere else.

 

[simon_lefisch]

There's also this if you haven't read it already. https://www.ixsystems.com/blog/yes-you-can-virtualize-freenas/

Like a number of people here, I run FreeNAS in a VM under ESXi, and I freakin love it. I only did this build six months ago, but both FreeNAS and ESXi have been rock solid in that time. FreeNAS is only used for providing storage (via SMB shares) and cloud backup (via Backblaze B2). Everything else I do in Ubuntu Server, Windows Server and Windows 10 VMs.

This was my first FreeNAS build, and I followed a lot of advice I found on these forums. After doing about two months of testing and burn in, I setup FreeNAS bare-metal and created my pool, ensured it was all working, and then installed ESXi and passed the HBA through to FreeNAS. This way I know I can just boot directly to FreeNAS should anything go wrong with ESXi. My FreeNAS config is backed up regularly, so the VM itself is pretty much throw-away.

I'd highly recommend virtualizing. Let FreeNAS do just what it's really good at (storing data), and do the rest somewhere else.

Thanks for the reply @zeebee, and for the link. I'll def check it out. I assume you mean you installed ESXi then FreeNAS (post sounds like it was the other way around)? What about using Proxmox?

 

[Pitfrr]

I'm also running FreeNAS as a VM in ESXi. It's been working for some years now (something like three at least) and I'm very happy with this (as "home use").
An other post I can advise for more reading...

I chose ESXi since it's the recommended hypervisor (recommended as "more used in the community here and therefore more resources are available").
I was interested in proxmox (being open source) but I didn't dare to try it out! :p
And I didn't consider KVM because... I don't recall... (not sure I checked it out at that time.)
Also, I didn't have any experience in virtualization prior to start this.

As said I'm very happy with the results but I also make sure my backups are in place. :p It's not that I don't trust my setup, just that when you virtualize, you add complexity and therefore more things can go wrong quickly...

You have to make sure FreeNAS has direct access to the disks (i.e. passthrough), definitely!

 

[zeebee]

I assume you mean you installed ESXi then FreeNAS (post sounds like it was the other way around)?

No I did mean that I installed FreeNAS before ESXi. I believe this was recommended in one of the threads linked above. Basically, install FreeNAS directly on a spare drive (or usb stick) first, and do a bunch of burn-in testing etc. Setup your pools, make sure everything works like you'd expect in the "direct to metal" setup. If you have any problems with your ram or drives etc, hopefully you find them at this stage. Once you're happy with it all, then you install ESXi on whatever drive you're really going to use. You then create a FreeNAS VM and pass-through your HBA. This means FreeNAS sees the HBA (and your drives) just as it did when you were booting directly - that is, it has direct access to the hardware. If ESXi ever blows up for whatever reason, you can just plug in your original FreeNAS drive (or do a new install) and boot directly to get access to your pools.

This was also my first experience with virtualization (well, first bare-metal, I'd used apps like VirtualBox), and with the information on these forums it was pretty easy to setup. I had done a lot of reading before getting started though!

What about using Proxmox?

As with @Pitfrr, I like open source stuff, but ESXi is very mature and the free version limitations weren't an issue for me. I was aware of Proxmox, but I decided to go with ESXi mainly because of the people here that had gone down this path with FreeNAS before.

The main thing that led me to virtualization in the first place was that I wanted to run XProtect for security cameras, and it needed a Windows install. I was originally looking at using Jails/Bhyve under FreeNAS, but people said Bhyve is a bit immature for running windows.

 

[sretalla]

The thing of paramount importance is that you are able to pass through the HBA/SATA controller in full. The rest is up to how much you're prepared to put up with:

With ESXi:
Passthrough works as long as you're on a supported CPU and chipset (VT-X VT-D extensions)
Drivers for v-NIC, video and VM guest agent are all pre-installed in FreeNAS and work without effort
The preferred platform for most folks who are doing this

With Proxmox:
Passthrough seems to be an option... I have not seen anyone with a lot of experience with this (certainly compared to how much I have seen with ESXi)
Drivers for v-NIC offered... take VMware option, same for video. No guest agent, so no clean shutdown from the hypervisor and no status feedback in the console
A new option, not very widely tested

 

[simon_lefisch]

@zeebee so just so I make sure I am understanding correctly, you connected a drive to your PC/server and installed FreeNAS, as well as your storage drives, and set everything up. After configuring FreeNAS, you disconnected the driveS, installed another drive, then installed ESXi on that. Once that was complete, you connected your FreeNAS drives back to the mobo and passed it thru in ESXi and was able to configure it as a VM. If so, that’s an interesting way to set it up. Was there any particular reason you installed Freenas on a separate drive rather than creating an HDD file (like in KVM)?

Just to give you an idea of how I setup my test VM for FreeNAS, I created hard drive file to install FreeNAS on using VirtIO. I then added to physical HDDs using VirtIO (I added both drives without partitions) so FreeNAS could access them directly (as directly as possible) and create the partitions/mirror itself. In my testing, I noticed that FreeNAS didn’t have access to SMART, but my understanding is because the drives were not passed thru. I wasn’t too bothered by that since my host machine (CentOS 7) has smartmontools running for all physically installed drives, so if there were any errors smartmontools would detect them and notify me (I setup email notifications for smartmontools and mdadm which is the software RAID on my server). So I’m wondering if this setup would be ok for my use case.

@sretalla I am looking into how to pass the SATA controller thru. I have the disks I would in FreeNAS on a separate PCI card so it’s a matter of passing that bus thru. If, for whatever reason, I’m unable to do that, would the scenario I described as I’ve be ok?

Just to give everyone an idea of my hardware (I will put this in the OP as well):
Motherboard: Supermicro X10SRi-F
CPU: E5-2683v4
RAM: 112GB ECC DDR4-2400
OS: current CentOS 7 version
Drives:
OS: 2x 250GB SSDs (RAID1)
VM storage: 2x 500GB SSDs (RAID1)
Data storage: 2x 4TB HDDs (RAID1)

I’m not too worried about the hardware testing since I’ve been running this hardware for the last year and a half to 2 years. My main concern is configuration. I currently have a CentOS 7 VM running as my file server, but I would like to move towards FreeNAS since its capable of Time Machine backups. I may purchase a couple 8TB WD NAS drives and use those for file serving and reuse my 4TB for TM. Or maybe go a different, but now I’m going on a tangent lol. I’ve never used ESXi so I know that’ll be a learning curve just to get my system back up (my server runs pfSense as my router so if I make the switch to ESXi, I’ll have to get that up and running first for internet). I’m also looking at either upgrading CentOS 7 to v8 or switching to Debian 10 stable, as both will allow me to use a new version of KVM than what I currently have (v1.5.3 if I recall correctly). Hopefully peeps can give me the best options with what I currently have.

 

[simon_lefisch]

If I absolutely need to switch to ESXi then I will. However I feel I should be able to get a stable system with my current setup.

 

[sretalla]

If, for whatever reason, I’m unable to do that, would the scenario I described as I’ve be ok?

VirtIO disks aren't OK for long-term FreeNAS stability as they won't pass the SMART data through. Keep working on the pass-through of the PCI card, that's the only way to have it work well.

 

[simon_lefisch]

VirtIO disks aren't OK for long-term FreeNAS stability as they won't pass the SMART data through. Keep working on the pass-through of the PCI card, that's the only way to have it work well.

Well like I said, I'm not too worried about not having the SMART data available in FreeNAS since all my drives are monitored by my host box using smartmontools. However it sounds like FreeNAS should have it regardless?

 

[zeebee]

zeebee so just so I make sure I am understanding correctly, you connected a drive to your PC/server and installed FreeNAS, as well as your storage drives, and set everything up. After configuring FreeNAS, you disconnected the driveS, installed another drive, then installed ESXi on that. Once that was complete, you connected your FreeNAS drives back to the mobo and passed it thru in ESXi and was able to configure it as a VM. If so, that’s an interesting way to set it up. Was there any particular reason you installed Freenas on a separate drive rather than creating an HDD file (like in KVM)

That's not quite it. The original FreeNAS install is no longer used, a new install is created in a VM. The idea is to test your hardware with FreeNAS before introducing virtualization. FreeNAS should have direct access to your drives - if I understand it correctly, it's not just about not having access to smart data, and if you don't give it direct access then you can run into serious problems that can result in the loss of your pool.

So I setup my hardware - all pool drives plugged into the HBA card - and installed FreeNAS on a USB stick. You then test everything, make sure you're happy, and only then introduce virtualization by installing ESXi. (For me, that was onto an SSD plugged directly into the main board). You then install FreeNAS into a VM and pass it the HBA. FreeNAS now sees your drives in exactly the same way it saw them when you were testing without virtualization. If you ever run into any problems, you know you can boot directly to a FreeNAS install (e.g by plugging in that USB stick, or installing a new one) and you'll be able to access your pool data.

You can skip the bare-metal-install step if you're confident you don't need it, but you should only use virtualization if you can 100% pass through your HBA (or whatever device your drives are connected to). From what I've read, all the advice about not virtualizing FreeNAS comes from people losing their pools because they didn't get the pass-through right. As @sretalla said, it is not OK for long-term stability of your pool if you don't get this right.

 

[simon_lefisch]

@zeebee ok that makes more sense lol. I'm still working on trying to get the PCI card I installed to pass through but haven't been successful yet. Looks like even though my mobo supports virtualization, it may not support iommu which is what supposedly is needed to pass through PCI/VGA buses. Will report back when I can.

 

[simon_lefisch]

@zeebee @sretalla @Pitfrr I was able to get PCI passthrough working on my CentOS 7 box. Also got UEFI mode/Q35 chipset as well. Stoked!!!

Now here's something that I observed.....I installed FreeNAS with UEFI, however the drive file I installed it on only works when using a SATA controller rather than VirtIO. Also the same for the NIC....had to change from VirtIO to e1000 (intel). I know you guys use ESXi, however I was wondering if you knew or anyone might know if it would make a difference as to install FreeNAS using BIOS or UEFI?

 

[sretalla]

if it would make a difference as to install FreeNAS using BIOS or UEFI?

Not to FreeNAS in terms of how it runs... folks might argue that UEFI is more secure as it can detect (if not prevent) operating system tampering. I'm not too worried about that.

 

[simon_lefisch]

Not to FreeNAS in terms of how it runs... folks might argue that UEFI is more secure as it can detect (if not prevent) operating system tampering. I'm not too worried about that.

Awesome, thanks for the info. I do have one more question (last one, then I'll start a different thread for anything else) and it is pertaining to storage/pools...

I currently have (2x) 4TB HDDs on my box in RAID1 configured on CentOS. I do want to switch those the FreeNAS which means I'll have to break the RAID in CentOS....and again, my reasoning for using FreeNAS is the Time Machine support. I'm also looking at possibly upgrading my storage capacity to 6TB-8TB. So here's my question (I think I already know the answer but want to verify)....say I install (2x) 8TB drives and add them to the pool (this now becomes my main storage array). Would it be better to create 2 shares in Sharing >> SMB and have 1 share assigned as Time Machine backups and the other be for normal file sharing on the LAN, or should I use separate drives for each (would rather not if I don't have to)? If the former, will the 2 shares be isolated ("partitioned") from each other?

 

[sretalla]

Would it be better to create 2 shares in Sharing >> SMB and have 1 share assigned as Time Machine backups and the other be for normal file sharing on the LAN, or should I use separate drives for each (would rather not if I don't have to)? If the former, will the 2 shares be isolated ("partitioned") from each other?

As I understand it, Time Machine is a dedicated share and can only be used as that once it's set up.

You can share out different datasets in the same pool, so you don't need separate drives... pool them together and separate functions out by dataset.

 

[simon_lefisch]

As I understand it, Time Machine is a dedicated share and can only be used as that once it's set up.

You can share out different datasets in the same pool, so you don't need separate drives... pool them together and separate functions out by dataset.

Yea that's pretty much what I was asking, just didn't know how to word if better. I did try that out....add 2 data sets (one for TM and the other for normal file sharing), configured a quota for TM and the rest for the other set. I did notice one thing tho....the HDDs I'm using to test are 500GB. So I set the TM share 200GB and the other share to 300GB, but when I connected my Mac to the TM share it said there was 215GB of available space. Same with the other share, said there was 305GB available. Juts to verify, I deleted both sets, created a single set and set the quota to 500GB. When I connected to it, there was 537GB of available space. Where is the extra space coming from?

 

[subhuman]

created a single set and set the quota to 500GB. When I connected to it, there was 537GB of available space. Where is the extra space coming from?

500*1024*1024*1024=536,870,912,000
It comes from computers liking base-2 and humans liking base-10. And from storage manufacturers liking to market their devices as having greater capacities..
Anyway, if you're seeing the above number (or close to it) you're fine.

 

[simon_lefisch]

Thanks @subhuman. Got my FreeNAS VM running perfect now. Thanks to everyone in here for the help, much appreciated :D