Windows Server 2012R2 as Host and FreeNAS as VM Guest

[SecCon]

I have searched for "virtual" and even if I see some relevant partial references I do not find any recent evaluation or description of such a contraption. (Well, this, but I have no ESXi )

THIS THREAD AND THE INFORMATION IN IT IS FREQUENTLY UPDATED SO CHECK ALL THE POSTS BEFORE DRAWING ANY CONCLUSIONS. I may write a summary eventually.

(The FreeNAS specified in my signature is not in any way involved in this topic.)

I have a Home server based on 2012R2. It has a Raid. It has a hardware Raid. It is a Raid 5 based on LSI 2008 controller, integrated with a Supermicro Opteron Motherboard. I read you guys like LSI, and Supermicro. Got two Opteron 4122 CPU (2 x 4 cores @ 2.2GHz) on it and 64GB RAM.

So the hardware is not old (ddr3) and I can do pretty much what I like with it. However I want to keep the Windows OS and setup FreeNAS virtually.

So the question is pretty much, is there a guide for Virtual FreeNAS on a Windows host that I have missed? There are a few considerations/decisions to be made before setting it up, so I would be grateful for your advice.

 

[Mirfster]

I guess the first question I would ask is "what is the intended use?"

Meaning, are you just doing this for testing or are you planning on having FreeNAS handle storage of vital data?

While I have not tried this in quite a while, I personally would not go this route unless it was just in a "testing" environment. IMHO I don't really consider MS as a "True Hypervisor"; but more of "ParaVirtualization" (but others may disagree).

For my lab, this is a general overview of what I have (I am only mentioning the Servers pertaining to this):

• ESXi Server running several VMs and FreeNAS is one of them
• FreeNas handles all the storage (except for the initial ESXi datastore, which is not much) via PassThrough
• Microsoft Server 2012 R2 DataCenter Server (its own box)

Now, within FreeNAS I have:

• DataSet called "ALS-ISCSI-V01"
• "ALS-ISCSI-V01" contains separate ZVols ("ESXI-ISCSI-ZVOL" and "MS-ISCSI-ZVOL")

My benefits:

• All Data is properly handled, controlled and maintained by FreeNas (MS VMs on "MS-ISCSI-ZVOL" and ESXI VMs on "ESXI-ISCSI-ZVOL")
  
• VMs housed appropriately of their own Volumes
  
• Scrubs, Drive Health, Expansion, etc. is controlled by FreeNAS
• ESXI VMs still benefit from VMWare abilities
  
• MS VMs still benefit from Hyper-V abilities and since I am using MS DataCenter I can run unlimited Server 2012 R2 VMs
  
• All items are connected with 10Gb so that helps as well, but I am only using direct connect right now (do have a 10Gb switch I will be installing later)

Hope this helps.

 

[Chris Moore]

If you're interested in doing this, you would need to have a separate controller and drives that you can pass through to the VM. The virtual instance of FreeNAS would still need to be able to directly address the drives in the storage pool.
If FreeNAS can't address the drives directly, there isn't any point in using FreeNAS. The advantage of FreeNAS is using ZFS and ZFS doesn't work right if not directly connected to the drives.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[tvsjr]

Another few things to bear in mind:
Opterons suck, these days. Lots of power, not a ton of performance.
You will need to pass through a controller to your FreeNAS instance. Having your drives on a RAID controller won't work.

I have a personal bias against virtualizing a core service like FreeNAS except in very specific instances, but that's just me. Make sure you know *why* you're doing what you're trying to do.

 

[SecCon]

@Mirfster I might get to where you are some day, but right now it just seems a lot over the top for me. You and @Chris Moore mention a separate controller, well I don't know if you are advising me to actually get a different drive controller, which seems unnecessary, because the motherboard comes with one, the LSI. It's on the mobo, if you check the linked specs. I am aware of the pass-through requirement.

If I don't use HyperV what would I use? VMWare Workstation?

Opteron is good enough and within any reasonable lower specs, but not the higher end hardware recommendation I am putting together in parallel with this one.

I could just keep the machine as it is until I know more about FreeNAS via the other one, but I find these matters interesting to look at and I currently need to squeeze the best possible solution out of my current hardware. From what I understand that would be scrapping current hardware Raid and making FreeNAS work from within Windows Server. I am not certain of any performance boost in regards to that, but anything above inherent Raid5 speeds is a welcome bonus, with less disk wear and safer management.

 

[tvsjr]

FreeNAS needs to have DIRECT access to the drives. That means no RAID controller, and no virtualization layer in the middle. The usual process is to use something like an LSI 9211, M1015, etc., flash it for IT mode (which makes the card just an HBA, disabling all RAID functionality), and then passing that through via VT-d to the virtualized guest. If FreeNAS can't see each individual drive, or even if FreeNAS simply can't get SMART data from the drives, you're setting yourself up for failure. It's just how ZFS works.

 

[SecCon]

Frankly I don't see that as a problem, I just don't make any Raid.

I can't disable the controller since it would be disabling the SAS2/SATA interface so the driver would have to be in place for it, but other than that... you even mention LSI 2008 as SAS controller in the Recommended Hardware Guide, so I am just blatantly assuming that I can use that controller to access the drives, without using the Raid functionality.

Additional SATA/SAS connectivity If more connectivity than is available from the PCH is desired, or if SAS is required (due to the use of expanders, for instance), the only reliable solution is to add an LSI/Avago/Broadcom SAS controller. Some motherboards recommended above provide equivalent functionality to the cards described below. SAS2 By far the most popular option, SAS controllers based on the LSI SAS2008 or SAS2308 chips provide up to eight lanes of SAS connectivity and are available in a number of configurations, with external and/or internal ports. Some cards are configured as RAID controllers and require a crossflash to standard HBA mode. Popular models include the LSI SAS 9211, SAS 9240 and SAS 9207 series, the IBM/Lenovo M1015/M1115 and the Dell H200 and H310. All of these, except for the LSI SAS 9211 and 9207, must be crossflashed. An additional option is the LSI SAS 9201, a 16-port card which uses the LSI SAS2116 controller, which is also known to work well and only supports HBA usage, not hardware RAID. It is not very popular because it is more expensive than a pair of 8-port HBAs.

Just hope there is a guide for the flashing you are recommending @tvsjr and its ok with Supermicro since I won't jeopardize any warranty.

 

[Chris Moore]

I am just blatantly assuming

Making assumptions can get you into trouble. If the SAS controller is flashed with the IT mode firmware, and passed into the VM, you could be fine. Still, you will be in largely uncharged territory trying to do this with Hyper-V.
The system board you have says it supports an optional RAID-5 hardware key. If you have that installed, you would want to remove it.

Detailed newcomers' guide to crossflashing LSI 9211 HBA and variants
https://forums.freenas.org/index.ph...o-crossflashing-lsi-9211-hba-and-variants.54/

 

[tvsjr]

so I am just blatantly assuming that I can use that controller to access the drives, without using the Raid functionality.

When you ASSUME, you make an ASS out of U and ME. There's plenty of documentation and words from the builders that say that FN expects direct disk access... not via an underlying RAID controller (even if that controller is set up in "JBOD mode").

If you choose to go forward, you do so at your own peril. Don't expect peak performance, and don't expect FN to give you proactive warnings, allow you to run SMART tests, etc. Your first warning of an impending problem will be when a drive goes totally offline.

 

[Chris Moore]

I was doing some more research on the new FreeNAS 11.1 release and found the following details about the underlying FreeBSD operating system that might be interesting to you:

Code:

PCI passthrough support has been enabled on FreeBSD virtual machines running on Microsoft® Hyper-V™. [r309312] (Sponsored by Microsoft)

The hv_netvsc(4) driver SR-IOV implementation has been updated to support Virtual Function (VF) devices, such as the Mellanox® Connect-X3™ network card. [r314091] (Sponsored by Microsoft)

Support for Microsoft® Hyper-V™ Generation 2 virtual machines has been added. [r316272] (Sponsored by Microsoft)

Support for synthetic keyboards has been added for virtual machines running on Microsoft® Hyper-V™. [r317119] (Sponsored by Microsoft)

reference: https://www.freebsd.org/releases/11.1R/relnotes.html#kernel-general

Also: http://doc.freenas.org/11/intro.html?highlight=resilver priority#new-features-in-release

 

[SecCon]

The system board you have says it supports an optional RAID-5 hardware key. If you have that installed, you would want to remove it.

Detailed newcomers' guide to crossflashing LSI 9211 HBA and variants
https://forums.freenas.org/index.ph...o-crossflashing-lsi-9211-hba-and-variants.54/

I have indeed a RaidKey and that was a good point. (The bugger cost me 150$ extra in addition to the Motherboard).

Interesting reading about the FreeNAS HyperV specs. Thank you good sir.

@tvsjr I won't tell anyone I failed despite your advice. :)

Going to configure things for a small trial. Keeping the Raid functionality on one array (I have two) I will remove the second, reconnect a couple of disks to the non LSI SATA connectors I got on the board on an AMD SP5100 chipset and putting in a SSD for the virtual OS allocation. Got SSD as OS disk already. Yes that chipset is probably crap too, but, hey, gimme some room here. ( I see some Grub issue, but no one saying it actually is bad, so maybe not so much after all...). Is there no search tool for the Errata and Release Notes? Ah its not a wiki.... bummer.

On a general note, this is all rather new to me and before some of you start jumping up and down in your chairs (I see that tendency) this thread exists not only for my benefit, but for others who may find this approach tantalizing. I also realize I am using far from the optimal hardware and thus have some issues that are not easily dealt with, or at all, but in my defense I would like to point out that I have had no reason to go all out Intel ever before and the rig in my signature is my first Intel CPU purchase since Pentium MMX. ;)

 

[Chris Moore]

Support for AMD will probably improve now that they have a new chip set out, or I should say a whole new line of processors out, but there's nothing saying you can't use AMD, it's just going to present some challenges for you.
A lot of the recommendations I make are geared towards trying to keep you from running into failure points. I wish you the best of luck.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Mirfster]

@Mirfster I might get to where you are some day, but right now it just seems a lot over the top for me. You and @Chris Moore mention a separate controller, well I don't know if you are advising me to actually get a different drive controller, which seems unnecessary, because the motherboard comes with one, the LSI. It's on the mobo, if you check the linked specs. I am aware of the pass-through requirement.

If I don't use HyperV what would I use? VMWare Workstation?

Opteron is good enough and within any reasonable lower specs, but not the higher end hardware recommendation I am putting together in parallel with this one.

I could just keep the machine as it is until I know more about FreeNAS via the other one, but I find these matters interesting to look at and I currently need to squeeze the best possible solution out of my current hardware. From what I understand that would be scrapping current hardware Raid and making FreeNAS work from within Windows Server. I am not certain of any performance boost in regards to that, but anything above inherent Raid5 speeds is a welcome bonus, with less disk wear and safer management.

Apologies, I do not visit the forums nearly as much as I used to.

Regarding whether or not to use Hyper-V is entirely up to you. However, VMWare Workstation is not what I was talking about. Instead, I meant "vSphere Hypervisor" (100% Free); which basically just requires you to create an account here: https://my.vmware.com/en/web/vmware/evalcenter?p=free-esxi6
(Reference "VMware vSphere Hypervisor, which includes VMware ESXi and vSphere Client")

In regards to your LSI 2008 controller, even if you do not set it up as a Raid; it can still interfere with trying to pass through to FreeNAS unless it is in true IT Mode. This is due to the fact that it is set at the controller level for whatever OS (MS or FreeNAS) is accessing it.

Seeing what was posted about FreeNAS and pass-through looks promising, but until I personally test it I wouldn't 100% recommend it.

So to circle back, it truly depends on what your required "Use Case" is if you want to use MS or VMware. Each has their own "Pros", "Cons" and limitations (Like MS Server Standard licensing limits you to 2 VMs; where VM doesn't care how many).

Let us know more about the proposed Server (Make, Model, Actual LSI Controller) and maybe someone will have more insight.

FWIW, from what I have gathered thus far; I personally would think of simply turning the Server into an ESXi host (flash the LSI - if possible) and house my FreeNAS as a VM in there. I would pass-through the LSI 2008 to FreeNAS and go from there. However, consider that is a "vague" response. :)

Lastly, please don't take anything personal if we all seem to "start jumping up and down in your chairs"; for us the first and foremost concern is preventing data loss. Last thing we want to do is put/let someone get into a situation where they come back yelling "I lost all my vital data!". ;)

 

[Chris Moore]

Lastly, please don't take anything personal if we all seem to "start jumping up and down in your chairs"; for us the first and foremost concern is preventing data loss. Last thing we want to do is put/let someone get into a situation where they come back yelling "I lost all my vital data!".

@Homdax what he said... It has happened many times that people begin putting data on their NAS before they have it fully configured and/or before they fully understand how it works and the data gets lost. It recently happened that a pool that had been encrypted became lost due to the user not updating the keys and storing the new ones after replacing a disk. Then they had an unrelated reboot and, subsequently, could not decrypt the pool. There are so many little things that can be a source of tragedy, that we try to guide new users in a safe path for the safety and happiness of all involved.

 

[SecCon]

Not to worry. I will be testing before using or even claim to have a working solution.
Thanks for the reply @Mirfster , all duly noted. HyperVisor for free? Who knew.

My first planned attempt failed, I have a 2 x 5,25" enclosure/backplane that I rigged with one SSD and two 1TB drives to test but turns out the enclosure won't start. Bloody thing took me over an hour to assemble in to the server chassi. Had to Mod the drive trays too (drill a few holes), since they had no holes for 2'5" drives and I was out of adapter plates. Ah well, back to dismantling stuff and see if I can spot any obvious burn marks or leaking condensators or something along those lines..

My data is safe and will be safe regardless of what I do with my FreeNAS configurations and trials. I have true redundancy. Well, I lack the offsite, but my house can burn down to 90% and I still have a working current backup. Christ, now you are going to get me digging down some USB drives in a hole in the ground at least 100 meters away from the house. ;)

 

[SecCon]

Finally installing my Virtual experiment.

Now, should I have added SCSI controllers to existing HDD before installing o r can I do it afterwards? I clearly must be able to install FreeNAS on a machine (virtual or otherwise) and setup storage (adding disks) afterwards. Ah well, we'll see.

Edit. I forgot to mention a snag. Yes, we get those buggers.

Snag: In following the HyperV Create Virtual Server Guide, it failed each time I had create Virtual HDD among the options.

Solution: Create the virtual HDD before, set the server to use it and then its creation went fine. However then a drawback occurred, you weren't asked about a boot iso for an OS. That can be configured later, before you start the VMachine, so just add a new SCSI > DVD device, and alter boot options accordingly, remember how you did that, because FreeNAS will tell you to remove the boot device, so remember to alter options back to boot from HDD, or just remove the DVD mount.

And now for a bash with 12 options. 1-6 NIC related that I am not touching.
9? Shell? No way, runs and hides...
Disconnects that window to prevent issues.

Bottom line shows me a local IP for Web Interface. now we are talking. And I am in. After initial very *nix inspired overwhelming options list in regards to Time Zone and Keyboard mapping I get to volume manager, which I EXIT, since I have to add HDD's via HyperV first to create a pool to have my data (well a trial amount of it) to swim in.

Ehmm, maybe I should have a peek at the manual.... presumed self explanatory interfaces can only take you so far.

 

[Mirfster]

since I have to add HDD's via HyperV first to create a pool to have my data (well a trial amount of it) to swim in.

Ehmm, maybe I should have a peek at the manual.... presumed self explanatory interfaces can only take you so far.

This is where I would start to get scared... From what you are describing it appears to me that you are creating Virtual Drives in Hyper-V and then allocating them to the FreeNAS VM? If so, you are for sure going down the wrong path to believe that any vital data should be stored there.

In plain and simple terms FreeNAS needs direct access to the hard drives. Anything else is not recommended. From what I have looked up really quickly, it was stated that The "Hyper-V SCSI controller is a synthethic device" (makes sense when I stop to think about it..).

Only possibility that comes to mind is to add a new SCSI controller to the VM and on that controller add all the "physical" hard disks to the VM. Of course, this means that the disks must be set to "offline" on the Hyper-V Host itself. If this appears to work, then I would suggest that you try to obtain SMART info on the drives from within FreeNAS to see if correct info is returned.

Still seems a bit sketchy to me; but I am curious at the same time. :)

 

[SecCon]

Hi there... don't get scared. This is an experiment after all. :D

I found out how to mount the drives directly, well as directly as I seem to be able to. Couple of things.

1. They need to be Offline in Windows Disk Management
2. You add them as SCSI Controller Physical Hard drive in the hardware management of the Virtual machine.
3. It seems you need to add one Controller per HDD (going to try that again, can't be right (?)
4. Not touching advanced features, like Virtual HDD sharing or QoS

So I am already doing what you tell me to, and verifying your findings.

Disks show up under "import disk" section in FreeNAS Volums Manager.

In regards to direct access, anyway comes to mind to check that from within FreeNAS Web UI somehow?

Snag2: Secure Boot option in Machine Settings Firmware will fail to load FreeNAS

Reminder: this is Windows Server 2012R2 on the Host Machine.

 

[SecCon]

An addition: I can see added disk as da1 and da2 in Reports / Disk

 

[Ericloewe]

The number of moving parts involved in that passthrough setup seems prone to breaking. It's totally possible that Microsoft took care of the little details, but it's equally possible that they didn't - let's face it, it's a highly unusual setup.