Where to setup RAID when using VM

[Rod Martin]

I have been fooling around with NAS devices for a while now. The FreeNAS interface seem pretty self explanitory, and I have been able to create one in VMWare. My question is in regards to the RAID array. I would like to manage the RAID in FreeNAS, but I think that requires pretty much a software raid at this point, because I think you have to mount each drive independently with the hardware in AHCI mode, then setup the array in FreeNAS. I could be wrong, but my thoughts is that this would not be very efficient, becuase the RAID would actually be done in software. I would like to use hardware raid, but then that would be done outside of FreeNAS, at the operating system level, or BIOS level. Then, I would share the volume created in Windows 10 Pro (in my case) and pass it through to FreeNAS. But, at that point, FreeNAS would not be doing any of the managing.

I could have this all wrong, or I need some tweaks... The main thing is I want this PC to be in a native Windows environment because I will also be running CAD on this workstation. I have plenty of RAM and CPU to spare. This is also going to be a 2nd NAS, mainly for backup of my primary stand-alone QNAP. But, I also want local space for use with my CAD files.

Thoughts??

Thank you.

 

[garm]

Actually you got it all wrong. Start by reading the manual, especially the ZFS Primer.

To give you a hint of the level of your understanding at this point, any form of RAID as you understand the term is unsupported by FreeNAS.

 

[Chris Moore]

I could have this all wrong, or I need some tweaks... The main thing is I want this PC to be in a native Windows environment because I will also be running CAD on this workstation.

So, I think you are saying that you want to run FreeNAS inside a virtual machine that is running on a Windows computer through something like VMWareWorkstation.
This is not something that will work reliably. Take a look at these guides to help with your understanding.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.
https://www.ixsystems.com/community...ide-to-not-completely-losing-your-data.12714/

Virtually FreeNAS ... an alternative for those seeking virtualization
https://www.ixsystems.com/community...ative-for-those-seeking-virtualization.26095/

The thing about ZFS is that is needs direct access to the drives, for this, we suggest passing the entire SAS controller through to the virtual machine.
Trying to do it the way you appear to be thinking will result in a system that is ripe for failure and data loss.

It would be better to build a stand-alone FreeNAS system as FreeNAS will protect your data better than the QNAP system. I have converted several of the QNAP systems where I work to run FreeNAS instead, but if you have one of their low end systems it will not have the processor and memory capacity for this.
FreeNAS, because of ZFS, is far superior at protecting your data to any other NAS solution I am aware of.

 

[Rod Martin]

Great reply, Chris. Yes, it was my impression that making virtual disks and passing them to FreeNAS would have lots of opportunity for bad things.

One think I just discovered... I upgraded to VMWare Workstation 15 Pro. I see there is now a direct disk function. This makes me happy, because it does eliminate one point of possible failure. I was able to create a RAIDZ2 in FreeNAS with this method.

I am curious about the QNAP conversion. I have a TVS-1582TU which is great because it has thunderbolt 3. Four ports of it. On an i7-7700 and 16G RAM with 4 open slots. It was expensive. I got it for 40% off which made it more palaltable. I like QNAP because they (I have had 5 of these now) have been extremely reliable and easy to setup, etc.

I also got an HP DL380 Gen10 for a super deal too. I was going to use it to play with FREENAS. It wasnt long after I found a super deal on a Supermicro Scalable CPU workstation. It has 8bays that I dropped in 8x4TB drives. This is the one I'm working on now. I was hoping to have something that I could backup the essential files from my QNAP.

I dont know if you followed all of that, but I tried to give you a snapshot of my gear. I was kinda in a hurry when I wrote this, so I dont know that it flowed very well.

 

[Rod Martin]

Actually you got it all wrong. Start by reading the manual, especially the ZFS Primer.

To give you a hint of the level of your understanding at this point, any form of RAID as you understand the term is unsupported by FreeNAS.

Ok... not sure what you were trying to accomplish except to blast my current level of understanding of FreeNAS... Thats what this forum is for, right? To help learn and share with others.

Well, whever you think my level of knowledge is... Z2 looks to be similar to RAID6, and the setup for that array went about the same as it does on my QNAP. It was pretty easy to figure out. I also found that VMWare Workstation 15 Pro has the ability to pass an entire disk through to the VM. FreeNAS saw it like it was a native install.

Anywho, whatever you may term RAID vs RAIDZ... Seems to be very similar fundamentals to me, even though what is going on behind the scenes may be much different, I'm not sure how that relates to my post.

 

[HoneyBadger]

You'll have to pardon the coarseness of some of the user base here; they see "ZFS" and "RAID" mentioned together and get flashbacks to dozens (or hundreds maybe) of posts from users who've done that against warnings and had problems ranging from "poor performance" to "total catastrophic pool failure with all data lost"

There's a fair bit going on "under the hood" that makes ZFS a bit more of a prima donna when it comes to how it likes its disks presented; it needs the ability to issue certain SCSI commands and get an "honest response" from the device; whenever an intermediary layer (such as a hypervisor or virtualization software) is involved, there's a non-zero chance of it ignoring the command or fudging the reply.

The only way to guarantee unfettered access to the disks is through a method like PCI passthrough of the SAS/SATA controller, and from the description here you only have one in the system which negates that possibility. If Win10/VMWorkstation does PCI passthrough (might be called IOMMU) then you could buy a second SAS controller and hand that to the FreeNAS VM.

In your situation I'd look at dropping an HP H220 HBA into that DL Gen10 and using that as the FreeNAS machine; alternatively, use a type 1 hypervisor like ESXi/KVM, stuff it with RAM, and pass the HBA through to a FreeNAS VM hosted there.

Either way the important part is that you're asking the question rather than just charging ahead. ;)

 

[Rod Martin]

Fantastic reply HoneyBadger, That is super helpful. I have the liberty to experiment a little since this is my sendondary device. The DL380 is unfortunately a SFF (2.5") drives. I have 8 LFF, so the next easiest was the supermicro workstation. I dont have hardware to run all of the gear yet. I'm experimenting to understand more what FREENAS needs, and so I can get what is proper.

What do you know about VMWare 15? Perhaps this is not new, and I just didnt notice it... but this is what it looks like when creating a Harddrive device.

 

[HoneyBadger]

It looks like unfortunately the VMware Workstation product (regular or Pro) doesn't support the VT-d/IOMMU/PCI passthrough functionality that's necessary to give ZFS full access to the drive controller. Giving the virtual machine access to the physical disk is likely similar to "local RDM" in the ESXi/vSphere world, which isn't 100% supported either (SATA controllers especially tend to bit more miss than hit on passing the appropriate SCSI commands) and I'd be worried that the Windows device driver/other layer of device obfuscation would make things behave in an odd fashion as well. Most ZFS users are rather risk-averse, and you're heading into mostly uncharted waters with this setup.

What are the CPU/RAM specs on the other machines? If your CAD/CAM workload is highly threadable, you might get fantastic results by putting a GPU enablement kit in the DL380 and using that for your "desktop" along with a few SSDs.

Or failing that, sell it back and use the proceeds to buy more traditional "workstation" (I'm a big fan of the Dell T3600/T3610 myself) and let the Supermicro box be a dedicated NAS.

 

[garm]

I’m sorry you read in aggression in my response.

Well, whever you think my level of knowledge is... Z2 looks to be similar to RAID6

This is actually completely false, which is why I wanted you to read the pre existing material before continuing. For starters, ZFS is a file system, a RAID array is a device found in /dev..

As honeybadger and Chris already mentioned, ZFS expects direct hardware access as part in performing its awesomeness. If you present a RAID array or a disk through a hypervisor you will have a bad experience, with an unacceptably high chance of losing data.

 

[Rod Martin]

Ok, it looks like we are getting down to the nitty gritty of it. I did have the impression that building a system via passing through a VM would have some fundamental flaws. It seems that has been confirmed. WHich is the idea of this thread.

I am a heavy "application" type of user. I am not a deep in the weeds type of user. I have had a lot of experience and success bringing things together at that level. Unfortunately, I have never had the time to go too deep into things. Thus, I have used the consensus of those in forums, etc to help with decision making.

This is meant to be a learning experience for me, and I am grateful for all those who are contributing to this.

Let me step back a bit, and outline my current hardware sets, and my goals. This may help.

Current production system:

• Qnap TVS-1582TU, i7-7700 16GB 10GBe over Cat5. 5x 8TB Enterprise Drives, Raid6, 4x 4TB Enterprise Drives RAID5. Uses: Main file server, plex

Systems in evaluation (I got these new under serious discounts, or I wouldnt have either of them):

• HP DL380 Gen10, SFF chassis. (No CPU, No RAM), 5x 1.8TB Enterprise 2.5" drives, 1x HP 250GB SSD
• Supermicro SYS-7049GP-TRT, Xeon Silver 4114, 32GB RAM, 8x 4tb Enterprise drives, HP 1TB m.2 NVMe boot.

Applications needed:

• CAD design
• Audio workstation
• Secondary NAS (backup of backup)
• pFsense firewall
• mail server
• plex.

What I would really like to do, if I can gain confidence in the FreeNAS, is to sell the QNAP box. Its expensive, and I can probably get a good amount for it. I got it at a great deal as well, but the HP and SM are much higher performance. I was hoping to cram all of my needs in those two boxes via various VMs and such. The best platform for the CAD and Audio is the SM box. The rest I was wanting to either do with a VM on the SM box, through VMWare ESXi on the HP. I suppose I could do FreeNAS native on the HP, then run pfsense through a VM through FreeNAS... Let me do this in a less verbose manner:

This is a straw-man to beat up...

HP: Native OS: FReeNAS (to avoid the disk issues)

• FreeNAS VM: Pfsense
• FreeNAS VM: Mail server

SuperMicro: Native OS: Win10 Pro

• VMWare Workstation 15 Pro: Some file backup schema for a secondary backup of the main NAS... it does not have to be FreeNAS, if thats not suitable for a VM. But, what could it be???
• Plex Server (native) (Plex server can definetely take advantage of the high end gpu)

Figuring out how FreeNAS works helps me with this plan. Thats the rationale for my questions.

 

[pro lamer]

A wild guess:

SuperMicro: Native OS: Win10 Pro

• VMWare Workstation 15 Pro: Some file backup schema for a secondary backup of the main NAS... it does not have to be FreeNAS, if thats not suitable for a VM. But, what could it be???
• Plex Server (native) (Plex server can definetely take advantage of the high end gpu)

Figuring out how FreeNAS works helps me with this plan. Thats the rationale for my questions.

Are you considering making this machine ESXi based with two VMs: one for Windows and one for FreeNAS? You may also search here and there if people are successful running Windows in bhyve (if so then you might virtualize Win10 in FreeNAS having FreeNAS bare metal)

Sent from my phone

 

[garm]

If you have the time and motivation I would suggest benchmarking what ever CAD tool you intend to use (or maybe you mentioned that somewhere?) in bare metal Windows vs ESXi Windows VM. I did some tests on PTC Creo Parametric a while back at work and although I could spot the different, most test users could not.

If you can live with a virtual workstation I would suggest you set up your primary FreeNAS box doing its thing and setting up the second machine with ESXi (as already suggested) running a virtual FreeNAS as secondary repo and a Windows VM. Set it up correctly and you can probably host Windows on ZFS through FreeNAS as well as storing a backup from your primary.

 

[Chris Moore]

Figuring out how FreeNAS works helps me with this plan.

Didn't you say there was a 1TB SSD in the Supermicro system?
That idea from @garm sounds like a good one. You can use ESXi to pass a SAS controller through to FreeNAS and have all the storage drives in FreeNAS and then share that storage back out to other things through a virtual network that lives inside the ESXi environment.
You can then use ESXi to pass the GPU and a USB controller through to a virtual machine for Windows and have ESXi give you a virtual storage on the SSD for the 'disk' that the windows OS is installed on and all your data can be on the network share that FreeNAS provides. It could be super fast. FreeNAS doesn't need much CPU resources and no GPU at all.
You might want to read through this build report about the ESXi configuration that another forum member did:

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://www.ixsystems.com/community...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[Rod Martin]

Now, that is a very interesting scenario I had not even considered. Using ESXi with a VM for a 3D modeler... So, you can use PCI bus direct in a VM from ESXi? I have fooled with ESXi on the HP... had it up with FreeNAS... I was trying to find my disks through the SAS controller... then heard that I would need a rather inexpensive LSI 9207-8i SAS card (though mine has a HPE Smart Array P408i-a card... not sure this is suitable??).

I would need to do some testing with that. Getting access to the GPU for the shading and such for Solidworks/Creo.. as well as if I can do my video work. Mainly video capture through a PCI HDMI/SDI capture card, etc while in a VM.

So, let me see if I can sort some of this out...

For FreeNAS...

BEST: Native on stand-alone.
2nd Best: ESXi VM (w/ suitable SAS controller)
Dont do it: VM through Windows.

If I am unable to do my normal workstation apps through ESXi VM, is there a comparable NAS software that can be efficiently run through windows? I would not need the full suite of tools that I do on my primary, but I would want enough to ensure good RAID redundancy control, and overall reliability.

On my test list:

Try Solidworks, Avid Media Composer, Nuendo DAW, on ESXi VM.


The good news is... I have a 2nd 1TB m.2 card I can simply swap out of my Supermicro system to try the ESXi scenario, without wiping what I've currently setup under windows.

Thanks for everyones input... and dealing with my somewhat disjoint posts... I have a LOT swimming around in my brain. I'm trying to get it all sorted, to come up with a good, reliable, and efficient setup.

Thanks again! I hope we can still banter this around a bit more.

 

[Chris Moore]

Thanks again! I hope we can still banter this around a bit more.

Absolutely. It is even possible that other contributors like @Stux that have more experience with this may jump in and give some guidance/suggestions.

 

[pro lamer]

That idea from @garm sounds like a good one. You can use ESXi

Wasn't it mine? ;-)

Are you considering making this machine ESXi based with two VMs: one for Windows and one for FreeNAS?

Sent from my phone

 

[Rod Martin]

Wasn't it mine? ;-)

Not sure what you are asking...??

Sent from my phone

 

[HoneyBadger]

Not sure what you are asking...??

It's a facetious comment about who proposed the idea of a "virtual CAD workstation" first, it's an argument for Internet Points.

(Besides, I think I said it first, guys. ;) )

 

[Rod Martin]

It's a facetious comment about who proposed the idea of a "virtual CAD workstation" first, it's an argument for Internet Points.

(Besides, I think I said it first, guys. ;) )

Well, i'm certainly willing to give credit where it is due... but I reread your post, and apparently there is a bit of a language barrier between me and what it is you were trying to explain. WIth that said, re-reading that I think you were suggesting turning the HP server into a workstation. That is an interesting possibility. However, I will need the capabilities of the Supermicro to do other workstation activity such as video capture/editing, and audio editing. So I do not think I'll be able to use that option. That requires more hardware that may not have a generic driver that will work in the HP.

WIth regards to what I thought @garm was suggeting twas to run ESXi, then run a VM on the SAME physical PC. I quickly found that running ESXi is pretty much all you can do on that PC. I would need to run a RDP to get to the VM. I'm sure you guys already knew that, but I did not. I also question what that buys me, since I will still need another PC to run the RDP client on to get to the workstation VM. Since I have a 3 monitor setup, and lots of GFX hungry applications, I dont know what that does for me. At that point I still need three hot PCs to cram all of my needs into. This maybe what I end up having to do, but I want to be sure I'm understanding what everyone is saying here.

Please, lets keep bantering about the options. At this point, unless I'm misinterpreting something, I am still in a 3 PC scenario as follows:

QNAP NAS:

• Primary NAS
• plex.

Supermicro Workstation: Native OS: WIndows 10 Pro

• CAD design
• Audio workstation
• Other graphics intensive applications

HP DL380: Native OS: ESXi

• Secondary NAS (backup of backup) FREENAS, VM
• pFsense firewall VM: Linux
• mail server

My only question is with the HP, will I need a different SAS controller such as the LSI 9207-8i, or should FreeNAS pickup the HPE Smart Array P408i-a. I have had mixed clues on whether the HP Smart Array will work for it. It can be put in AHCI or MIXED mode... so it does not have to be setup as a RAID. If I'm understanding that correctly now...

Please keep the comments coming!

 

[Chris Moore]

When you pass a video card into the VM, you should be able to connect local monitors to the video card instead of only accessing the Windows VM remotely.