TrueNAS Help! (Hardware Raid or vMware Virtual HDD)

[emrekoc]

Hello,

I've been using FreeNAS and TrueNAS products for my backup systems for a long time, and I am very satisfied. I have a question that I would like to ask but haven't dared to try. Can you help?

I have a Dell R740 18LFF model server. I've installed 18x10 TB SAS HDDs on this server, and there are also 4 x 6.4TB NVMe SSDs in the PCIe ports.

What I want to do is install VMware on the server and virtualize my customer virtual servers on NVMe SSDs while also virtualizing my system backups on the virtualized TrueNAS. I'm worried about whether I'll encounter any problems if I even set it up in HBA mode because disks presented from VMware won't work as HBA, and this concerns me. If I use hardware RAID with this usage scenario and add a non-mirrored disk on TrueNAS, would there be any security issues I should worry about? Or should I run the disks as non-RAID and present them to the system via VMware? My goal is not to achieve performance, as I am only concerned about security since I will be taking backups. Thanks to all friends who will help.

 

[Arwen]

First off, TrueNAS does not want any hardware RAID, (though RAID-1 boot pools in a VM client environment can be acceptable).

For reliability, you pass through the disk controller to TrueNAS VM;

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

Their are very specific requirements for a reliable TrueNAS VM. The top one is passing through the disk controller. Even if you have to buy another to dedicate to TrueNAS.

Others can add more, as I don't use VMWare or TrueNAS as a VM...

 

[emrekoc]

First off, TrueNAS does not want any hardware RAID, (though RAID-1 boot pools in a VM client environment can be acceptable).

For reliability, you pass through the disk controller to TrueNAS VM;

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

Their are very specific requirements for a reliable TrueNAS VM. The top one is passing through the disk controller. Even if you have to buy another to dedicate to TrueNAS.

Others can add more, as I don't use VMWare or TrueNAS as a VM...

Thank you for your response. So, what you're suggesting is that I should set the RAID card to HBA mode, and through VMware, I should present the disks to the TrueNAS server opened as passthrough, providing direct physical access. Is that correct? In the opposite case, would I be unable to use the Smart features in a TrueNAS ZFS structure added as a virtual disk, and would it pose a security risk beyond performance issues?

 

[emrekoc]

Hello again,

I have found the most suitable scenario that I can use in the VMware vSphere environment, and I would like to share the scenario I created for others to use. We have tested this scenario for demo purposes in the current environment, and we were able to achieve the desired results.

Hardware configurations were added to the Dell R740xd 18LFF server as follows: 4 x 6.4 TB NVMe PCI Express 2 x 100GB Enterprise SSD (For Mirror TrueNAS) 16 x 10 TB Enterprise nSAS

For this configuration, VMware vSphere 8.0 was installed on the USB SSD disk on the server. Virtual servers are running on SSD disks attached through the PCIe slot. In addition, HBA mode was activated, and all disks are operating in passthrough mode.

One virtual server was created for TrueNAS, and all passthrough disks were assigned to the TrueNAS server with physical access via the "Add Harddisk => Raw Disk" tab. The 100GB SSD disks were given as a mirror for the TrueNAS boot partition, and 16 SAS disks appear to be physically connected. The brand, model, and serial number information of the disks, as well as smart features, are active and everything seems to be in order. In this way, a ZFS Pool was created, and some disaster scenarios were tested. Everything is working great. Thank you to everyone who read the topic and tried to help.

 

[Arwen]

Unless you pass the plain, non-RAID CONTROLLER(s) through that has the disks, data integrity can be compromised. This is because ZFS writes data in a specific order;

1. Data, to currently free blocks
2. Metadata, (directory info), to currently free blocks
3. Uber block that activates the Data & Metadata, written to oldest Uber block slot, (aka ring buffer)

Some RAID controllers will do elevator seeks & writes to optimize write speed. If that happens, and the Uber block is written first, then a crash occurs, (yes I know it's rare), you've just corrupted your ZFS pool. Potentially catastrophically.

Having VMWare virtual disks, or VMWare passing through physical disks but not the controller can have the same effect. Please read the supplied Resource, it has more details.

This write behavior is not the fastest for file systems. But, it is designed to maintain data integrity of existing data by using COW, Copy On Write. Meaning, until the Uber block is written, (in the proper sequence I list), any data to be over-written still exists.

ZFS was tested to maintain existing data integrity over multiple un-expected crashes / power cycles. Even 10s of thousands of them! Of course, any data in flight will be lost... but that is true for any file system. (Now hardware failures can occur during un-expected crashes or power cycles that can cause data loss... redundancy of Mirroring or RAID-Zx can help here.)

 

[emrekoc]

Unless you pass the plain, non-RAID CONTROLLER(s) through that has the disks, data integrity can be compromised. This is because ZFS writes data in a specific order;

1. Data, to currently free blocks
2. Metadata, (directory info), to currently free blocks
3. Uber block that activates the Data & Metadata, written to oldest Uber block slot, (aka ring buffer)

Some RAID controllers will do elevator seeks & writes to optimize write speed. If that happens, and the Uber block is written first, then a crash occurs, (yes I know it's rare), you've just corrupted your ZFS pool. Potentially catastrophically.

Having VMWare virtual disks, or VMWare passing through physical disks but not the controller can have the same effect. Please read the supplied Resource, it has more details.

This write behavior is not the fastest for file systems. But, it is designed to maintain data integrity of existing data by using COW, Copy On Write. Meaning, until the Uber block is written, (in the proper sequence I list), any data to be over-written still exists.

ZFS was tested to maintain existing data integrity over multiple un-expected crashes / power cycles. Even 10s of thousands of them! Of course, any data in flight will be lost... but that is true for any file system. (Now hardware failures can occur during un-expected crashes or power cycles that can cause data loss... redundancy of Mirroring or RAID-Zx can help here.)

Thank you, Arwen,

Based on your recommendation, it seems that the issue has been resolved by using my RAID card in HBA mode and VMware raw disk pass-through. I've tested a few disaster scenarios, and everything is working great. This has allowed me to have an efficient system usage.

 

[ChrisRJ]

@emrekoc , did you read the resource provided by @Arwen ?

Unless you want to increase the risk of loosing data, you cannot use a RAID adapter in any configuration. You need an HBA flashed to IT mode. A RAID controller in HBA mode is very different from an HBA.

Your setup is pretty advanced, so please be prepared to invest a couple of days with reading up on the details. If possible, it might be better to keep things (NAS and VMs) separate to reduce complexity and by that operational risk.

 

[emrekoc]

@emrekoc , did you read the resource provided by @Arwen ?

Unless you want to increase the risk of loosing data, you cannot use a RAID adapter in any configuration. You need an HBA flashed to IT mode. A RAID controller in HBA mode is very different from an HBA.

Your setup is pretty advanced, so please be prepared to invest a couple of days with reading up on the details. If possible, it might be better to keep things (NAS and VMs) separate to reduce complexity and by that operational risk.

Currently, my RAID card is operating in HBA mode, and the disks in HBA mode are defined as pass-through (RAW) to the TrueNAS virtual server created on VMware. Isn't this exactly the desired configuration? Virtual servers are running on PCIe NVMe SSDs independent of the RAID card, and all disks connected to the RAID card are in non-RAID HBA mode.

 

[Ericloewe]

No and no. RAID controllers are typically disastrously non-performant when pretending to be dumb HBAs (in addition to being much less tested than LSI IT mode HBAs). Worse, disk passthrough is known to be buggy and lead to corrupted data, besides the added overhead.

 

[emrekoc]

@emrekoc , did you read the resource provided by @Arwen ?

Unless you want to increase the risk of loosing data, you cannot use a RAID adapter in any configuration. You need an HBA flashed to IT mode. A RAID controller in HBA mode is very different from an HBA.

Your setup is pretty advanced, so please be prepared to invest a couple of days with reading up on the details. If possible, it might be better to keep things (NAS and VMs) separate to reduce complexity and by that operational risk.

I continued reading the documentation, and they strongly advise against Raw disk pass-through. This has confused me a bit. I thought I had everything sorted out, but it seems I may have rushed. Even though everything seems fine in my demo environment for now, continuing with a scenario that is not recommended wouldn't be sensible.

I also have an HP DL380e 12 LFF Gen8 model server that I can use. Would it be a safer scenario to activate IT mode on it and install TrueNAS without virtualization?

 

[Patrick M. Hausen]

Currently, my RAID card is operating in HBA mode, and the disks in HBA mode are defined as pass-through (RAW) to the TrueNAS virtual server created on VMware. Isn't this exactly the desired configuration?

You wrote this two times already and no, it isn't. Using an LSI HBA and PCIe pass through of the entire HBA is the only way to run virtualised TrueNAS and not lose data.

 

[emrekoc]

You wrote this two times already and no, it isn't. Using an LSI HBA and PCIe pass through of the entire HBA is the only way to run virtualised TrueNAS and not lose data.

Dell or HP servers use LSI chipsets in the background. The H730p cards, for instance, have the LSI LSISAS3108 chip. Does this signify something, or should the card directly be an LSI card?

 

[emrekoc]

Additionally, due to my lack of English practice, I can sometimes misunderstand what is written. I apologize for that. Upon re-reading the documents, I understood that ESXi does not recommend RAW pass-through and that it can lead to various other problems. Therefore, I am abandoning the scenario of performing backups on the same server as the VMs. So, the only question remaining is whether a Dell server with an LSI chipset (H730p - LSI SAS3108 chip or H740p - LSI 3508 chip) should be used in HBA mode, or is it necessary to have the actual LSI card for this purpose?

 

[ChrisRJ]

Would it be a safer scenario to activate IT mode on it and install TrueNAS without virtualization?

Please read again what I wrote! Using IT mode of a RAID controller or HBA is vastly different from using a suitable HBA that is flashed(!!) to IT mode.

At the risk of coming across as impolite, which I certainly don't want: I don't know what your background in IT is and how much time you spent on this topic. But we are talking about serious stuff here. Unless this is for a playground system, which you indicated it is not, this is not something you should approach with the expectation of solving in a day or two. People who do this professionally have usually spent several years learning things. Assuming ZFS is new to you, please plan at least more than just 2 weeks for this. And even this will be stretching things.

So far all the things you asked are covered in the resources on virtualization and HBA. That makes me wonder how you are approaching this project.

 

[Ericloewe]

Does this signify something,

Yes, but the SAS3108 is not a good choice, since it's a RAID controller - although the hardware is very similar, the software stack is completely different.

Dell server

Depends on the Dell server... If you have a Gen 13, 14 or single-CPU AMD Gen 15 server, you can just get an HBA330 mini for the proprietary mezzanine socket. If you don't, you might as well get a standard LSI card.

 

[emrekoc]

Please read again what I wrote! Using IT mode of a RAID controller or HBA is vastly different from using a suitable HBA that is flashed(!!) to IT mode.

At the risk of coming across as impolite, which I certainly don't want: I don't know what your background in IT is and how much time you spent on this topic. But we are talking about serious stuff here. Unless this is for a playground system, which you indicated it is not, this is not something you should approach with the expectation of solving in a day or two. People who do this professionally have usually spent several years learning things. Assuming ZFS is new to you, please plan at least more than just 2 weeks for this. And even this will be stretching things.

So far all the things you asked are covered in the resources on virtualization and HBA. That makes me wonder how you are approaching this project.

Hello ChrisRJ,

I want you to know that, due to your helpful nature, I won't perceive anything you say as rudeness. My IT background goes way back, and I actually encountered FreeNAS six years ago. I've been using it in a less-than-recommended scenario on an old Dell R510 12SFF (with H700 Raid Controller) server without IT mode for six years and without HBA controller. The reason for making these less recommended settings in the past was due to limited resources, but it has saved me from disaster scenarios numerous times over the past six years, and I haven't experienced any data loss. On the contrary, I'm quite satisfied. My perspective on ZFS is not like yours, with professional use cases. For that, I use a Dell EMC Unity filled with Flash SSDs. TrueNAS is an area I'm interested in mainly for backup scenarios (iSCSI, SMB, CIFS, NFS, FTP, etc.). Therefore, my goal is to create the most reliable setup with the resources at hand or at minimum costs. I believe it's time to transition my FreeNAS server, which I've been using for six years, to a new structure, and I think seeking help from professionals like you who use the ZFS system is the best choice.

I've also read every document you've sent. However, as you may appreciate, obtaining clear answers in a language that is not your native language for specific questions can sometimes be challenging, and the responses in the documents might not always be satisfactory.

Yes, but the SAS3108 is not a good choice, since it's a RAID controller - although the hardware is very similar, the software stack is completely different.

Depends on the Dell server... If you have a Gen 13, 14 or single-CPU AMD Gen 15 server, you can just get an HBA330 mini for the proprietary mezzanine socket. If you don't, you might as well get a standard LSI card.

Thank you for your response. I'm glad to hear that I can solve this at a low cost by ordering the HBA330 mini.

 

[emrekoc]

Yes, but the SAS3108 is not a good choice, since it's a RAID controller - although the hardware is very similar, the software stack is completely different.

Depends on the Dell server... If you have a Gen 13, 14 or single-CPU AMD Gen 15 server, you can just get an HBA330 mini for the proprietary mezzanine socket. If you don't, you might as well get a standard LSI card.

I just placed an order for the HBA330. Is there anything I should be concerned about in my VMware ESXi scenario when using the HBA330? When the card arrives, should an IT Firmware did with the H330, or is it unnecessary since the HBA330 is already an HBA card? I'm aware that I'm asking a lot of questions. Please excuse my concern. I wouldn't want to regret this after moving from a system I've been using for six years without any issues and designed without following recommendations, especially after transitioning to a new system at a high cost.

 

[Ericloewe]

No, the HBA330 only supports IT mode. Just make sure the firmware is the latest from Dell.

 

[emrekoc]

No, the HBA330 only supports IT mode. Just make sure the firmware is the latest from Dell.

Hello, I bought the HBA330 card based on your recommendations and performed the installation. I can view the disks, but when trying to assign them to a virtual server, only the "Add Harddisk => New Raw Disk" option is available. The PCI passthrough option does not appear to be active. I believe there have been some changes in VMware 8.0.

 

[Patrick M. Hausen]

You need to go into Host > Hardware settings first and activate pass through for the controller. Then in the VM settings add a PCI device, not a disk.