Is running Truenas on VMs (still?) risky business?

Status
Not open for further replies.

skimon

Dabbler
Joined
Jun 3, 2012
Messages
37
I havent really kept up to date with advancements in freenas/truenas. I installed freenas on recommended hardware years ago on 3 different machines and I havent had any issues but I was extremely conservative with hardware choices (only those that at the time were recommended on the stickies) and zpool , vdev configurations. I have only replaced disks for the most part.

I see that unofficially the same years old thread regarding vms has been reposted but I also notice a large influx of people installing vm based truenas . So i am wondering if thereare any technology changes in either the vm hypervisors or truenas itself that would make vms more reliable for freenas installation. Or is it still a matter of how much do you care whether you lose your data (ie using a vm is inferior) or are there best practices , which if followed would make using a vm installed truenas just as safe?

Thanks
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I havent really kept up to date with advancements in freenas/truenas. I installed freenas on recommended hardware years ago on 3 different machines and I havent had any issues but I was extremely conservative with hardware choices (only those that at the time were recommended on the stickies) and zpool , vdev configurations. I have only replaced disks for the most part.

I see that unofficially the same years old thread regarding vms has been reposted but I also notice a large influx of people installing vm based truenas . So i am wondering if thereare any technology changes in either the vm hypervisors or truenas itself that would make vms more reliable for freenas installation. Or is it still a matter of how much do you care whether you lose your data (ie using a vm is inferior) or are there best practices , which if followed would make using a vm installed truenas just as safe?

Thanks

It's like asking if hauling a fifth wheel with your truck is any safer now than it was years ago. You are physically doing about the same thing, so the overall risks are quite similar. It is still a big, heavy, potentially tippy, hard to back up arrangement that will lose badly if you do any number of things.

Some improvements have been made. These are core technological improvements, similar to how electric brakes or rearview cameras are things that are common today that weren't common years ago. They ease some of the pain points like mountain hauling or parking the beast, but don't change the overall physics of the situation or the general danger involved.

One of the biggest problems a decade ago was that PCIe passthru was a nascent technology, and the implementations on many systems were very poor and unreliable. This problem has definitely gotten vastly better; I wouldn't EXPECT a broken implementation to be a problem if you buy a server-grade platform that was designed in the last five years. I have chosen those words carefully, though. The server-grade boards are the ones most likely to get all the i's dotted and t's crossed correctly. I have definitely seen Sandy and Ivy systems with PCIe passthru weirdness, and maybe the occasional Haswell or Broadwell. I have no recollection of having run across people with newer stuff than that with issues on name brand commodity systems (i.e. Supermicro, HP, Dell). These companies are able to dump sufficient engineering effort into their systems to make obscure features like PCIe passthru work correctly.

Likewise, picking VMware as your hypervisor platform is the best path to success. While some people have had success with Proxmox, the PCIe passthru support there is still listed as experimental and was only introduced maybe five years ago. It has been problematic for some people. VMware, on the other hand, rakes in big bucks from their enterprise licensing and has put a lot of polish on their support for PCIe passthru.

So those are definite improvements, but you are still not guaranteed success.

Regardless of the date you see on the post, or the fact that it was posted when the product was still named "FreeNAS", the information I posted at


still represents an accurate summary of the state of the art in safely virtualizing TrueNAS. Ignore bullet points at your own peril. If you truly know why you can ignore them safely, good for you, go right ahead. My goal is simply to make sure you know the nature of the potential minefield you're walking into.

there best practices , which if followed would make using a vm installed truenas just as safe?

There is no such thing as "just as safe". There are inherently additional risks with virtualization. Even if you follow my guidance, for example, it is entirely possible for your automated deployment of ESXi to misfire and overwrite one of your TrueNAS data disks. You are blending together two incredibly complicated software systems, one of which is generally intended to be hosting somewhat simpler virtual machines, the other of which was designed to be running on bare metal. You are pushing the boundaries and it would be foolish to proceed without awareness of the sharp edges present.
 

CyberdineX

Cadet
Joined
Mar 26, 2016
Messages
6
It's like asking if hauling a fifth wheel with your truck is any safer now than it was years ago. You are physically doing about the same thing, so the overall risks are quite similar. It is still a big, heavy, potentially tippy, hard to back up arrangement that will lose badly if you do any number of things.

Some improvements have been made. These are core technological improvements, similar to how electric brakes or rearview cameras are things that are common today that weren't common years ago. They ease some of the pain points like mountain hauling or parking the beast, but don't change the overall physics of the situation or the general danger involved.

One of the biggest problems a decade ago was that PCIe passthru was a nascent technology, and the implementations on many systems were very poor and unreliable. This problem has definitely gotten vastly better; I wouldn't EXPECT a broken implementation to be a problem if you buy a server-grade platform that was designed in the last five years. I have chosen those words carefully, though. The server-grade boards are the ones most likely to get all the i's dotted and t's crossed correctly. I have definitely seen Sandy and Ivy systems with PCIe passthru weirdness, and maybe the occasional Haswell or Broadwell. I have no recollection of having run across people with newer stuff than that with issues on name brand commodity systems (i.e. Supermicro, HP, Dell). These companies are able to dump sufficient engineering effort into their systems to make obscure features like PCIe passthru work correctly.

Likewise, picking VMware as your hypervisor platform is the best path to success. While some people have had success with Proxmox, the PCIe passthru support there is still listed as experimental and was only introduced maybe five years ago. It has been problematic for some people. VMware, on the other hand, rakes in big bucks from their enterprise licensing and has put a lot of polish on their support for PCIe passthru.

So those are definite improvements, but you are still not guaranteed success.

Regardless of the date you see on the post, or the fact that it was posted when the product was still named "FreeNAS", the information I posted at


still represents an accurate summary of the state of the art in safely virtualizing TrueNAS. Ignore bullet points at your own peril. If you truly know why you can ignore them safely, good for you, go right ahead. My goal is simply to make sure you know the nature of the potential minefield you're walking into.



There is no such thing as "just as safe". There are inherently additional risks with virtualization. Even if you follow my guidance, for example, it is entirely possible for your automated deployment of ESXi to misfire and overwrite one of your TrueNAS data disks. You are blending together two incredibly complicated software systems, one of which is generally intended to be hosting somewhat simpler virtual machines, the other of which was designed to be running on bare metal. You are pushing the boundaries and it would be foolish to proceed without awareness of the sharp edges present.
Great example thanks
 

Scharbag

Guru
Joined
Feb 1, 2012
Messages
620
I have been virtualizing TrueNAS for years on server grade hardware (old SuperMicro X8 and X9 systems).

1665813737217.png


I followed all of the recommendations above and so far, I have not run into any "oh shit" moments. Key is to do it properly with good gear. This is very affordable today with used server grade gear.

I am now on ESXi 7.0.3 (started on 5.5) and it runs great.

1665813455164.png


I do give TrueNAS decent resources and passthrough my SAS2008 based HBAs. So far, all is golden.

Now, would I run it this way at work? Hell no. But for my personal use, it has been just fine.

Cheers,
 

Syptec

Dabbler
Joined
Aug 3, 2018
Messages
42
When you say virtualize it, are you saying run it as a VM no pass thru or run it as a VM with PCIe pass through (Limits migrations, DRS, HA and so on). We have made an appliance that is a VM with no pass thru, removed compression, no sync and limit it in size. It has been up over 700 days. We also run it as a Raidz so that the actual virtual drives can be replaced should a backing NAS have issues and call for it. Similar to physical drive replacement. Or are you asking something outside that?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
When you say virtualize it, are you saying run it as a VM no pass thru or run it as a VM with PCIe pass through (Limits migrations, DRS, HA and so on). We have made an appliance that is a VM with no pass thru, removed compression, no sync and limit it in size. It has been up over 700 days. We also run it as a Raidz so that the actual virtual drives can be replaced should a backing NAS have issues and call for it. Similar to physical drive replacement. Or are you asking something outside that?

Please read the linked resource and the article that it also links to, which discuss the topic in depth. I'm going to close this thread as it seems to be encouraging people to ask questions for which there are extensive answers already posted.


 
Status
Not open for further replies.
Top