Truenas Core or Scale as a guest VM in Type-1 hypervisor

[MyRequiredName]

I am speccing out a compact, energy efficient virtualization host. One of the main guests will be some type of shared folder service.

I have reviewed all of the posts and recommendations on the forum on how to run truenas as virtualized guest and do this properly.

Now, this server will be made from consumer parts, but with ecc ram and obviously virtualization and pci device pass through support. The only question I have is on the hba.

I understand the hba should be passed through to vmware or hyper v or proxmox or whatever the type 1 hypervisor is. My issue is with the heat and power consumed by these lsi hbas. This server will absolutely need to keep power and heat to a minimum.

The board I am looking at is a consumer grade AM4 socket board with 4 available sata ports and 2 m.2 slots, one of which being an nvme. I have heard about the perils of using nvme sata controller splitters. But they are the least energy hungry option. All i need are 3 disks passed through with the controller to the guest. Of the crappy controllers I see that JMicron JMB585 may be the best option as packaged in the silverstone adapter. These controllers consume a fraction of an hba.

The bottom line question:
Do you think it makes the most sense to pass through the motherboard-integrated controller, and use the JMicron JMB585 to host the 1 or 2 OS drives?

Meaning, is the onboard sata controller on the avergae motherboard better than JMicron JMB585? And if the OS fails to boot because of the controller, the 3 disks should be transferable to another truenas instance for recovring the pools?

And yes, this design will be backed up religiously.... TBH I am more concerned with unreported & unresolved ECC memory errors that are written to disk than I am of obvious sata controller failures. Because at least with latter, you can go to good backups. The former, could destroy the quality of your backups for years and you wouldn't know unless you test them.

Thanks in advance

 

[jgreco]

I have reviewed all of the posts and recommendations on the forum on how to run truenas as virtualized guest and do this properly.

passed through to vmware or hyper v or proxmox or whatever the type 1 hypervisor is.

You sure you read all the posts and recommendations? Because I distinctly recall saying that Hyper-V is not suitable.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

My issue is with the heat and power consumed by these lsi hbas.

That is not a magic wand to make some other solution work correctly, unfortunately. No one said that you would be able to make your host out of whatever arbitrary parts you wanted. There's some level of hazard in picking random consumer grade parts, especially since some of them are going to be knockoffs, which may not work correctly with FreeBSD/Linux driver support. This is a known issue. The counterfeiters do not contribute patches to make their knockoffs work with FreeBSD or Linux.

You should definitely review

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

Especially the warning against port multipliers, just in case you are tempted towards that path. There are also occasional discussions of working and non-working AHCI silicon now and then. The winning choice appears to be ASM1061 or 1062 based add-on cards, if you can find legit ones that include true ASMedia controllers and not knockoffs. The 1064 is okay unless it is bundled with a port multiplier, which is often the case.

 

[MyRequiredName]

That's the problem with truenas, is that it requires such particular hardware that it's simply not that attractive a solution, even if it qualifies as software raid, and even if its free. If I am going to be running enterprise grade hba's ecc ram, bare metal truenas scale, and be tied to upgrading vdevs in multiples of pairs, then the value proposition for most home users is very low.

At this point, I think truenas would fit on a low-risk implementation, as a backup server because of the versioning available in zfs.

I think unraid is more suitable a software "raid" platform. Because it has all of the conveniences of simple ah hoc, unplanned upgrades, single drive utilisation with in-built support for aggressive HDD sleep cycles (obviously at the cost of multi drive performance you would normally have in raid or multi vdevs) and when things do go wrong, the files are kept in tact on a single disk as opposed to the block-level spread you get with SS\refs and truenas scale.

The only feature that unraid doesnt have is file versioning and file scrubbing (i won't be using btrfs because I won't be running a ups and power loss is a factor). Which is a risk i have to take. Its too expensive for home usecase to run drives 24/7 burning electricity, running a powerful hba 24/7 and be forced to purchase sets of drives for expansion, or wastefully perform single drive upgrades in sequence for differential increases in storage by virtue of larger drives...

fingers crossed ecc ram and the in-built ecc on wd drives is good enough to for bitrot to not be an issue.

 

[jgreco]

That's the problem with truenas, is that it requires such particular hardware that it's simply not that attractive a solution, even if it qualifies as software raid, and even if its free. If I am going to be running enterprise grade hba's ecc ram, bare metal truenas scale, and be tied to upgrading vdevs in multiples of pairs, then the value proposition for most home users is very low.

That's of course your choice to make. You may have misunderstood what TrueNAS is, though. If you are thinking that someone set out to write a NASware package for you, you'd be incorrect.

TrueNAS CORE is the beta testing version of TrueNAS Enterprise, a package that iXsystems sells on their dedicated and generally very pricey enterprise NAS systems. This is how iXsystems pays the salaries of developers. In the meantime, they have graciously agreed to allow the use of TrueNAS CORE in other applications for your own benefit. This isn't entirely altruistic; they are hoping for you to act as a beta tester to help eliminate bugs from their commercial product.

It is a common mistake for people to come in and be all offended that TrueNAS hasn't been made with this or that feature, or optimization towards their small home platform. Rather, the truth is that the closer you get to hardware that resembles what iXsystems is selling to its enterprise customers, the better your results will be. The system is optimized for that, for hopefully obvious reasons.

And here's where it seems to go off the rails. Community members should please try to be aware that the hyperconverged storage market is filled with competition. iXsystems is trying to build an amazing product with a limited staff, and they need to be able to focus on the things that pay the bills. If they allow themselves to go off tilting at windmills like comprehensive device support, they are taking limited developer resources away from more important things, and this will hurt iXsystems and this community in the long run. We need to be respectful that they've chosen to decline to try to support this, and all of us here in the community would be well advised to take a moment to be thankful for the generosity of iXsystems, who has offered you the ability to use the fruits of their labor on your own server, free of charge.

You do have options. My suggestion?

Go use a product that works the way you prefer. TrueNAS is simply not aimed at your small scale, low port count, low power NAS build. It is a big fat resource-intensive package that would prefer to find itself running on a host with 64GB or more RAM, lots of CPU, etc. This is probably not the right fit for a true low power hyperconverged box like the one you describe wanting. No one here is going to be offended if you decide that TrueNAS is the wrong product for your needs. As a moderator, I want to make it clear that you're welcome to ask for suggestions for alternatives, and that this is certainly an allowed discussion on these forums. We will not judge you poorly or treat you poorly just because ZFS isn't the right thing for you. We do want you to have a succeessful outcome, no matter what route you choose.

As you said, the value proposition for home users wanting a small Synology- or QNAP-class solution is indeed relatively low. However, lots of home users want a powerful and competent NAS platform and are willing to properly resource their NAS, as evidenced by the nearly 100,000 users who have signed up for the forums alone over the last decade.

 

[MyRequiredName]

I agree. But to add to the discussion, I think Truenas may be able to capitalize on the burgeoning pro-sumer market... much in the way unraid has. But would indeed require additional development on device support, extra sanity checks for resiliency due to less than ideal hardware.

Because we are in an age where most people have many gigabytes and even many terabytes of data from their digital lives piling up every decade that needs decent storage. The avg person now has multiple reasonably powered systems that are unused to due to frequent upgrades. and things like this.

If truenas were to compete with unraid, leverarging the strengths of zfs but more consumer friendly, that would be a market you could dominate in.

Most enterprise scale implementations Ive seen have either moved to the cloud these days or using netapp or similar san technology.

 

[Davvo]

That's the problem with truenas, is that it requires such particular hardware that it's simply not that attractive a solution, even if it qualifies as software raid, and even if its free. If I am going to be running enterprise grade hba's ecc ram, bare metal truenas scale, and be tied to upgrading vdevs in multiples of pairs, then the value proposition for most home users is very low.

At this point, I think truenas would fit on a low-risk implementation, as a backup server because of the versioning available in zfs.

I'm sorry, but I don't agree with your opinions.

Most home users do not want truenas as a guest VM in a type-1 hypervisor, they use it for storage/backup and a few VMs or jails on bare metal. For such needs, you only need to pay attention to a few things: not buying a realtek NIC, not using hardware RAID and not buying sata multipliers, then you have to assemble a proper pool layout. It's something easily achievable with consumer or prosumer grade hardware, especially if you don't use ECC (which is the limiting factor in basic hardware picking, NOT the FreeBSD/Linux driver issues).

You just have to read before buying; of course it can get pretty expensive if you want the safest, most reliable machine with the greatest performance ever.
Also if you are using ZFS then you care about your data, so there is no "low-risk" implementation: either you want to safely store your data or you don't.

Also in my experience the highest cost is usually the drives, so I think that's worth spending a few bucks more for a more robust system since I am already spending at least a few hundreds for those spinners.

Do note that the driver incompatibilities do not lay in iX's hands, but in FreeBSD/Linux.

Anyway, I read that once there was the following saying:

Freenas is not for everyone.

Edit: corrected NIC brand and $$ amount for HDDs.

 

[jgreco]

Most enterprise scale implementations Ive seen have either moved to the cloud these days or using netapp or similar san technology.

TrueNAS is a direct competitor to NetApp and other storage vendors. Last year it was ranked as one of the fastest growing storage companies.

Blog - iXsystems: Fastest Growing Storage Systems Company in 2021 with TrueNAS

iXsystems reported 70% annual growth for its TrueNAS storage systems in 2021, which was the highest reported growth in the industry for storage systems. According to the analysis of 2021 data by Storage Newsletter, iXsystems was reported to stand alone in the systems category among the...

www.truenas.com

If Enlyft is to be believed --

Data Storage Hardware

• FreeNAS (0.67%)
• NetApp Storage Systems (15.31%)
• HP Storage Systems (6.46%)
• EMC SAN (Storage Area Network) (3.89%)

So TrueNAS is used at about 1/6th the places that use EMC SAN, or about 1/23rds the places that use NetApp. This is quite respectable for a small privately held company.

I agree. But to add to the discussion, I think Truenas may be able to capitalize on the burgeoning pro-sumer market... much in the way unraid has. But would indeed require additional development on device support, extra sanity checks for resiliency due to less than ideal hardware.

I have no idea who funds UnRAID's developers or what the business model is. Perhaps you could explain, since you're advocating it.

I do know that iXsystems has been building hardware for many years, but that they came to the realization that the hardware alone was a rough sell since there's a limited amount of margin that you can credibly stick on top of the price of a bog standard server. TrueNAS was an attempt to add value to the hardware by producing a fully featured appliance that was much easier to deploy. This nets them a LARGE profit margin which allows them to pay the developers.

You would need to explain where you think developer salaries can be squeezed from in the "pro-sumer market". Nexenta accomplished this by charging for the software once you got past a certain size, which was VERY unpopular with the "pro-sumer market". VMware decided to give away ESXi and instead charge for the management infrastructure (vSphere), knowing that enterprise customers would pay big bucks but there was still value to be had in exposing free users to the product as well.

But would indeed require additional development on device support, extra sanity checks for resiliency due to less than ideal hardware.

That just doesn't work. If the combined Linux and FreeBSD worlds have been unable to accomplish this over the years, there is no chance that a much smaller team such as the iXsystems team would be able to.

The iX crew has wisely decided that there is plenty of NASware in the world. They are fine with focusing on an underserved segment, which is the petabyte data market. If you want to hop on the train and go along for the ride, they make this easy to do. But it's not for everyone, and it isn't expected to be for everyone. No one here on these forums will think less of you (I hope!) if you believe UnRAID to be more suitable to your needs. It isn't a one size fits all deal. Even I make lots of VM-based NFS servers for various applications; a 256MB/1c sized FreeBSD host is very, very efficient and easily customized.

 

[MyRequiredName]

I guess the question is whether running truenas with less than ideal HW is WORSE than if someone were to plug an external drive to their laptop and store data there?

Is it in fact worse or better? Reliability wise?

If it is on par or better, than what is all the fuss with data loss FUD? When other vendors would gladly sell you an external usb hdd that can be easily corrupted, your laptop likely having no ecc, and things like this.

Most people store all kinds of personal data on these devices. I guess knowing the fragility of this, they also tend to store it elsewhere or burn to disk or use some cloud service naturally...

Are the glaring warnings about running truenas in an unsupported way a tactic to dispel any sense of security, should they store data on an ill configured truenas implementation; despite this implementation exceeding the quality of a mere usb drive?

Or is truenas in fact a more dangerous and fickle system, that requires careful implementation to do right? ANd that indeed, an external usb hdd is MORE SAFE than a "poorly implemented" truenas configuration?

Because that is essentially the use case for most home users. They want a big external hdd, over the wire.

 

[jgreco]

Most people store all kinds of personal data on these devices. I guess knowing the fragility of this, they also tend to store it elsewhere or burn to disk or use some cloud service naturally...

"Naturally"? I've had to tell people, numerous times, their external HDD is dead, and they have no backup. This was extremely painful to tell my sister-in-law, who had an external, with all the pictures of her kids on it, that one day gave up the ghost. It was probably recoverable, but then her ex-husband decided to "send it in" -- for RMA replacement. No more pictures.

If it is on par or better, than what is all the fuss with data loss FUD?

If you've ended up here at the TrueNAS Forums, we tend to assume your data is important to you. If it isn't, then you may be in the wrong place. If it is important to you, then it is indeed important to have a properly built system. Back in the early days of this forum, we had a lot of people showing up with janky setups such as AMD APU-based stuff, and had a lot of people with trashed pools due to insufficient memory. I actually generated a list of these and then raised the minimum memory requirement to 8GB for the project, because there was a commonality in that systems with 4GB or 6GB were failing. We still don't know what the exact bug was; none of these users were sophisticated enough to debug the actual failure at the needed level. Not all such systems died, and no one who was knowledgeable was ever able to duplicate it. This is well-documented project history.

Or is truenas in fact a more dangerous and fickle system, that requires careful implementation to do right?

There are definitely lots of things you can do to create a dangerous and fickle system. Insufficient memory, SMR HDD's, turning on dedup, using bad SATA controllers, using cheap mainboards, using a RAID controller, using an improperly sized PSU. The whole point of a lot of the guidance on this forum is how to to it right, because doing it wrong is bad. In some cases, badness might only result in poor performance, such as the selection of a Realtek ethernet chipset. In other cases, you can actually lose data or even damage or destroy your system.

I have a friend who has a massive camper trailer. Despite the fact that it could maybe be towed by a smaller vehicle, there are many reasons for the giant Ford Excursion SUV that he chose to tow it. It's a better choice to pick a vehicle that is designed appropriately for the job.

Storing big data reliably is difficult. The forum is here to help you do it right.

ANd that indeed, an external usb hdd is MORE SAFE than a "poorly implemented" truenas configuration?

An external USB HDD can be more safe, that is, until it fails.

Because that is essentially the use case for most home users. They want a big external hdd, over the wire.

TrueNAS isn't trying to address "most home users". It's intended for use by those who want a professional quality product that is going to reliably store their data. It is not intended to try to address those who simply want to store large amounts of data and who do not care much about it.

 

[Davvo]

To be honest you sound like a troll to me.
If you think TN isn't right for you, feel free to use other ways to achieve your objective.

 

[MyRequiredName]

I appreciate Grinch's responses. I am just reviewing all the options available to solve a need, thats all. I am not a troll. I am not a fanboy either. Just looking at what makes the most sense overall, for my usecase. And I am just pointing out things that do not make sense to me.

To me, because of all the various components and software stacck involved, it just strikes me as more dangerous if we do not follow Grinch's advice all the way through. Because any one of these components can fail resulting in different outcomes.

But with a usb disk, there are only a couple of components involved. Of course you are subject to the disk dying and lack of self-healing zfs properties, but it is simpler and fewer components = safer for careless use, especially if you take the time to back up data elsewhere periodically....

And unraid is in between these options where yes, it has the same type of server components between you and the data BUT the way the data is stored is very much like a usb drive, where files in their entirety are distributed across multiple disks as opposed to blocks. So a catastrophic failure on an unraid at most would result in the loss of certain disk and not the whole array. And you can take out single drives, dock them on a linux distro and easily copy off whats left, its very convenient and safe from the perspective of crappy hardware, and wreckless management. You are unlikely to lose absolutely everything... Its pretty ingenious... if you think otherwise, you are lying.

ZFS has its strengths though and would be perfect as a backup array with versioning and compression enabled.

 

[jgreco]

To be honest you sound like a troll to me.

In defense of @MyRequiredName , I had considered this possibility especially in light of the cheeky handle, but I also noticed significant commonalities with other new-to-ZFS users who have appeared in the forums, some of whom have ended up as today's power posters, because time and effort was spent on them to actually try to educate. I will indeed be annoyed of MRN is a troll, because I have other things I could be doing. Until this is clear, however, I prefer to view MRN as someone with an intensely curious mind who is diving deep to explore available options. I respect that. I do it too. I know I damn well appreciate the hell out of the people who bother to provide insight into topics I am unfamiliar with. So I do not mind being the person who tries to offer that insight here.

I'm that guy guy working in the woodshop who spies a newcomer about to use a table saw, unfamiliar with kickback or blade guards. From one viewpoint, it's quite reasonable. It's just a saw blade. Isn't it just like a hand saw, but faster? That's what I'm hearing from MRN with the USB HDD stuff. But the truth is, it's nothing like a hand saw other than it cuts. It cuts very well, likes to eat fingers, loves to launch wood projectiles at speeds sufficient to cause significant injury. You don't berate this person. You take them aside, and explain to them in detail the dangers of the table saw. They might not really appreciate the favor you've done for them for years, until they've had the opportunity to be on the other side of the equation, having witnessed tragedy. Whether they learn from what you told them, or learn from what they've seen happen to others, you still spend the effort to make sure that they don't lose a finger or a kidney in the process. Or else you're a poor excuse for a human.

ZFS has its strengths though and would be perfect as a backup array with versioning and compression enabled.

The ability to treat all your storage as a single pool, making any portion of it available for filesharing, iSCSI, etc., with just a few clicks is a big thing.

The ability to build an array of hundreds of terabytes or even petabytes, and then treat it as a single resource, is amazing.

The ability to store data on a filesystem that detects and repairs corruption is appreciated by everyone who has ever experienced bitrot.

Its pretty ingenious... if you think otherwise, you are lying.

It doesn't scale. As the designer of the hardware platform that Newshosting used for article storage, and author of the deterministic location feature that made it possible to use this hardware model, I feel free to explain to you that the loss of a drive and the data on it can turn into a problem in that it has a tendency to create I/O hotspots that impact performance, PLUS it's just a damn PITA to manage. Now, admittedly, we're talking large scale storage where you have 24 drives per chassis, and 11 chassis per rack, and then a whole row of racks. At 4TB drive size, that's 264 drives or a petabyte per rack. The problem with your "ingenious" is that when a drive fails, it causes a hotspot on the redundant storage cluster which now has to source all that data to make up for the lost disk. With 264 filesystems times however many racks, you're losing several filesystems a week and you are spending a lot of time and effort to re-establish optimal service. By way of comparison, if you could reduce that to one filesystem per chassis, and it is self-healing, then you just slot the replacement drive in and go on your merry way, letting the system handle it.

Because any one of these components can fail resulting in different outcomes.

Sure. This is why we spend so much time exploring these various failure modes, and how to engineer to avoid them.

But with a usb disk, there are only a couple of components involved. Of course you are subject to the disk dying and lack of self-healing zfs properties, but it is simpler and fewer components = safer for careless use,

Not everyone needs a freight train. If a moped is more your speed, well by all means, moped it up.