Pool and vdev layout for ultimated mixed-drive and expandable setup

[crc]

I'm upgrading from Synology to TrueNAS, and I'm trying to get the best of all worlds here (TrueNAS as software, but with the expansion functionality in Synology).

I think I've figured out how to make that happen.

I have a mixture of drive sizes, and I plan to add drives over time. I have 18 bays to fill up, and I currently have 7 drives (4 6TB, 2 8TB, and one 18TB).

Since drives usually come in multiples of 2TB, I'm planning to break each drive down into multiple 2TB partitions (up to 15 per drive). And then combine partitions across drives into a single vdev. And then combine those vdevs in parallel into a single pool.

Then when I add drives, I can use a script to "fail" individual 2TB partitions from existing drives and redistribute them to new drives (until I have enough drives with 2TB free on them to create a new vdev of parallel partitions). This should allow me to grow the array one drive at a time, upgrade drive sizes anytime I want, and not lose any space due to being able to effectively have one huge array (which can survive 2 or 3 drive failures, depending on how I set it up--without needing extra space).

I'm pretty sure I can script this all out using the excellent command-line functionality available with TrueNAS scale. I'd honestly love to see this built into TrueNAS, as it would make it a no-brainer upgrade from Synology (especially since Synology hasn't added many (or any) new high-capacity 3rd-party drives to their compatible list).

Has anyone tried this before? I don't see any obvious issues, as long as it's scripted (so that I don't configure it wrong). My only concerns at the moment are that ZFS might try to write files to multiple partitions in separate vdevs thinking it's spreading out the load but unintentionally put a lot of extra wear and tear on a small number of drives that happen to have those partitions--but I would presume that ZFS typically tries to spread the load so that shouldn't be too big of a deal.

I figure that this group of fine folks has tried just about everything and might have some good input before I go and do something very bad :)

BTW, it should be possible to maximize drive usage (without needing to wait for the OpenZFS update that expands drives in real time) in this way, albeit with a little shuffling around of data necessary. As long as there are >=3 (or >=4) drives available with the same or greater capacity, those drives could all get additional 2TB partitions on them in a new vdev that was added to the pool, expanding the storage space further. The only downside there is that those larger drives would get a little more wear and tear, but I think that's just the normal kind of tradeoff with mixed-size arrays (and something that could be assuaged by TrueNAS prioritizing storage of infrequently-used data on larger drives).

 

[Arwen]

Yes, you can do that. But, see below.


TrueNAS does not support partitioning drives for multiple vDevs or pools. Performance, disk reporting and changing out of disks will be complicated. This can lead to inadvertant data loss, something us users of ZFS & TrueNAS want to avoid.

This functionality will almost certainly never be built into TrueNAS SCALE or Core. iXsystems, the developer, writes and maintains TrueNAS for Enterprise customers, (aka paying customers). In the Enterprise space, supportability out weighs clever disk usage. Being able to remove 1 disk and have exactly one impacted vDev reduces risk to the pool.

Here is some useful information for prospective TrueNAS SCALE users:

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

Some of that applies to TrueNAS Core, but Core has not gotten as much attention from new users as SCALE.

I am not trying to stop you, just letting you know that their are risks. And if / when problems develop, we might not be able to help.

 

[crc]

That's very helpful Arwen. Thank you.

It's great that TrueNAS is so flexible, but it's probably best to not overcomplicate things.

 

[WI_Hedgehog]

@crc : Use LVM on an OS you like. LVM does what you're looking to do dynamically, quickly, and it "just works."

Hard drives have error detection/correction on-board, so every once in a while do a smartctl test. When your system has time do a long test (live) as that will read the entire drive, and if there are any checksum issues the drive will usually use Error Correction to recover the data and try rewriting the corrupt data block which may simply suffer from magnetic degradation. If the block is bad the drive will write the data elsewhere and update the Bad Blocks List. When a manufacturer-defined threshold is exceeded the drive will report the smartctl status as FAILED (it's pass/fail) and you replace the drive.

The way to make all this work well is to do a system burn-in so you've verified you're starting out with working hardware. Use ECC memory, preferably Xenon CPUs which support 4-bit ECC. We typically use server-grade hardware because gaming hardware is awesome for running the latest bleeding-edge games fast, and the hardware lifespan is 2-years, a few errors mean what, a few dropped pixels on one frame and no biggie. On a data storage server an error may mean your song, movie, family pictures are forevermore corrupted and that's that, so we tend to use solid hardware. Use an APC or Eaton UPS so you have clean power with no brown-outs. There are threads on all of these subjects here, reading up on how to build a solid TrueNAS system is just as meaningful even if not using TrueNAS.

 

[phanter]

Did you have success with your setup?
How is the performance?
And are you willing to share your script?

 

[artlessknave]

I'm planning to break each drive down into multiple 2TB partitions

you can use mixed drives just fine with mirrors. plan your sizes out s your vdevs are as closesly sized as possible and then just replace them and/or add new vdevs. this would be *by far* better than the frankenstein you are imaging that would become a nightmare.

Did you have success with your setup?
How is the performance?
And are you willing to share your script?

likely not. this is so far outside a supported truenas design that there is absolutely no point in using truenas. defeats the point.

 

[phanter]

you can use mixed drives just fine with mirrors. plan your sizes out s your vdevs are as closesly sized as possible and then just replace them and/or add new vdevs. this would be *by far* better than the frankenstein you are imaging that would become a nightmare.

No it wouldn't. It would just waste drive space in many cases

likely not. this is so far outside a supported truenas design that there is absolutely no point in using truenas. defeats the point.

Why should truenas care if it is using partitions instead of drives?

 

[Arwen]

...
Why should truenas care if it is using partitions instead of drives?

Simple.

Some of the stock TrueNAS GUI functions would not work, like disk replacement. TrueNAS does not keep track of a disk used multiple times through different partitions. So you try and replace a disk, just to get another, (or more), disk's partition failing as you pull one out. If not properly planned, you could reduce redundancy to nothing. Or out right cause data loss.

During disk scrubs ZFS would not know that a "disk" is used by multiple vDevs or pools even. Thus, queue up more I/O than expected, causing extreme head movement. ZFS now has an optimized scrub function based on disk block offset. This is to reduce the overall time needed to perform a scrub. But, with 2 vDevs in the same pool sharing a disk, it's head movements could be so much that scrubs would not only be slower, but also make the pool less usable.


I am not trying to talk anyone out of using disk partitioning with ZFS. My own home computers, miniature desktop, laptop and miniature media server all use 2 storage devices, with partitions & ZFS. In my case, I use a piece of each media for a mirrored root pool, (aka OS and my home directory). Then the rest mirrored for a dumping ground. Since I am not intending to have 10s of Tera-bytes of storage, it's not been a problem.

So the 40GBs or so for my OS pool does not really impact the other pool much. Only during OS updates is their any concern and at those times I am not using my dumping ground pool. So, less conflict. Plus, I don't use the same storage device in the same pool more than once.


Of course, in reality, TrueNAS DOES partition the data disks, one for minimal swap and the rest for a data pool. Now the swap is not expected to be used, or used commonly. Further, having a little bit of extra assigned space allows disk replacements where the new disk is slightly smaller than the existing disks. This would normally cause problems. But, you can work around it by deleting the swap partition on the new replacement disk.

 

[artlessknave]

No it wouldn't. It would just waste drive space in many cases

truenas is an appliance designed for reliability and relative simplicity. it is FAR simpler to have dedicated boot drives separate from dedicates data drives. the "wasted" space is functionally dedicated to reliability, with a bonus of making the boot drive last longer due to SSd wear levelling and overprivisioned size.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

to be clear, I am not saying you can't do this; I am saying do not bother trying to do it with TrueNAS. it is a waste of time and defeats most of the purpose of using TrueNAS in the first place - you will be constantly fighting the GUI for control of the disks, constantly working around the structure of the apliance and constantly at risk of weird happenings. most of the people who try mangling things like this come crying to forums about inevitably lost data. if you DO try this, you're unlikely to find much help here if things go wrong, because it is highly and actively discouraged. here be dragons

the recommendations are recommended for a reason; we care that your data is safe and accessible.

if you want to make something custom, thats completely fine; use bare linux or freebsd and webmin/cockpit, or something like OMV.

 

[WI_Hedgehog]

Did you have success with your setup?
How is the performance?
And are you willing to share your script?

Yes, it was up and running great with mirrored 2.5" boot spinners, and (5) 2.5" spinning data drives. It was something like 5 TB and incredibly fast, but I needed more storage and added (8) 3.5" HDDs.

Then I started diving into drive analysis because it wasn't well explained here and the tools seemed underutilized. Backblaze had done some great analysis, though diving into their data it looked incomplete and I felt important factors external to the data sets were in play.

I poured over member reports of drive failures, data loss, storage configurations, and also what systems didn't have problems. I bought test equipment and learned how to use it as related to servers then built 3 test rigs. Then I started watching international tech conferences discussing the core of system design.

What I discovered was many undiagnosed problems are due to electrical noise, followed by premature aging, and the rest are primarily to using unreliable hardware. User undereducation and bugs also minimally contribute.

Electrical Noise:

• Few people use Uninterruptable Power Supplies, and you should. Transient power fluctuations are like diabetes, a silent killer.
• Hard-core hardware veterans realise power filter capacitors age and they replace them in power supplies and mainboards, but unfortunately most people do not.
• Cheap cables are great. Poor coonectors and shielding are not. Buy good cables (on the cheap, hopefully) and treat your cables like they're the most delicate thing in the world and they'll serve you much, much better.
• Cheap power supplies are not great. Do your research and only buy quality power supplies (and stop eating fast food--that stuff is similarly bad for your body). If the foundation is bad the building will fail.
• Buy mainboards that support ECC memory and buy quality ECC memory. This results in far fewer OS-related issues and cuts support overhead like...well, that's why I get paid so much, because I freed up the IT Support funds and the company can afford to give me a portion of them (and are actually happy to do so given how much money is saved).

Premature Aging:

• #1 Cause: Heat. Power filter capacitors, hard drives, almost everything lasts longer if systems are kept uner 32C. My research (which is quite limited) shows this is the major factor Backblaze failed to take into consideration.
• Unreported HDD failure turned out to be my company's #2 source of "something not working right." (#1: unfiltered power on workstations) Even though we're running [ungodly expensisve] Enterprise RAID packages on Enterprise hardware, the problems related to HDD data corruption is not being reported at the level it needs to be.
• The solution is simple in concept but harder to impliment: use more cooling and better designs and planning.

Unreliable Hardware:

• Remember when Intel was over-priced and AMD saved the day? I do, and I'm a huge AMD fan. Cyrix and a few other companies were also great in the 386 days, but those days aren't today. I dove into ECC heavily and Intel--they are so far and beyond everyone else and at reasonable prices I'd be foolish to run anything but. All our systems are now Intell and ECC, saving LOTS of support costs.
• Realteck: Wow.The TrueNAS forum contributors are really to thank for the whole hardware section--just go read that and stick to it. I cannot thank members here enough for their efforts related to researching and documenting reliable hardware, including @jgreco who threw me into the deep end of the pool on that topic. YOU probably have favorite brands and such--divorce yourself from that, read the resources and advice related to hardware, and buy responsibly. --or don't, that's your choice, and your money.
• Virtual Machines: You gamers and VM guys are killing me. I'M a VM guy, but not on my TrueNAS servers--DON'T DO IT! I run TrueNAS instead of Ubuntu because data retention [without corruption] is vitally important to me, which is why I bare the expense of running a ZFS system (with all the overhead ZFS adds). I could roll my own ZFS system, but TrueNAS makes things so fast and simple it's kind of crazy not to use it. At the end of the day I need to get the job done, and TrueNAS does that, reliably.

Somewhere along this journey I decided to script a bunch of stuff, add SSD boot drives, and see if I could fit more storage (a lot more) in my main TrueNAS box. This involved custom brackets and cabling and...

So some of you are saying, "Would you please stop chasing young office hotties and get back to work." --I don't see this as likely happening; it's Summer here and it's best to enjoy the weather and company now as people hibernate the rest of the year.

Others are saying, "Would you please post more stories of chasing young hotties and let the IT stuff slide for a while." Also not likely. While I live near the lakefront, enjoy great food and drink, and am dashingly handsome thanks to not letting my good looks go to pot, none of this is free. The upper-crust finally decided to put in a high-end VOIP system and specifically demanded i stay out of it in order to save money. Well, let me tell you how that project is going--complete disaster on every front at every turn. That impacts my job as communications is indeed important in business, and for as wealthy as I should be at this point thanks to my generous income, as I said I like to spend it about as fast as I make it, so I actually have to remain employed and currently that means participating in more Executive Meetings than any well-run business should have. (though Elaine does have a layover in Milwaukee starting at 3PM today, so there's that...)