Stripe a 6 disk vDev RaidZ2 or start over with a new 12 disk vDev?

[67RS427]

I'm sure it's been asked before and I did try to search but I have a few other questions as well... I'm a TrueNAS newbie, I just built my first NAS after having a hardware raid controller fail on me in another server. So, totally newbie to ZFS and I have a few questions.

My TrueNAS server:
Supermicro SSG-6028R-E1CR16T, X10DRH-CT Motherboard
Dual Xeon E5-2640 V4, 64GB ECC RAM (4 x 16GB PC4-2400)
IBM M5110 HBA (LSI 9207-8i IT Mode)
Boot: 2x SP A55 256GB SSD
Storage: 6x WD 14TB Ultrastar DC HC530 (WUH721414ALE6L4) - RaidZ2
NIC: Dual 10Gbit LAN w/ Intel X540

This is in a home storage environment, mainly used for storage of large files (media/video for Plex) with added storage for some documents/file backups and some VM vmdk files for a VM I run from a Win 10 machine to do my downloads with SabNZB and serve as my main Plex server. I know I could use TrueNAS for Plex & SabNZB but I'm not ready for that yet.

I started with 6x WD 14TB Ultrastar DC HC530 (WUH721414ALE6L4) hard drives configured in RaidZ2. This gave me about 50.74 TB of usable space or so which is already 60% full and I still have a lot of data to move over. I still have at least another 21 TB of data to move over and that doesn’t include another 20-30 TB that I’m still trying to recover from my raid array that lost its card. So let’s say I need 70 TB or so with room to expand in the future.

Speed doesn’t seem to be an issue for the 6 disk RaidZ2 that I’ve been testing. So far I’ve been able to get 700MB/s to 1GB/s write speeds to my TrueNAS over my 10 Gbit network without really making any optimizations. Running the VM that’s stored on the TrueNAS seems speedy enough for now.

My big question is, should I add another 6x 14TB disks and stripe my existing RaidZ2 or should I start all over with a 12 disk RaidZ2? Since my network speeds are so fast, starting over isn’t a big deal. The question I have is more related to 1 vDev vs 2 vDev performance (with large 14TB disks) and also the amount of time it would take to resilver after replacing a bad disk when the vDev is pretty full. I haven’t seen too many posts about resilvering when it comes to large/full drives. I could care less about having 1 or 2 vDevs since Plex doesn’t really care and I can manage it just fine with Datasets.

Questions:

1. Resilvering time with a 6 disk RaidZ2 vs 12 disk RaidZ2 using 14TB Disks and 70-80% full?
2. Would adding more RAM help my resilver times or am I just limited to the disk speed?
3. Swap size on large 14TB disks? Currently set at 2GB…
4. System Data Set - It’s currently stored on my vDev, should it be on my boot drive?
5. I’m running dual boot drives (SP A55 256GB Sata SSDs)… Is this a waste? They were cheap so I figured having a mirrored boot drive couldn’t hurt.
6. Should I use a zVol for my download drive and/or media drives instead of a dataset? Would it improve performance for unzipping/unpacking large files? I think it might make things easier for when I’m moving files from a temp folder/zVol to my media folders so I can drag/drop to “move” instead of “copy” a file without holding shift in Windows.
7. Are there any “must have” plug ins for TrueNAS? What is everyone else using for file backups?

I know that’s a lot to ask in one post, I really do appreciate anyone that took the time to read all of this and any help or knowledge you can pass along. It’s much appreciated!!!

 

[sretalla]

Resilvering time with a 6 disk RaidZ2 vs 12 disk RaidZ2 using 14TB Disks and 70-80% full?

Very long and even longer.

Would adding more RAM help my resilver times or am I just limited to the disk speed?

Not much can be done to speed it up. RAM certainly won't hurt, but don't expect a change.

Swap size on large 14TB disks? Currently set at 2GB…

Fine

System Data Set - It’s currently stored on my vDev, should it be on my boot drive?

Can be... that will help take a small amount of load off your data pool if your boot device is capable enough (i.e. an SSD.. or even better a mirror of them)

I’m running dual boot drives (SP A55 256GB Sata SSDs)… Is this a waste? They were cheap so I figured having a mirrored boot drive couldn’t hurt.

Not exactly a waste, but more than really necessary.

Mirrors only really help to reduce downtime in case of a failed drive in the boot pool otherwise, it's pretty easy to restore a config to a rebuilt boot pool.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

Should I use a zVol for my download drive and/or media drives instead of a dataset? Would it improve performance for unzipping/unpacking large files? I think it might make things easier for when I’m moving files from a temp folder/zVol to my media folders so I can drag/drop to “move” instead of “copy” a file without holding shift in Windows.

Mounting block storage would put you in a different situation that you're not prepared for with a RAIDZ pool... https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

Are there any “must have” plug ins for TrueNAS? What is everyone else using for file backups?

For you, I would say Plex and SAB... your current setup is a terrible way to do it.

File backups are best with replication tasks if you have a good target for it. Cloud sync tasks also can get the job done if you have the bandwidth.

 

[67RS427]

Very long and even longer.


Not much can be done to speed it up. RAM certainly won't hurt, but don't expect a change.


Fine


Can be... that will help take a small amount of load off your data pool if your boot device is capable enough (i.e. an SSD.. or even better a mirror of them)


Not exactly a waste, but more than really necessary.

Mirrors only really help to reduce downtime in case of a failed drive in the boot pool otherwise, it's pretty easy to restore a config to a rebuilt boot pool.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

Mounting block storage would put you in a different situation that you're not prepared for with a RAIDZ pool... https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/


For you, I would say Plex and SAB... your current setup is a terrible way to do it.

File backups are best with replication tasks if you have a good target for it. Cloud sync tasks also can get the job done if you have the bandwidth.

Thanks for the reply... So I know resilvering times will vary based on disk performance and file types but how long are we talking here? 24 hours per disk or several days? Any real world examples?

Since I have a mirrored SSD for my boot drive should I then move my System Data Set?

My setup might not be perfect for SabNZB currently and I can migrate that over to the TrueNAS but I don't think I have an option for Plex right now. Most of my content is 4K and almost none of these Xeon processors support Intel QuickSync (zero for my FCLGA2011-3 socket). A lot of my content locally is direct played so no big deal but I still want the ability to transcode video if needed. Even with Dual Xeon E5-2640 V4 CPUs, I don't think the TrueNAS is a good solution as a Plex server since it already uses all of the RAM for storage and would put a huge load on the CPU while transcoding, even with 40 CPU cores.

Bottom line, what's my best option for expanding this RaidZ to 70TB+ and eventually more down the road? I don't care about having to re copy my data but if resilver times are reduced by days, I will run multiple 6 disk vDevs instead of one large 10-12 disk vDev. All of the articles and info I've seen on resilver times are 5-8 years old.

Is there any reason I might need to add more RAM for my use case? Large file storage with a bunch of SMB shares on 70TB+ of storage. Just wondering if I should add another 32 or 64GB...

 

[sretalla]

So I know resilvering times will vary based on disk performance and file types but how long are we talking here? 24 hours per disk or several days? Any real world examples?

I think I posted some a while back, but can't seem to find it.

In any case, in a relatively full (70%) VDEV of 8 10TB drives in RAIDZ2 it took around 24 hours to resilver 1 disk... for 14TB and higher % used in 6 wide, you would probably be looking at at least 24 hours too, you can expect much longer the closer it gets to full.

Since I have a mirrored SSD for my boot drive should I then move my System Data Set?

No reason not to.

My setup might not be perfect for SabNZB currently and I can migrate that over to the TrueNAS but I don't think I have an option for Plex right now. Most of my content is 4K and almost none of these Xeon processors support Intel QuickSync (zero for my FCLGA2011-3 socket). A lot of my content locally is direct played so no big deal but I still want the ability to transcode video if needed. Even with Dual Xeon E5-2640 V4 CPUs, I don't think the TrueNAS is a good solution as a Plex server since it already uses all of the RAM for storage and would put a huge load on the CPU while transcoding, even with 40 CPU cores.

I think you might find plex would transcode fine if you give it a small RAM drive as the temp location.

Bottom line, what's my best option for expanding this RaidZ to 70TB+ and eventually more down the road? I don't care about having to re copy my data but if resilver times are reduced by days, I will run multiple 6 disk vDevs instead of one large 10-12 disk vDev. All of the articles and info I've seen on resilver times are 5-8 years old.

adding the VDEV is good if you have the option to... seems you do.

Is there any reason I might need to add more RAM for my use case? Large file storage with a bunch of SMB shares on 70TB+ of storage. Just wondering if I should add another 32 or 64GB...

More RAM is always better, but as I mentioned about plex, a temp space for transcode on a RAM drive really helps. (it needs to be as big as the largest file you want to be able to transcode... I use about 30GB out of my 128GB for it)

 

[67RS427]

I think I posted some a while back, but can't seem to find it.

In any case, in a relatively full (70%) VDEV of 8 10TB drives in RAIDZ2 it took around 24 hours to resilver 1 disk... for 14TB and higher % used in 6 wide, you would probably be looking at at least 24 hours too, you can expect much longer the closer it gets to full.


No reason not to.


I think you might find plex would transcode fine if you give it a small RAM drive as the temp location.


adding the VDEV is good if you have the option to... seems you do.


More RAM is always better, but as I mentioned about plex, a temp space for transcode on a RAM drive really helps. (it needs to be as big as the largest file you want to be able to transcode... I use about 30GB out of my 128GB for it)

So after reading through the iSCSI thread you posted, it looks like I should be moving my VMs to a separate mirrored SSD pool of some sort. I could also just keep it on the client side as well since it's not critical to have a real time backup and I can run it off of an NVMe locally and it would be much faster.

Thanks for the insight on the resilver times, that's about what I expected and I can live with 24hrs or so, just not several days. I don't think I will expand my pool to 10 or 12 disks since adding another 6 disk pool makes no difference to me except maybe all of lost space efficiency. It does kind of suck to limit myself to 70% of 2 separate 56TB pools instead of 70% of a larger 140TB pool. Some quick math tells me I'm losing almost 20TB of the pool for 2x 6 disk vDevs vs 1x 12 disk vDev (losing maybe 13TB of usable space)

I'll run some tests with Plex transcoding and maybe add another 64GB of RAM to help things out. I'd say that 60% of my 4K files are 30GB or less but I do have a lot that are 30-60GB and a few that are 70-110GB files.

Thanks for your help, it's much appreciated! If anyone else has any real world data for resilvering larger RAIDZ2 pools with 12-16TB drives and 10-12 disks I would be really interested to know how long it takes.

 

[Morris]

My big question is, should I add another 6x 14TB disks and stripe my existing RaidZ2 or should I start all over with a 12 disk RaidZ2?

You seem satisfied with performance. Your applications are video storage and playback. Also VMs

In my experience the more drives in a RAID Z or Z2 the higher the CPU demand. You have a lot of processor yet even this can run out. Also, you have put a decent amount of effort to move your video library and now need more storage. Do you want to go through the effort of migrating all that storage? If you create a strip of another 6 drives in RAID Z2 you will double the throughput once both strips have the same amount of space used. Till you get there the performance will be that of the new RAID Z2 drives which is the same as you have now. CPU demand will not go up as much with the two RAID Z2 strips as with one larger one. The cost of the improved performance is two drives and you need to judge the value of this. I've done the experiments and the strip is both faster, less CPU intensive and easier.

Spinning disks in RAID X* is the best solution for lots of bulk storage. For VMs, a pair of mirrored SSDs will provide great performance with redundancy. Migrating your boot off your SSDs to mirrored USB or even a single SSD if you are willing to have some down time with an easy recovery is the way to go.

 

[67RS427]

You seem satisfied with performance. Your applications are video storage and playback. Also VMs

In my experience the more drives in a RAID Z or Z2 the higher the CPU demand. You have a lot of processor yet even this can run out. Also, you have put a decent amount of effort to move your video library and now need more storage. Do you want to go through the effort of migrating all that storage? If you create a strip of another 6 drives in RAID Z2 you will double the throughput once both strips have the same amount of space used. Till you get there the performance will be that of the new RAID Z2 drives which is the same as you have now. CPU demand will not go up as much with the two RAID Z2 strips as with one larger one. The cost of the improved performance is two drives and you need to judge the value of this. I've done the experiments and the strip is both faster, less CPU intensive and easier.

Spinning disks in RAID X* is the best solution for lots of bulk storage. For VMs, a pair of mirrored SSDs will provide great performance with redundancy. Migrating your boot off your SSDs to mirrored USB or even a single SSD if you are willing to have some down time with an easy recovery is the way to go.

I don't mind moving the data again, this was more of a test anyways and I should have probably used a single Dataset for Media with subfolders instead of multiple Datasets to replicate my old drive structures for the sake of making my Plex library migration easier with less duplicate and "new" file entries. I think I've managed to solve the new file entry issues although it will still take a lot of work once it's done (fixing mismatched media that shows up as new instead of syncing with the duplicate file and then me able to remove the old library).

I started with 6 drives just to get a feel for TrueNAS as I'm very new to it coming from hardware raid controllers. The raid set that I'm recovering is a RAID 50 so on par with a striped RAID Z2 (for space efficiency) from my understanding.

I don't really see a lot of load on my CPUs right now and with 40 cores will one large 10 or 12 disk pool really even be noticeable? I was more concerned with the time to resilver but don't really know what the CPU demand is when stepping up to larger 10-12 disk pools.

Also, you said the striped 6 disk RAID Z2 is faster than a 12 disk RAID Z2 but are you talking about reads or writes? Using some basic RAID Z2 calculators, I'm seeing a mirrored 6 disk RAID Z2 gets me 168TB (75-78T usable @ 70% full) of storage with "8x read speeds" but I can lose 2 disks from each set and resilver times will be much quicker. Compare that to a single 12 disk RAID Z2 that shows I'll get "10x read speeds" and 168TB (86-98T usable @ 70% full). So again, that "performance" from the striped set is costing me 11-20TB of usable space. I don't really need the performance for my storage needs but the additional redundancy and faster resilver times are nice to have but so is the extra space lol.

Migrating your boot off your SSDs to mirrored USB or even a single SSD if you are willing to have some down time with an easy recovery is the way to go.

Are you referring to migrating the boot drive of my VM or TrueNAS? I was under the impression booting off a USB is a big NO NO with TrueNAS. The SSDs I'm using are cheap $25 256Gb drives that are mirrored so not sure I would change that unless there is a good reason to. I considered using a SATADOM but the cost to have a larger mirrored SSD boot drive was about the same.

 

[Morris]

Both read and write are faster with strips of RAID Z*

USB boot was the recommended approach a year or two ago. With inexpensive SSDs there more reliable yet I don't know that they are more reliable than mirrored USB sticks.

As you have reasons to migrate your plex data again, you might as well try both ways and see what works best for you. Re-silvering is uncommon and for plex you are not going to notice it happening in the background.

If you want your VMs to run best, move them to SSDs, even inexpensive ones will be much better than hard drives.