RAID Setup Help - Home Movie Server

[ecronic]

Hi Everyone,
I'm new to using TrueNAS - my previous NAS setup being on a Buffalo Linkstation LS-QVL. The hardware failed and after some searching around I found TrueNAS as the best solution with a PC with appropriate hardware.

I've made sure the hardware is at par for the setup I'm planning on by reading through the manual. But I'd really like to know the best raid setup I can have for my setup.

Goal: Watch movies over the home network and also share with friends and family over the internet via Plex Server.

Current Setup/Data:
1. Plex Server - NVIDIA Shield
2. TrueNAS Core - TrueNAS-12.0-U2.1
3. PC Hardware - HP Pro 3500 - Intel(R) Core(TM) i5-3340 CPU @ 3.10GHz, 12GB DDR3 RAM(will be upgrading to 16GB), 64GB msata SSD(OS), 4x4TB(16TB) WD NAS Red HDD.
4. Network - TrueNAS Core connected to the main router directly via Gigabyte Ethernet & CAT6 cable.
5. Data - 1x3TB 93% full.

I am very new to saving data or even setting up RAID configuration(s). My previous setup on Buffalo was on 4x3TB HDDs WD Black as individual data drives without redundancy. I've learnt it the hard way to have a backup plan to save the pain of recovering the data lost.

So, I ask all you experts with the above mentioned hardware and goal(s) what would you recommend as the best setup so I can have enough storage space for adding more and also keep redundancy for it.

thank you very much for you time.

Regards
Dev

 

[lpwevers]

Install TrueNAS on the SSD. Then I'd use RAID 5 on the 4 WD disks. That will leave you with 12TB storage for your movies. And 1 drive may fail.

 

[Etorix]

Thre is no "RAID 5" with TrueNAS, that would be RAIDZ1.
And actually there no longer is a reasonable certainty that RAIDZ1 could rebuild data after the loss of one drive in a multi-terabyte array.

Why RAID 5 stops working in 2009

The storage version of Y2k? No, it's a function of capacity growth and RAID 5's limitations.

www.zdnet.com

Considering you have only 3 TB of data at this point, and appears to be wary about data loss, I'd suggest a RAIDZ2 with the four 4 TB. That's only 8 TB worth of raw storage (of which about 6 TB could be used before having to increase capacity or incur degraded performance), but RAIDZ2 can definitely sustain the loss of one drive. (A backup is still strongly advised, tough.)

Safety checks: Are the drives directly attached, without a RAID controller? Are the new drives of the CMR type?

 

[ecronic]

Safety checks: Are the drives directly attached, without a RAID controller? Are the new drives of the CMR type?

Thanks for replying. Yes, I'll be attaching the drives directly via sata cable without a RAID controller. I'm getting the WD RED which is a CMR type. The RED Plus & RED Pro are SMR.

Regards
Dev

 

[danb35]

And actually there no longer is a reasonable certainty that RAIDZ1 could rebuild data after the loss of one drive in a multi-terabyte array.

...but we're 12 years later, and guess what? Single-parity RAID still works most of the time. The "RAID5/RAIDZ1 is dead" thing borders on FUD, assuming that a single URE on the remaining disks dooms the entire array--which just isn't how ZFS works. No, recovery isn't guaranteed--nor is it guaranteed with RAIDZ3--but there's a pretty fair chance of recovery, and for something like bulk media storage it would probably be safe enough. But even with all of that said, I wouldn't recommend it either--though for four disks, I think I'd favor mirrors over RAIDZ2. This blog gives some points to consider on that:

ZFS: You should use mirror vdevs, not RAIDZ. – JRS Systems: the blog

I think this blog post overstates many of the points, but they remain valid points to consider. For four disks, I'd probably put them in striped mirrors, so I could easily add another pair to expand the pool in the future.

 

[ecronic]

For four disks, I'd probably put them in striped mirrors, so I could easily add another pair to expand the pool in the future.

Isn't striped mirrors used for a pair of disks. So, do you mean that for 4 disks I should create a pool with 2 sets of striped mirrors - each containing 2 disks? Sorry if it sounds dumb, but as I mentioned not really familiar with Raid setups.

 

[Patrick M. Hausen]

I'm getting the WD RED which is a CMR type. The RED Plus & RED Pro are SMR.

Are you sure you did not switch the two disk types here? The RED Pro are definitely CMR and recommended specifically for ZFS workloads by WD marketing.

 

[danb35]

Isn't striped mirrors used for a pair of disks

A mirror is a pair of disks (or possibly more than a pair; you could have a three-disk, four-disk, or whatever width mirror you want). Striped mirrors would be two or more mirrors, striped together--minimum of four disks--in a single pool.

 

[ecronic]

A mirror is a pair of disks (or possibly more than a pair; you could have a three-disk, four-disk, or whatever width mirror you want). Striped mirrors would be two or more mirrors, striped together--minimum of four disks--in a single pool.

And what about redundancy? How does that work with this set up?

 

[ecronic]

Are you sure you did not switch the two disk types here? The RED Pro are definitely CMR and recommended specifically for ZFS workloads by WD marketing.

Sorry, you are correct. I got it the other way. My bad on that one...

 

[danb35]

And what about redundancy? How does that work with this set up?

Each mirror is redundant; if one disk fails the other still has your data. If both disks of a mirror fail, your pool goes away. In this regard, RAIDZ2 does have better redundancy. It hasn't been my experience that two disks fail simultaneously, especially with regular SMART testing and with alerting properly configured.

 

[jgreco]

...but we're 12 years later, and guess what? Single-parity RAID still works most of the time. The "RAID5/RAIDZ1 is dead" thing borders on FUD, assuming that a single URE on the remaining disks dooms the entire array--which just isn't how ZFS works. No, recovery isn't guaranteed--nor is it guaranteed with RAIDZ3--but there's a pretty fair chance of recovery, and for something like bulk media storage it would probably be safe enough.

The real issue here is whether it is acceptable to lose redundancy.

ZFS is a filesystem without recovery tools such as fsck. Its nearly exclusive strategy for avoiding damage to the pool is to make sure that all of the data is redundant and handled carefully in the software. You can definitely reduce the likelihood of things like unrecoverable read errors (URE) if you have an aggressive SMART self-testing regimen, but I've had people tell me that my three times a week long tests are "excessive". No, it's not excessive, sequential background reads are cheap and fast compared to something like a scrub, which doesn't check empty sectors anyways. But there are things such as temperature issues or power events that can cause disk problems, so having additional redundancy can be a saving factor. It's like seatbelts in a car. It's an added expense (or was in the '70's) and an additional constant inconvenience, right up to the point where it stops you from being ejected through the windshield, at which point your perception of value changes.

I sleep pretty well knowing my important data is stored on RAIDZ3 with a spare, replicated across the country to another RAIDZ3 pool with a spare.

 

[Nakedape]

I sleep pretty well knowing my important data is stored on RAIDZ3 with a spare, replicated across the country to another RAIDZ3 pool with a spare.

Quick off-topic question if I may:

Assuming for the sake of the question that your pool on the other side of the country is not part of a grander scheme--ie it's a backup server dedicated for this pool--is there then any reason not to use another TrueNAS instance for that box? Would you still prefer TrueNAS as OS even for the sole task of receiving snapshots from the primary server? I'm trying to plan my own backup scheme and toyed for a moment with the idea that I could simply use whatever OS I wanted on the backup server as long as it played well with ZFS and simply do ZFS send/rcv as necessary. But with scrubbing and perhaps more that I have not yet studied I realized there probably was no OS (espescially with me behind the wheel) better capable of taking care of my data than TrueNAS. Seems self-evident for my own case as I type this now, but I guess I'm wondering if this holds true even for those of you with 10k+ posts on this forum? I.e., is whatever it is TrueNAS does to keep the data safe easily replicated on e.g. plain FreeBSD for those of you who actually know what you're doing most of the time?

 

[Etorix]

My assumption, not being an IT expert with 10k+ posts here, would be that TrueNAS does nothing more than FreeBSD to keep data safe (it's the same OpenZFS everywhere) but does provide an interface to administer the systems and setup replication, scrubs, SMART tests, etc. that is much more convenient than writing scripts and cron jobs. Having TrueNAS for the main NAS, it would not made sense to use anything else for the backup NAS and have to deal with two different interfaces.

 

[Patrick M. Hausen]

Assuming for the sake of the question that your pool on the other side of the country is not part of a grander scheme--ie it's a backup server dedicated for this pool--is there then any reason not to use another TrueNAS instance for that box? Would you still prefer TrueNAS as OS even for the sole task of receiving snapshots from the primary server?

Yes, of course. That's precisely what I do. Replicate all jails, VMs and share datasets to another TrueNAS system. See "Home NAS" and "Backup NAS" below.

 

[pschatz100]

To my mind, for a media server where maximizing performance is not critical, a 4 disk Raidz2 will be preferable because it allows any two disks to fail before losing data, whereas a pool made up of two mirrors could fail if both disks in the same mirror fail.

Admittedly, I have never had two disks fail at the same time, but I have had disks fail and always felt better knowing that I still had some redundancy while rebuilding the pool.

Of course, any RaidZ implementation is not a substitute for a proper backup.

 

[danb35]

a 4 disk Raidz2 will be preferable because it allows any two disks to fail before losing data, whereas a pool made up of two mirrors could fail if both disks in the same mirror fail.

True. The counterpoint in favor of a mirror is that it resilvers much more quickly, which in principle would make it less likely that a second disk (particularly the "wrong" second disk) would die while this is ongoing. And, of course, it's easier and cheaper to safely expand a mirrored pool.

There are pros and cons each way. Personally, with four disks, in the stated application, I'd probably favor mirrors for the ease of expansion. With six disks, RAIDZ2 all the way.

 

[jgreco]

Quick off-topic question if I may:

Assuming for the sake of the question that your pool on the other side of the country is not part of a grander scheme--ie it's a backup server dedicated for this pool--is there then any reason not to use another TrueNAS instance for that box? Would you still prefer TrueNAS as OS even for the sole task of receiving snapshots from the primary server? I'm trying to plan my own backup scheme and toyed for a moment with the idea that I could simply use whatever OS I wanted on the backup server as long as it played well with ZFS and simply do ZFS send/rcv as necessary. But with scrubbing and perhaps more that I have not yet studied I realized there probably was no OS (espescially with me behind the wheel) better capable of taking care of my data than TrueNAS. Seems self-evident for my own case as I type this now, but I guess I'm wondering if this holds true even for those of you with 10k+ posts on this forum? I.e., is whatever it is TrueNAS does to keep the data safe easily replicated on e.g. plain FreeBSD for those of you who actually know what you're doing most of the time?

My assumption, not being an IT expert with 10k+ posts here, would be that TrueNAS does nothing more than FreeBSD to keep data safe (it's the same OpenZFS everywhere) but does provide an interface to administer the systems and setup replication, scrubs, SMART tests, etc. that is much more convenient than writing scripts and cron jobs. Having TrueNAS for the main NAS, it would not made sense to use anything else for the backup NAS and have to deal with two different interfaces.

So, the thing is, basically the answer @Etrorix provided.

I use FreeNAS for ZFS storage pools, because all the stuff that should be there is actually there. This certainly includes all the scaffolding for scrubs and SMART tests. But there's another aspect, which is that some significant effort has been put into Samba integration, which I've always found to be a nightmare.

Any idiot can set up an NFS server, and I have lots of single-purpose NFS servers set up as virtual machines on top of ESXi for a wide variety of purposes (see, I said, any idiot can set up an NFS server). These are generally only serving numbers in the less-than-TB range, and are protected by hypervisor RAID storage, and UFS/FFS for a filesystem, so setting them up is really VERY simple. Using FreeNAS for this would actually be inconvenient because most of them get away on 1 core and 256MB RAM (that is correct). I've got hosts in inventory with 32GB or less of memory, so using 8GB and at least two cores for a virtualized FreeNAS would be very heavyweight. FreeNAS isn't the fix for everything.

But if you want to handle a bunch of hard disks, or offer SMB/AFS, etc., FreeNAS brings a click-and-go experience to the table that isn't something FreeBSD itself is able to provide, plus, of course ZFS.

Anyways if I wasn't clear in my original post: The ZFS pools on both sides are FreeNAS. I use rsync for replicating between them because there are occasionally updates that push out terabytes of data, and I can propagate those easily on a mass storage device shipped from one location to another.