Three drive setup

[OJ2k]

I managed to get my grubby hands on 3x 4TB drives for a home file store and can’t afford to buy more for quite a while. I have a gut instinct on which configuration would be best for me, but would appreciate comments and corrections.

Relevant points:

• This is for home use only. But I really do value my data.
• I’ll be doing regular rsync to another Linux box (ext4) to act as a backup.
• My data is currently about 2.5 TB. I don’t see it exceeding 3TB in the next year, but I like the principle of snapshots.
• I’m not very concerned about performance, as most access will be over wifi.
• Two of the drives are from the same batch (HGST), the other is WD Red.

Options:
2 drives in mirror, plus one on the shelf as a spare.

• + Less power
• - If one drives dies I’m at risk of bit rot or failure to rebuild
• - Only 4 TB space despite using 2 drives

Three-way mirror

• + Super safe?
• - More power
• - Only 4 TB space despite 3 drives

RaidZ1

• + Would give much more space
• - If one drives dies I’m at risk of bit rot or failure to rebuild

RaidZ1 + copies=2 (not sure if this option would work)

• ? An overly complex way to get the same usable space and safety characteristics as a 3-way mirror?

Gut feeling is a 3-way mirror. Any views?

 

[Robert Smith]

The last option is good against bit-rot, but gives no added protection against drive failure over RaidZ1.

I agree, three-way mirror—for safety.

 

[titan_rw]

If you have regular backups like you mention, I might consider a 3 disk Z1.

Alternatively, what about a 2 way mirror with the 3rd disk as a single disk pool to replicate to. True, the replication destination has no redundancy, but I'd probably prefer this to a 3 way mirror. Even better if the 3rd disk could be in a different computer. This way you have two separate pools that both contain your data, while the primary pool also has redundancy. But if you already have an separate computer backup on ext4, this might be a bit overkill.

 

[OJ2k]

Sounds like I'll accept the loss of space and give a 3-way mirror a go. I'm particularly attracted to the idea that, while learning, I might be able to 'borrow' one of the mirror disks and take it to another PC to play 'recovery'.

@titan_rw can you comment or point to something on the benefit of a second pool, assuming it were on the same computer? I'd guess it wouldn't provide means for the first pool to heal bitrot (being merely a replication target), so there must be some other benefit like if I accidentally wipe out the first pool or something.

 

[anodos]

You could always set up a mirror and not use the third disk till you have saved enough for one more disk. Once you have purchased a fourth disk, create a second mirrored vdev and stripe them (i.e. do RAID10). In the mean time you will have a disk standing by in case one of your mirrored drives fails.

 

[titan_rw]

The first pool would have redundancy, so it should already be protected from bitrot. And protected from a failed drive. But not both at the same time (bitrot while a drive is being replaced). That's where either the 3 way mirror, or a secondary pool that's a replicated copy of the first would help.

I'd just save up for the 4th disk, and do a 4 disk z2 pool. Good redundancy (can recover from bitrot during disk replacement), decent capacity (2 disk). Much better redundancy wise than a pair of striped mirrors. Both have the same capacity, but the striped mirrors can't help you in the case of bitrot during disk replacement. The 4 disk z2 has 2 disk redundancy anywhere in the pool, vs 1 disk redundancy in each of 2 mirrors. But of course you need the 4 disks upfront.

 

[pschatz100]

Please help me understand the recommendations in this thread. I know what bit-rot is and everything I read suggests that, on modern disks which do error correction on the fly, the probability of bit-rot actually affecting data is pretty small. I would have thought that the probability of having a problem while resilvering a disk in a 3 disk Raidz array would be vanishingly small.

Given that the OP is already backing up to an external system (a very good idea,) it seems to me that a 3 disk mirror is major overkill.

 

[titan_rw]

That's why in post 3 I mentioned that with backups, I'd consider a z1 setup.

The issue with any single parity setup is that during disk replacement you're relying on the remaining disks operating 100% perfectly, no bad sectors, no silent corruption, etc. If there's any 'hiccup' at all during disk intensive resilvering there's no recovering. If there are issues, hopefully it only affects certain files, and you can recover the affected files from backup. Worst case is there's issues with the entire pool due to metadata corruption or something, and you're stuck recovering the entire pool from backup. It this likely to happen? No. Can it happen? Sure. With backups though, at least it's possible to fully recover.

 

[diedrichg]

Food for thought: If you don't want to rebuild your pool to Z2 later, you could build a 4 disk pool with the 4th being a smaller drive, just keep in mind that the entire pool would look like it was the size of the smaller drive*4. You could then later swap out the smaller drive with a matching drive and you'd be back up to 4*4TB and you wouldn't have to go through the hassle of rebuilding your pool.

 

[OJ2k]

Food for thought: If you don't want to rebuild your pool to Z2 later, you could build a 4 disk pool with the 4th being a smaller drive, just keep in mind that the entire pool would look like it was the size of the smaller drive*4. You could then later swap out the smaller drive with a matching drive and you'd be back up to 4*4TB and you wouldn't have to go through the hassle of rebuilding your pool.

Perfect. I was wondering how to get away with something like that. I've got a third smaller driver lying around somwehere.

 

[pschatz100]

When using disks of different sizes, the capacity of a 4 disk Raidz2 will be two times the formatted capacity of the smallest disk. Exactly the same capacity as a three disk Raidz1. The only difference is in the amount of redundancy.

Raid performance begins to suffer when you use more than 80% of available capacity, so if you plan on storing up to 3TB of data, then you really want at least 3.75 TB of formatted capacity. Don't forget that a 4TB drive has less than 4TB formatted capacity.

Bit-rot is a non-issue. With modern disks and ZFS, the probability of bit-rot affecting your data is so small that it takes special techniques just to measure it. One of the reasons that ZFS and RaidZ is becoming popular is its improved reliability over hardware raid when recovering degraded arrays. There are far greater things to worry about.

The 3-way mirror is your safest bet, no doubt. A 2-way mirror with the third disk as external backup is also a good choice. I run this configuration, with the external backup running at 1:00 am in the morning, each day. In this way, I have the extra redundancy and a convenient way to recover specific files when I do something stupid - such as modify a file and save it over one I really want to keep. The entire system is backed up to an external source as protection against a catastrophic melt-down.

I presume you have your server running on a UPS, right?

 

[OJ2k]

I presume you have your server running on a UPS, right?

I was saving up for that 4th disk, but UPS probably makes more sense as a next step. I'm struggling to identify a decent brand (APC/Cyberpower?) UPS which is small and cheap yet fully compatible. My current 3 drive system peaks at 91 Watts on startup and consumes 52 W at moderate load. I'd only need enough capacity to power down safely in the event of a cut.

I'll spend some time doing a proper forum search....

I don't know much about the issues affecting FreeNAS/ZFS in the event of a power cut. I would assume that journaling would prevent a individual write operation failing half way through, but that 'atomicity' wouldn't be true for file integrity as a whole. Any links on the subject appreciated.

 

[diedrichg]

I love my Cyberpower 1000 with the LCD. Highly recommended!

 

[OJ2k]

I love my Cyberpower 1000 with the LCD. Highly recommended!

Cyberpower Value 1000EILCD? Looks pretty perfect if it's supported.

 

[diedrichg]

It is on both accounts. I like that it tells me how many outages I have had as well as the current watt consumption.

Edit. Sorry. No, wrong one. Here is the one I have with my 4-drive NAS: CyberPower CP1000PFCLCD UPS 1000VA / 600W PFC compatible Pure sine wave

 

[OJ2k]

Just to round off my questions about UPS. I had difficulty getting a Cyberpower which was definitely compatible and reasonably priced in the UK. In the end I got an Eaton Ellipse Eco 650 which appears to be working very well.