Sun recommends not using more than 9 disks in a VDEV. Does this apply to media/backup?

Status
Not open for further replies.

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs-quickstart.html

Note: Sun™ recommends that the number of devices used in a RAID-Z configuration be between three and nine. For environments requiring a single pool consisting of 10 disks or more, consider breaking it up into smaller RAID-Z groups. If only two disks are available and redundancy is a requirement, consider using a ZFS mirror. Refer to zpool(8) for more details.

Somebody on Reddit mentioned this and cited the FreeBSD documentation. I was curious if anybody here knows why Sun recommends this and if having a single VDEV of 12 disks would have any negative side effects when being used as a media storage and file backup server. Is this something like the 80% capacity rule, where it doesn't really apply to a home media server use case?
 

CraigD

Patron
Joined
Mar 8, 2016
Messages
343
So let me clear up some confusion. 12+ disk vdevs are totally "okay" from a ZFS coding standpoint. There's nothing unstable about that (with some exceptions, see below).

Obviously, the more disks you have in a vdev, the more prone you are to multiple simultaneous disk failures in a vdev, which translates to a higher chance of losing the entire zpool. This could be classified as "unstable".

Your iops will be limited to that of 2 disks since you have 2 vdevs. The reason for not recommending large width is because of performance (iops). I understand this is for backup storage only, but even in these cases I don't recommend it. The reason is several fold:

1. Over time, your zpool will fragment more and more. This is the nature of ZFS and since there's no defrag option, you can't fix this at all. This *will* translate to slower and slower performance of your zpool over time.
2. Over time, your zpool will have to work even harder to integrate snapshots. I've worked with people who would send a snapshot, and then it would take an hour or more to integrate a snapshot that was just a few 100GB of data.
3. I've seen people try to do zfs send back to the main system for data recovery purposes, and then they realize that a 10+ day return trip is totally unrealistic and they have no choice but to wait 10+ days for the data to replicate back to the original (production) server.
4. Some of the ways that ZFS is tuned by default has been tested by the FreeBSD community, and virtually nobody tests ultra-wide vdevs like yours. So you can also run into other bottlenecks over time as well, and you're likely to be alone with trying to figure them out (assuming you even can figure them out). Tuning ZFS is crazy hard (even I avoid it whenever possible).

So I hear you about the "iops don't matter to me". If this is for home use and you are okay with simply not having access to your movies for a week or two, go for it. If this is a production server where a 10+ day recovery is totally not an option, then I'd strongly recommend you reconsider vdevs that wide.

I've worked with lots of people that say "iops don't matter" and 3 months later they're asking why things are slow, and when I explain it they realize that they do actually have a lower threshold. I promise you that if iops and fragmentation get badly enough, you will eventually be unhappy with it. ;)

I personally ran a 16 disk vdev for home use just to "see what would happen". Worked great for a few months. No problems at first and I thought that "the man" was holding us back by telling these fables about how much wide vdevs suck. But after a few months it was hard to stream a video and even do things like move a directory without bottlenecking. I was an only user and had these problems.

So my general rule is "just don't do it". Not to sound like a jerk, but in my experience, when people want to go really wide and think that iops won't matter for their workload, they probably aren't realizing how bad it can get. I can promise you won't be the first to think this will be fine, and you probably won't be the last I'll hear from this month either.

I thought the same thing you do... iops don't matter to me. But they eventually can bite you, and once it's bitten you, there is no fix except to destroy the zpool and start over (this is a very time consuming and generally really crappy waste of time). I eventually had to trash that zpool and recreate it because it was totally nonfunctional for me as a single user for home use.

tl;dr: I never, ever recommend wide vdevs. ;)

PS: In my experience, if this is something like a backup for a production server, I'd probably go no wider than about 8 disk RAIDZ2 for performance reasons unless you are totally not concerned about extremely long data restoration times. ;)
 

oRAirwolf

Explorer
Joined
Dec 6, 2016
Messages
55

I really appreciate you finding that and posting it.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
There's no hard limit. Use the rule of thumb along with Matt Ahrens' blog post on vdev widths and your own constraints to determine what's best for you.

As an example, 12-wide RAIDZ2 may be acceptable if your workload consists of large video files with blocksize 1M and you have a 24-disk 4U.
 
Status
Not open for further replies.
Top