Long answer, but I read a post somewhere:
I have a general question regarding your opinions on using ZFS mirrors vs a RAID-Z(n) scheme for a home media server I will be building in the very near future.
Specifically, my use case will involve serving DVD movie and audio CD data inside as well as outside my home LAN from inside a jail(s), and I may also install an www/owncloud server for personal file syncing in a separate jail.
The server will be mainly for read-only usage, and not a whole lot of concurrent writing past the initial data uploaded to it is expected. This could of course change in the future. All media data will come from disks I legally purchased so while redundancy is important, most of the data can be re-uploaded to the server resulting only in my time should a failure happen. All other important data would be backed up accordingly. The total data will most likely occupy no more than 4 TB initially, and there will only be a couple of users other than myself accessing the server. The OS will be installed on a separate disk(s) and HBA to separate it from the data should a future failure occur with either. The OS version installed will be FreeBSD 10-STABLE as I prefer to source update to add new features as needed, and to compile in security updates within a timely fashion. I am aware of the pros and cons of using STABLE. I would prefer a flexible setup that would be at least modestly easy to administer (as far as the filesystem(s)), and expand should I need to. Monetary flexibility is also an important consideration of course although not first priority.
Given the information and short summary above of my use case for the tentative server, I'm under the assumption from what I have read here on the forums and elsewhere, that multiple ZFS mirrors would work best for me as opposed to a RAID-Z(n) scheme. However, I have also read plenty of conflicting opinions to this (mainly regarding IOPS and redundancy, which I'm really not sure I need to focus on given my use case and expected load). I'd very much appreciate anyone with a similar use case and/or experience to weigh in with their opinions or options to consider regarding which ZFS scheme to use before I green light my build and go ahead with the setup given this is my first actual server build. I'll be happy to provide any other pertinent information needed which may have been left out above. I'm probably over-analyzing things, and it may really not matter in the end, but I would like to be a little more confident in my choice before moving ahead.
Thanks in advance for your time.Last edited by protocelt on 05 Jan 2014, 18:04, edited 2 times in total.
protocelt
Member
Posts: 126
Joined: 19 Jan 2013, 02:43
Location: 25.0000° N, 71.0000° WTop
--------------------------------------------------------------------------------
Re: ZFS mirrors vs RAID-Z(n) scheme for home media server.
by phoenix » 03 Jan 2014, 17:29
Which is more important to you: raw throughput or raw storage space? If throughput is more important, then go with multiple mirror vdevs. If storage space is more important, then go with multiple RAID-Z2 vdevs.
Either way, go with multiple smaller vdevs instead of one or two giant ones. Try to keep the number of drives in a RAID-Z vdev under/around eight.Freddie
Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
phoenix
MFC'd
Posts: 3368
Joined: 17 Nov 2008, 05:43
Location: Kamloops, BC, CanadaTop
--------------------------------------------------------------------------------
Re: ZFS mirrors vs RAID-Z(n) scheme for home media server.
by protocelt » 04 Jan 2014, 15:16
Thank you phoenix, that does simplify things quite a bit.
I am much more concerned with total storage space and expandability.
One last question. Hopefully it isn't too far off topic. Having said the above, would it be advantageous to add an L2ARC device(s) to the pool(s)? I have four new Intel 520 120 GB SSD drives I can use for such, but will save them for other uses if not really needed. I'm under the impression that adding L2ARC devices would really only be useful if RAM usage is being maxed out by the ZFS ARC, and/or no more RAM can be added to the server. I will be installing 128 GB of RAM which should be well more than enough for the default ZFS ARC on my 4 TB of data I think.
protocelt
Member
Posts: 126
Joined: 19 Jan 2013, 02:43
Location: 25.0000° N, 71.0000° WTop
--------------------------------------------------------------------------------
Re: ZFS mirrors vs RAID-Z(n) scheme for home media server.
by ralphbsz » 05 Jan 2014, 06:40
What is your performance requirement?
If you are serving music, you need less than a MB per second (way less). For full video, you need a handful of MB/s. Your disk drives can do way over 100 MB/s. I just checked my ZFS file system (mirrored 2-way, on two 5-year old Seagate 1 TB SATA disks that spin at 7200 RPM), and I'm getting 135 MB/s when reading "large" files (many megabytes), using sequential 1 MiB I/Os. This is on a machine with 4 GiB RAM, an Intel Atom motherboard with built-in SATA (no fancy HBAs here), and on a cold read (data not in cache yet).
I think your performance will be so much better than your requirements that using SSDs for the L2ARC cache would be a waste of perfectly good SSDs.
Now, if you want to simultaneously serve a few hundred video streams, things get interesting. But your setup looks like a home server (the 4TB disk size matches a typical home user's pile of ripped CDs and DVDs, plus baby videos from your camcorder), so massive bandwidth is probably not your issue.
Personally, I like to go with mirrored storage at home. I could be flippant and say that I don't trust that newfangled parity encoding stuff (it's a joke, my day job is developing storage systems), but the real reason is that you can get reasonable reliability with just two disks, which makes configuration and management much easier.ralphbsz
Member
Posts: 149
Joined: 26 Feb 2012, 05:52Top
--------------------------------------------------------------------------------
Re: ZFS mirrors vs RAID-Z(n) scheme for home media server.
by protocelt » 05 Jan 2014, 12:13
Thanks for the feedback ralphbsz.
I should have been more specific in my questions. You are correct in stating (basically) that the hardware setup will be overkill for what I want to use it for right now. My aim is to future proof the server as much as I can. I may run a mail server, multiple VMs, and various other servers from it in the future for various purposes and don't want to worry about upgrading the hardware other than storage space and/or cache devices for quite a while if I choose to do so. Right now my focus is on storage space but of course still want to make sure things are running efficiently. Regarding my question about the L2ARC devices, I gather then at this point of time, it would just be a waste given my present requirements and can be added later as needed. Would that be correct?
protocelt
Member
Posts: 126
Joined: 19 Jan 2013, 02:43
Location: 25.0000° N, 71.0000° WTop
--------------------------------------------------------------------------------
Re: ZFS mirrors vs RAID-Z(n) scheme for home media server.
by ralphbsz » 05 Jan 2014, 16:35
protocelt wrote:
Right now my focus is on storage space but of course still want to make sure things are running efficiently.
Personally, for a home server (which tends to live for a pretty long time, and has changing requirements), I would go for flexibility and easy of management/operation/upgrading. Since with today's disk drives, capacity is usually not an issue (or rather, the cost of extra capacity is so low, it's not worth economizing), I would go for mirrored over parity (RAID-Z etc.).
Regarding my question about the L2ARC devices, I gather then at this point of time, it would just be a waste given my present requirements and can be added later as needed. Would that be correct?
It seems to me the sensible course is to buy solid hardware now (in particular, large enough disk drives), and set it up in a simple fashion, with as few moving parts as needed. You can always add cache devices later, if needed.
Another thing is this: If you ever find yourself in a situation to need a heck of a lot of random I/O (because you're running a very busy web server, or hundreds of VMs, or a busy transaction database, all very unlikely in a home setting), you have two options: either hope that your data all fits into flash (possible, but not guaranteed, and pretty expensive), or get a lot of extra disk drives (preferable fast-spinning drives). That's another reason to not install too much hardware up front, because you don't know what hardware you will need later, and you don't want all your slots filled.ralphbsz
Member
Posts: 149
Joined: 26 Feb 2012, 05:52Top
--------------------------------------------------------------------------------
Re: ZFS mirrors vs RAID-Z(n) scheme for home media server.
by protocelt » 05 Jan 2014, 18:04
Personally, for a home server (which tends to live for a pretty long time, and has changing requirements), I would go for flexibility and easy of management/operation/upgrading. Since with today's disk drives, capacity is usually not an issue (or rather, the cost of extra capacity is so low, it's not worth economizing), I would go for mirrored over parity (RAID-Z etc.).
Thanks. This goes toward the exact question I was trying to answer originally. Please excuse my naivety as I have no prior experience with real server hardware setup or administration. I was unsure whether to go with mirrors over parity to start. Given I don't know when I will be running more applications on the server, I've decided to go with multiple 3-way mirror vdevs to start. That will simplify administration in the interim and I can add additional vdevs in the future as needed. The added throughput is always a welcome bonus as well even if not exactly needed at this time.
Another thing is this: If you ever find yourself in a situation to need a heck of a lot of random I/O (because you're running a very busy web server, or hundreds of VMs, or a busy transaction database, all very unlikely in a home setting), you have two options: either hope that your data all fits into flash (possible, but not guaranteed, and pretty expensive), or get a lot of extra disk drives (preferable fast-spinning drives). That's another reason to not install too much hardware up front, because you don't know what hardware you will need later, and you don't want all your slots filled.
True. I don't I see myself in this position, at least certainly not in the near future. I am interested in learning administration much more in depth however, so this setup will be used as my playground as well. This is one of the reasons I will using real server hardware(Supermicro to be exact) as opposed to a good desktop setup that would probably work just as well for serving my media needs.
As further discussion may likely lead off topic, I will mark this thread as solved. Thanks to everyone who offered their feedback. Much appreciated
protocelt
Member
Posts: 126
Joined: 19 Jan 2013, 02:43
Location: 25.0000° N, 71.0000° WTop
--------------------------------------------------------------------------------
Re: [Solved] ZFS mirrors vs RAID-Z(n) scheme for media serve
by throAU » 10 Jan 2014, 03:25
I went with striped mirrors (2-way, I'm reasonably confident I can replace a failed drive in an acceptable turn-around time and the data isn't that critical. Critical stuff is in OwnCloud which is synced to other devices anyhow) for my home setup mostly due to expandability - I can add another mirror VDEV when my pool gets to say 70% full, and then when I run out of space for new VDEVS I can replace them one by one to expand each VDEV as I need more space. Yes, you lose a bit of capacity, but it is so much easier being able to upgrade by adding/replacing VDEVs with 2 drives rather than the 5 or 6 required (?) for RAIDZ2. Especially when my NAS is a 4 bay HP N54L
No, performance is not ideal the way I am doing it, but it is plenty (saturates GbE on sequential read and write with my current setup of 2x1 TB and 2x500 GB. And that's with a sector size problem in the 1 TB mirror (it's FreeNAS and just picked up on it with a recent FreeNAS upgrade). I plan to fix that by replacing the 1TB drives with 2 TB drives and using the 1TBs to replace the 500 GBs (the current version FreeNAS should fix the block size on them when the 1 TBs are wiped to replace the 500 GB drives, I believe? This pool was originally built with version 8 over a year ago).
Yes, unbalanced VDEVS = not ideal and also different sized VDEVs = not ideal, but these are drives out of previous desktops I have upgraded from and I always used to run a mirror in those, so they were already on hand at the time.
In actual use, the performance (mostly streaming media as with the OP) is plenty for my gigabit network. I'm pretty happy with it. Also - it looks like that on read at least, ZFS will read from both drives in a mirror in parallel for better throughput. i.e., I was getting 100 MB/sec over the network (LAN limited) out of a single 1 TB, 2 drive mirror and the disks were only showing as 50% busy.
Summary: I don't think IOPs will be a problem for your purposes irrespective of which RAID type you use, but striped mirrors are more flexible come expansion time.
edit:
Oh, and this machine currently only has the 2 GB RAM it shipped with. It's actually only recently installed to replace my old Q6600 desktop machine with 6 GB, as basically the PSU and case fans in the old box were knackered. An entire N54L chassis + CPU + 2 GB RAM (included) was only $250 and draws less power, has ECC, is smaller, quieter, etc. I just popped the old disks and USB flash out of the old box, installed in the new one, changed the network driver and was back up in 30 minutes
The box can handle 16 GB of RAM (which I intend to fit), but even with 2 GB (way less than recommended) performance is still reasonable - I thought it would be worse and don't feel a desperate need to upgrade it throAU
Member
Posts: 912
Joined: 05 Jan 2012, 05:37
Location: Perth, Western AustraliaTop
--------------------------------------------------------------------------------
Re: [Solved] ZFS mirrors vs RAID-Z(n) scheme for media serve
by protocelt » 10 Jan 2014, 14:18
Summary: I don't think IOPs will be a problem for your purposes irrespective of which RAID type you use, but striped mirrors are more flexible come expansion time.
Thanks. After further research, feedback, and contemplation, I agree. At least for the time being. I can upgrade/alter the storage later as it requires. The hardware setup will be a scaled down and somewhat altered version of the setup Terry_Kennedy used in his updated RAIDzilla2 project(Which BTW, thanks for putting that article up Terry. It was interesting and very informative. Hope you don't mind me linking to it . ) as I have access to most, but certainly not all, of the same hardware for use. At this point, it's overkill for sure, but will scale as my needs/requirements do.
protocelt
Member
Posts: 126
Joined: 19 Jan 2013, 02:43
Location: 25.0000° N, 71.0000° WTop
--------------------------------------------------------------------------------
Re: [Solved] ZFS mirrors vs RAID-Z(n) scheme for media serve
by throAU » 15 Jan 2014, 01:04
Another benefit of striped mirrors:
I've been running FreeNAS, and since before it automatically detected correct ashift (and my drives lie, claiming to be 512 byte), I had some 4k sector drives in my pool (2 drives in a mirror) with ashift 9. This can not be corrected without destroying the pool - you can't change a VDEV's ashift after creation, and you can't remove a VDEV. You'll run into the problem also if you have ashift 9 on 512 byte sector drives (i.e. your ashift is CORRECT, currently) and want to replace them with new drives in future. Or in the far future where we're changing from 4k sectors to 8k or whatever.
I have 4 drives in 2x 2 drive mirrors. If I was running RAIDZ1, I'd need to back up to external media, destroy the pool, re-create and restore - or I'd need to add enough drives to replicate the VDEV layout in a new pool (which means spending).
Because only some of the data on the pool is critical (and the rest would just be inconvenient to lose) and I'm running 2 mirrors however, I've gotten away with it (last night) by detaching 1 drive from each mirror VDEV, re-creating a replica of the pool using the single disk VDEVs I freed up with the correct ashift, and replicating the data via zfs send to the new pool.
No Gigabit Ethernet to slow things down, no significant downtime waiting for data to get on and off the box, etc.
Once the data has copied across (had to go to work this morning and I'm pretty sure I got all the filesystems but didn't have time to check), I'll just destroy the old pool, attach the freed drives to the new pool's single disk VDEVs as mirror VDEVs and let it resilver.
Yes, there's a window of risk for approximately 5 hours or so whilst my data copies from the old pool to the new pool, and again whilst waiting for the pool to resilver once the old pool's disks are attached, so YMMV and if you have sufficiently fast backup media it may be quicker to just backup destroy and restore. But for me (home media pool) it was the least annoying alternative.
