Planning a FreeNAS Server for Media storage and streaming

[amitkhas]

For let me start off that I'm a complete noob when it comes to setting up a RAID. I have never done one before, but I have been reading quite a bit about it, which has ultimately led me to this site. I am now planning to build a RAID server, and had some questions about it. Mainly to dispel the conflicting stuff I've read online.

The intention is to build a RAID 5 (5x or 6x 2TB HDs) that will be used as a media server for my music, pictures, and videos. I would like to stream 1080p video from this RAID'd media server. Media server should be Windows friendly.

1. I like the idea of RAID 5 in that you get a fault tolerance while having a lot of storage space. However, I've read that RAID 5 is risky in that sense that corruption can occur and that recoveries often fail. They say that restoring with a RAID 5 can be very difficult, and can fail. Despite having no more than 1 HD fail, it is very possible that the RAID won't be re-buildable. Are these statements true?

2. In the event that you have a 1x HD failure in a RAID 5, how easy is it to rebuild the RAID with FreeNAS? Is it just a matter of shutting down, placing in a new HD, and starting back up?

3. Once the RAID 5 is up and running, how hard is it to expand it? Let's say I fill up the space, and want to expand it. Can I just replace the HDs one by one with larger HDs, and let the RAID 5 rebuild itself each time until I've put in a whole new set of HDs?

4. Having a software RAID 5 is very slow. However, I had trouble determining what "slow" is, and if it matters for my case. I'm not using it for any kind of industrial purpose. It's mainly for home use. I want to store large video files (~15GB) 1080p files. Ideally, I would like to be able to write and read at the same time, with read speeds sufficient for streaming on a home network (my LAN speeds are sufficient as I currently do this with a single HD on a 2.8GHz P4 w/ 2GB ram)

5. What kind of Ram/CPU power would be sufficient for my home use, media needs? Like an Intel cerelon, or an Intel quad-core i7? I was thinking along the lines of a dual core i3 or potentially a i5. Quad core or Dual core? 8GB ram sufficient? or need 16gb preferably?

6. I've read that ZTS is very good in preventing corruption and for monitoring the RAID 5. However, major downfall is that it "slows" the system. Again, what does "slow" mean? Would it be enough to meet my purpose? If so, what kind of hardware would I need?

Thanks a lot!

 

[amitkhas]

Anyone have experiences with RAID 5?

 

[TimHenrion]

I would recommend reading up on ZFS

I would look into ZFS and RAIDZ. ZFS has distinct advantages over just about any other filesystem, not the least of which is total data integrity checking, protection against silent data corruption (bit rot), snapshots, infinitely expandable storage pools (even of dissimilar devices) and a host of other totally cool stuff.

I just took my first foray into FreeNAS building a box based on an HP N40L Microserver after having a Drobo roast my data while migrating a disk pack from one Drobo to another. The N40L has an AMD Turion II embedded CPU with an integral RAID controller in the Southbridge that you leave switched off for RAIDZ. I'm running a 4-drive RAIDZ that is able to to achieve sustained read rates over Ethernet (AFP file service) of up to 100 megabytes/second over gigabit Ethernet. I'm thrilled to say the least to be able to effectively max out gigabit ethernet using this setup.

As to ZFS "slowing things down, as you can see I don't see that happening. To contrast this, I've got a dedicate RAID10 box (which should be faster than RAID5) attached to desktop machine via Firewire 800 that can only achieve 50-60 megabytes per second of sustained read throughput. So basically my dinky little RAIDZ FreeNAS box gets me almost twice the throughput of a Firewire 800 attached dedicated hardware RAID. One of the things that you do need with a ZFS NAS box is a good chunk of RAM for optimization. Read the ZFS docs for guidelines. I have 8GB of ram (an HP N40L tops out at 8GB) and it works fine.

As someone who has dealt with the downsides of proprietary hardware RAID, I'm quite happy to have a ZFS RAIDZ system. It's fast and if the NAS server ever croaks, I can take the drives out and load them onto any other system with ZFS support and recover my data. This is coming from someone who just had a Drobo toast my data when trying to migrate from one Drobo to another after a hardware failure.

For let me start off that I'm a complete noob when it comes to setting up a RAID. I have never done one before, but I have been reading quite a bit about it, which has ultimately led me to this site. I am now planning to build a RAID server, and had some questions about it. Mainly to dispel the conflicting stuff I've read online.

The intention is to build a RAID 5 (5x or 6x 2TB HDs) that will be used as a media server for my music, pictures, and videos. I would like to stream 1080p video from this RAID'd media server. Media server should be Windows friendly.

1. I like the idea of RAID 5 in that you get a fault tolerance while having a lot of storage space. However, I've read that RAID 5 is risky in that sense that corruption can occur and that recoveries often fail. They say that restoring with a RAID 5 can be very difficult, and can fail. Despite having no more than 1 HD fail, it is very possible that the RAID won't be re-buildable. Are these statements true?

2. In the event that you have a 1x HD failure in a RAID 5, how easy is it to rebuild the RAID with FreeNAS? Is it just a matter of shutting down, placing in a new HD, and starting back up?

3. Once the RAID 5 is up and running, how hard is it to expand it? Let's say I fill up the space, and want to expand it. Can I just replace the HDs one by one with larger HDs, and let the RAID 5 rebuild itself each time until I've put in a whole new set of HDs?

4. Having a software RAID 5 is very slow. However, I had trouble determining what "slow" is, and if it matters for my case. I'm not using it for any kind of industrial purpose. It's mainly for home use. I want to store large video files (~15GB) 1080p files. Ideally, I would like to be able to write and read at the same time, with read speeds sufficient for streaming on a home network (my LAN speeds are sufficient as I currently do this with a single HD on a 2.8GHz P4 w/ 2GB ram)

5. What kind of Ram/CPU power would be sufficient for my home use, media needs? Like an Intel cerelon, or an Intel quad-core i7? I was thinking along the lines of a dual core i3 or potentially a i5. Quad core or Dual core? 8GB ram sufficient? or need 16gb preferably?

6. I've read that ZTS is very good in preventing corruption and for monitoring the RAID 5. However, major downfall is that it "slows" the system. Again, what does "slow" mean? Would it be enough to meet my purpose? If so, what kind of hardware would I need?

Thanks a lot!

 

[amitkhas]

Thanks for the help!

Between the FreeNAS guide and your post, I was able to get most of my questions answered.

I did have another question:

One of the criticisms of RAID5 is that it is vulnerable to corruption that is otherwise resolved in RAID6. I wasn't able to find exactly what type of corruptions these were, but they are widespread on the internet. Does ZTS (i.e. RAIDZ) protect against this? I'm familiar with ZTS protecting against the "write hole" issue, but I'm not sure if the posts around the internet are referring to the "write hole" issue with RAID5 that is resolved in RAID6, or if it's another vulnerability. At this point, I've come down to decide between RAIDZ and RAIDZ2.

 

[jgreco]

RAID5 is just as "vulnerable" to corruption as RAID6. Once you lose sufficient redundancy, your data's toast. RAID6 just adds more redundancy.

ZFS is less vulnerable to some forms of corruption because there are additional checks being made on the consistency of the data. You can still lose data with ZFS but it is not likely to be "corrupt" - just gone.

With RAID5, let's say you have four 1TB disks. To understand RAID5, let's simplify things a bit by explaining RAID4: three data disks and a parity disk.

http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_4

You write your first block to disk 0, your second to disk 1, your third to disk 2, and your fourth to disk 0 again. In the meantime you're computing parity and writing it to disk 3.

Now you can lose an entire disk and the data can still be recalculated: losing the parity drive is a no-brainer; you just read the data from the data drives. Losing a data drive requires you to calculate the contents from the other data drives and the parity that you know you should have; you derive the data that way.

But it is not unusual that during that process, you are unable to read at least one other disk block on those other disks. You lose that data, plus the data that would have been computed using that block. That comes out as "corruption." Or disk errors, on a reasonable controller. Or your array is just toast, on a dumb controller.

RAID5 differs from RAID4 in that the parity rotates among the disks, this avoids creating a hot spot because the parity block must be read and rewritten for many data writes, but otherwise RAID5 pretty much equals RAID4.

RAID6 introduces an additional disk of parity. Now you can suffer the loss of two blocks of data and still be able to recover the data.

In actual practice, it does actually happen that you can lose multiple disks, or lose one disk and have read errors on others. RAID6 is therefore more _reliable_ than RAID5.

ZFS goes in a slightly different direction with these concepts, but you can have a RAIDZ or RAIDZ2 pool, which are essentially RAID5 and RAID6 for your storage. However, you can also do things like ask ZFS to store multiple copies of your data, so that perhaps you store your tax returns on a dataset with "copies=3" and your ZFS filesystem will take additional steps to protect the data by writing multiple copies to disk.

No filesystem completely resolves the problems inherent in physical storage devices: they fail, at the worst times, in the most catastrophic ways imaginable. Unless you've protected your data, haha. Because if you're paranoid, and you're using RAIDZ2, and you're saving multiple copies=, and you're replicating onto a backup server, then of course you'll never encounter a disk failure and you'll feel silly for all the paranoia.

 

[amitkhas]

Thanks for the post! Very valuable info :)

Few more questions:

1. As per the FreeNAS documentation, ZAID2 is "slower" than ZAID1. Anyone have any estimations on how much slower? Example numbers would be awesome to give me a frame of reference.

2. Does HDD speed matter? Is 5400RPM sufficient, or 7200RPM really needed? Again, sample read/write numbers would be awesome on a 5400RPM. It'll give me a frame of reference.

 

[jgreco]

RAIDZ2 will be slower than RAIDZ1, in the worst case it's basically because you're doing two parity read/update operations per block. But what's slow? I've got a Microserver N36L writing to four old 7200RPM WD drives in RAIDZ2 and it can write out at 60MBytes/sec sequential. But is that meaningful? What's your workload like? Random reads/writes? Sequential? Parallelism? These are all factors. It's certainly not a real fast fileserver by some measures... try doing parallel operations on it and you will feel the limits imposed by the small number of spindles. An older server with four newer EARS 5400RPM drives in RAIDZ2 tops out at more like 20MBytes/sec write. It's hard to say whether that difference in speed is due to different CPU's, different controllers, etc. It's probably fair to say that *some* of it is due to the difference in disk speed.

7200RPM will serve random transactional workloads better than 5400RPM. It'll also generate more heat, and may tend to fail somewhat more quickly. Which of these variables is meaningful to you is worth thinking about.

 

[amitkhas]

Thanks for the quick reply! Most of the writes will be sequential. At a minimum, it needs to be able to stream 1080p video files. I think the Read speeds should be sufficient for this. As for the write speeds, I'll have to determine if I can settle for 20MB/s. I'd prefer 60MB/s and ideally have 100MB/s.

Thanks again for providing numbers!

 

[amitkhas]

Just to add, if you could, can you take a look at the hardware setup?

http://forums.freenas.org/showthread.php?3682-Hardware-Specs-for-New-Build-(links-included)

 

[jgreco]

For sequential activity, and modest read/write speeds (which is really all that 1080p video files require for streaming), there's no real benefit to 7200RPM drives. If you like to burn money, yes, 7200. If you like to heat your home more, yes, 7200. The main "problem" with 5400 is that they cut costs in every way possible, but to some extent that's true of the 7200's too. 7200 (or faster) tends to help more when you have random workloads.

There are basically a lot of factors other than drive speed that affect all this. For example, we have two virtually identical storage servers that have only one significant difference: one was set up on a 3Ware controller as JBOD, the other was set up as SINGLE disks, so that caching on the 3Ware controller works. That appears to be the difference between 11MBytes/sec write and 19MBytes/sec write. These servers are single core, and so they kind of suck at ZFS... sigh

I've actually got some new hardware coming in later today, I'll have some nice Supermicro X9SCL+-F's and E3-1230's that aren't immediately spoken for, and I kind of intended to spend a little time messing around with FreeNAS on one of them. I may be able to dig up some spare drives of both the 5400 and 7200 persuasion to do some other test that I wanted to do, so maybe I can remember to try some speed tests and post them here. I've been frustrated with ZFS up to this point because most of the deployments I've done with it have sucked for performance, on hardware that rocks for UFS.