Hardware for 72TB 24Bay Media NAS / Hardware and ZFS questions

[MaxRackmount]

I'm going to soon build my first Freenas Build for homeuse to mainly store movies, tv shows and server them for up to 4 XBMC clients. So Performance Isn't the highest priority here.
Main Goal is high capacity and good enough redundancy for home use.

Hardware Spec:

CPU: Intel Xeon E3 1230 V2
RAM: 32 GB EEC Ram (4x8 GB, which is the maximum that E3 series can handle)
Board: Intel S1200BTLR Server Board (http://www.intel.com/content/www/us/en/motherboards/server-motherboards/server-board-s1200bt.html)
NIC: 2x Onboard Intel 82574L Gigabit
Raid Cards: 3x IBM M1015 flashed to LSI 9211-8 IT mode
HDD: 9x 3TB Seagate Baracuda ST3000DM001 in Stage1 (8 in the case and 1 cold spare) and later extend to 16 and then 24 drives.
Case: 24 Bay Sas/Sata Backplane 4U Rackmount Case
USB: 8GB Sandisk cruzerfit

My idea is to create a pool with one vdev of 8 drives in a RaidZ2 config. and then later add another Vdev of another 8 drives when the storage get tight and later the last 8 drive Vdev to bringt it up 24 drives in the end which should leave me with a usable capacity of 54 TB. every Vdev will have its own dedicated Raid card.

i've read that for Raid Z2 6 or 10 drives should be in one pool for best performance but with 6 drives in a pool i loose to much usable capacity and 10 drives seems a lo insecure for Raid Z2 and with 10 drives i could only create 2 vdevs because there is not enough space for a 3rd Vdev of 10 discs in my case. so i would prefer to go with 8 disc vdev even i loose a little performance.

my bigger concern is the Ram here. i can only squeeze 32GB Ram on the board and my usable storage will be 54 TB which means i should have 54 GB of Ram according to the thumb rule of 1 GB for each 1 TB of storage. But actually i will never demand more then 32 GB of data at once to serve the XBMC clients with 1080P streams.

Another question i have about the ARC cache is does it load for example a movie of 10 GB in the Cache when a client start streaming it so that the discs can go back to sleep after its copied in the cache to save energy and increase the Life of the discs? i would also consider to ad a SSD as L2Arc cache if it help to discs to have less power on time but from what i have found L2Arc is not really helpful for streaming.

Thanks in Advance
regards

Max

 

[jgreco]

Relatively speaking, what you're describing is a pretty solid system that would be adequate to handle a lot more than what your intended use scenario suggests.

I doubt you'll have any performance problems. Even with your less-than-optimal vdev sizes. However, I will point this out:

With 8 3TB drives in RAIDZ2, that vdev will be about 18TB of space (~16.5TB of actual space and ~13TB of usable space to maintain 80%-or-less utilization under ZFS, which isn't mandatory, for the remainder of this, I use HDD marketing sizes). With 3 identical banks of vdevs, that's 54TB of space.

If you were to reorganize that: With 10 3TB drives in RAIDZ2, a vdev would be about 24TB of space, two of them would be 48TB of space. If you really wanted that last 6TB of space, you add a 4-drive RAIDZ2 vdev and you're still at 54TB, but you now have more optimally-organized vdevs. You fell into the trap of not realizing that you could use a different-sized vdev.

I wouldn't worry too much about RAM. You'd probably be fine with 16GB. So the thing to do might be to buy 2x8GB, see how it flies for you, and then you have the option to bump it up. ZFS is able to take advantage of large amounts of RAM, and more RAM usually leads to more performance, except when it doesn't, and getting all hung up over it isn't worthwhile.

As for prefetching, no, not that lucky. ZFS is designed for general-purpose computational use. Very handwavy-speaking: The file-level prefetch speculatively prefetches a few extra blocks. If those are accessed, then ZFS will increase the number of blocks and prefetch those. It eventually works its way up to something like 256 or 1024 blocks, don't really remember, but there's a fixed limit. If your application can predict that it is going to stream, it could issue an async read command to force the system to pull it all into buffer, and assuming you had sufficient RAM or L2ARC, it would then be able to avoid hitting the disks further. If your streaming software can be tweaked to do that, then more RAM or L2ARC is a great thing.

But spinning disks up and down is an "expensive" process, it decreases the life of the disks if you do it more than necessary. Be cautious about being overly aggressive when spinning disks down.

 

[MaxRackmount]

thanks for your comment Jgreco.

if i would do your suggestion with a 10 drive Raidz2 than one vdev would be connected to 3 different controllers (2 controller with 4 discs and 1 controller with 2 discs) if i would like to keep my drives in an organized group in the case.

What will happen if the controller with the 2 discs goes bad / offline? my guess is the vdev will continue operating because a 2 disc dropout shouldnt be a problem for RaidZ2. but what happen when the controller with the 2 drives goes back online? The entire vdev will be resilvered or just the new written data since the drives droped?

what will happen if the controller with the 4 discs goes offline? will my vdev be destroyed or will the Vdev go offline and just wait and the 4 droped discs are reconnected?

My Idea was if all drives of the vdev are connected to one controller only and the controller goes offline the entire vdev will be offline but since the discs cant be accessed nothing should happen to my data.

For the Prefetaching its a pity that ZFS doesn't support it. but ZFS should store frequent accessed files in the ARC right? i have about 2 GB of thumbnails of my media collection which are requested frequently from all streaming clients and pull those many files from disc drives is always really slow.
I will look into your mentioned option with async after i finish my build and do some testing.

for spinning down the discs came from the specification from seagate because the baracuda drives are only rated for 8 hours power an / day and most of the day no data will be accessed from my Nas.

 

[cyberjock]

I understand your thought on what to do if a controller fails. Lets do 2 scenarios. Pretend you have 4 disks on a RAIDZ2.

Scenario 1: All 4 disks are on 1 controller.
Scenario 2: 2 disks each are on 2 controllers.

If I had a choice, I think scenario 1 is better. Here's why:

If a controller fails in scenario one your zpool goes offline. You replace your controller like a good little admin and the server is right back up where it was. You do a scrub for good measure, but all is well.

If a controller fails in scenario two your zpool stays online, but 2 disks go away. You have no redundancy for your zpool. Like a good little admin you replace the bad controller, but now you have to do a scrub with effectively no redundancy. You kinda sorta do have some redundancy because all of the disks are back online, but those 2 disks that were offlined earlier aren't completely in sync with the rest of the zpool.

Scrubs are hard on disks, and sometimes disks fail during scrubs. So you're taking a bigger risk with scenario 2 IMO. The only downside to scenario one is that your server is likely to go offline from a controller failure as opposed to scenario two.

For the Prefetaching its a pity that ZFS doesn't support it.

I'm not sure why you think prefetching isn't supported. The manual clearly states that it is supported if you have enough RAM. It may not be supported in the way in which you hoped, but ZFS wasn't designed for media servers ;)

 

[MaxRackmount]

I understand your thought on what to do if a controller fails. Lets do 2 scenarios. Pretend you have 4 disks on a RAIDZ2.

Scenario 1: All 4 disks are on 1 controller.
Scenario 2: 2 disks each are on 2 controllers.

If I had a choice, I think scenario 1 is better. Here's why:

If a controller fails in scenario one your zpool goes offline. You replace your controller like a good little admin and the server is right back up where it was. You do a scrub for good measure, but all is well.

If a controller fails in scenario two your zpool stays online, but 2 disks go away. You have no redundancy for your zpool. Like a good little admin you replace the bad controller, but now you have to do a scrub with effectively no redundancy. You kinda sorta do have some redundancy because all of the disks are back online, but those 2 disks that were offlined earlier aren't completely in sync with the rest of the zpool.

Scrubs are hard on disks, and sometimes disks fail during scrubs. So you're taking a bigger risk with scenario 2 IMO. The only downside to scenario one is that your server is likely to go offline from a controller failure as opposed to scenario two.

scenario 1 was my plan too with 1 controller for one vdev. but all 3 Vdevs will be in one Zpool. or will it be better to create a separate zpool for every vdev / controller? The Safety of my data is me more important then performance.

I'm not sure why you think prefetching isn't supported. The manual clearly states that it is supported if you have enough RAM. It may not be supported in the way in which you hoped, but ZFS wasn't designed for media servers ;)

sorry i meant its a pitty that ZFS doesn't support prefetching for media streaming ;)

 

[cyberjock]

My friend's server has a 24 port controller and currently has 2 vdevs of 8 disks each in 1 zpool. He'll likely upgrade by adding 8 more. Overall, whatever you do with your disks, your performance will be bottlenecked BIGTIME by your NIC. My friend's configuration does something like 500MB/sec+ to the zpool. Anything above about 150MB/sec is overkill because 1 NIC can't do more than 133MB/sec. For 2 NICs your zpool only has to meet 266MB/sec. That's pretty low demands for 8 drives.

Unless you do something really stupid like try to install a bunch of PCI SATA controllers your zpool will far exceed your theoretical networking maximum regardless of how you break up your data. I prefer 1 zpool only because I want 1 place for all of my data.

 

[jgreco]

scenario 1 was my plan too with 1 controller for one vdev. but all 3 Vdevs will be in one Zpool. or will it be better to create a separate zpool for every vdev / controller? The Safety of my data is me more important then performance.

So your fear is that you're going to run into a controller failure that gives you 8 of 10 drives in the vdev, eliminating redundancy in the vdev? I would think the obvious solution would be to "design it so that can't happen." If your goal is to have 24 drives and 3 8 port controllers, you make each vdev out of 3 ports on one controller, 3 ports on another, and 4 ports on the third. Problem solved, optimal ZFS RAIDZ2 sizing maintained, well, at least for the first two vdevs.

sorry i meant its a pitty that ZFS doesn't support prefetching for media streaming ;)

That's really the application's responsibility. There's no reason a program can't inform the system that the information is going to be needed (and in fact there are underlying API's to do that, both by actually forcing the issue by reading the data, which is guaranteed to get it into any available cache, via madvise/MADV_WILLNEED for more sophisticated programs, etc).

 

[MaxRackmount]

My friend's server has a 24 port controller and currently has 2 vdevs of 8 disks each in 1 zpool. He'll likely upgrade by adding 8 more. Overall, whatever you do with your disks, your performance will be bottlenecked BIGTIME by your NIC. My friend's configuration does something like 500MB/sec+ to the zpool. Anything above about 150MB/sec is overkill because 1 NIC can't do more than 133MB/sec. For 2 NICs your zpool only has to meet 266MB/sec. That's pretty low demands for 8 drives.

Unless you do something really stupid like try to install a bunch of PCI SATA controllers your zpool will far exceed your theoretical networking maximum regardless of how you break up your data. I prefer 1 zpool only because I want 1 place for all of my data.

i know that my performance will be bottlenecked by the NIC but as i always mentioned i dont really care about the performance at all because i mainly do streaming from this server which will hardly peak at 20-30 MB/sec.

Most important for me is the safety of my media library because there are years of work involved in ripping and organizing my media collection so i dont want to loose it. Keep a backup of everything at a 2nd place is not an option as this would be to costly for home use. Extremly important data is off course backed up to external drives and the cloud.

the reason i use 3 x M1015 8 port controllers is because they were recommended here in the Forum and they are much cheaper than a single 24 port controller.

My question is what will happen if i have 3 vdevs of 8 discs in one zpool and one controller goes bad and bring down one vdev in the pool? According to your Guide the entire Zpool should go offline and the data is not available. So after i replace the controller and bring back the failed vdev the entire Zpool will be back online without any lost data correct?

 

[cyberjock]

My question is what will happen if i have 3 vdevs of 8 discs in one zpool and one controller goes bad and bring down one vdev in the pool? According to your Guide the entire Zpool should go offline and the data is not available. So after i replace the controller and bring back the failed vdev the entire Zpool will be back online without any lost data correct?

In theory that is how it should work. Keep in mind that your controller CAN corrupt the data being read/written to the disk. The data is passing through your controller. It's not likely, but not in the realm of impossible.

If your data is very important to you make backups!