BUILD Please valid my build plan / hardware for FreeNAS 8.3 / ZFS

[jeeps]

Hi, folks

I happened choose to build a backup and streaming media system (priority is backups) using Windows Home Server 2011. I chose to go with multiple RAID 1 arrays because I couldn't trust the failures caused by the RAID controllers (not to mention the cost involved in getting a good one) and other reasons you all know, if I go with RAID 5 or 6. I ordered all the parts listed below on Black Friday! While researching in the last couple of days, I learnt about FreeNAS 8.3 and the support for ZFS 28. I wish I did this home work before I ordered the below parts :(

List of parts purchased:

• Intel Core i5-3570K Quad-Core Processor 3.4 GHz 4 Core LGA 1155 - BX80637I53570K (Ivy Bridge)
• ASUS SABERTOOTH Z77 LGA 1155 Intel Z77 HDMI SATA 6Gb/s USB 3.0 ATX Intel Motherboard
• Corsair Vengeance 16GB (2x8GB) DDR3 1866 MHZ (PC3 15000) Desktop Memory (CMZ16GX3M2A1866C10R)
• Fractal Design Define XL Black Pearl w/ USB 3.0 ATX Full Tower Silent PC Computer Case
• Samsung 840 Series Solid State Drive (SSD) 250 sata_6_0_gb 2.5-Inch MZ-7TD250BW
• Corsair Professional Series AX 850 Watt ATX/EPS Modular 80 PLUS Gold (AX850)
• Western Digital Red 3 TB NAS Hard Drive: 3.5 Inch, SATA III, 64 MB Cache
• Lite-On LightScribe 24X SATA DVD+/-RW Dual Layer Drive IHAS424-98

Except the Fractal Design case, none of the parts are opened out of the boxes, so I could return it (without feeling guilty :) )

Some additional details of my plan:

• Greatness of ZFS too appealing, hence the change of heart and want to go with FreeNAS.
• I plan on having a RAIDZ2 using 4 3TB drives initially and grow it. I don't have much data to begin with and I prefer to be safe than sorry, hence RAIDZ2. I envision having no more than 12 3TB hard drives in the array.
• I think 16GB of RAM is a lot for what I will be doing, but I don't plan on enabling deduplication for now just to keep things simple and to take baby steps... possibly in future.

Greatly appreciate if some of you could review my questions below and provide your feedback:

• Please confirm if my above hardware config makes sense for my plans and for FreeNAS 8.3 / ZFS.
• I have read that ECC RAM is recommended for ZFS. As you can see, I already purchased DDR3 non-ECC RAM. Please let me know if I should really go with ECC. Then I can return what I have and buy ECC memory.
• Between a 8GB USB thumb drive and a small (40GB?) Intel SSD, please let me know which is recommended for FreeNAS OS. Though I have purchased a 250 GB SSD for OS partition, I will be happy to return it.
• Motherboard has 4 SATA III ports and 2 SATA II ports. I plan on using them before I add a SATA PCIe card. Do you think it’s a recommended approach? If not, please suggest a card I should go with. (FYI, Motherboard also has 2 eSATA ports in the back)
• I have bought an i5 processor. Do you think I am better off spending a few buck now and getting an i7?
• If I want to upgrade from RAIDZ2 to RAIDZ3 in future, is it easy to do without affecting the stored data?

I am newbie here, so apologies if I am asking some basic questions.

Thank you very much!

 

[jgreco]

You've got a mix of questions that are somewhat addressed in the forums and some not quite, so I'll give you quick answers and expect you to employ some search-fu if you want more details.

Greatly appreciate if some of you could review my questions below and provide your feedback:

• Please confirm if my above hardware config makes sense for my plans and for FreeNAS 8.3 / ZFS.

• 
  
  No deal breakers noted
  
  

  [*]I have read that ECC RAM is recommended for ZFS. As you can see, I already purchased DDR3 non-ECC RAM. Please let me know if I should really go with ECC. Then I can return what I have and buy ECC memory.

  
  ECC isn't mandatory; many people use Atoms, for example, which don't support it. Only some combinations of hardware support it. You get 90% of the value of ECC by simply testing your memory with memtest86 for a week. But for max data protection, yes, ECC.
  
  

  [*]Between a 8GB USB thumb drive and a small (40GB?) Intel SSD, please let me know which is recommended for FreeNAS OS. Though I have purchased a 250 GB SSD for OS partition, I will be happy to return it.

  
  8GB thumb
  
  

  [*]Motherboard has 4 SATA III ports and 2 SATA II ports. I plan on using them before I add a SATA PCIe card. Do you think it’s a recommended approach? If not, please suggest a card I should go with. (FYI, Motherboard also has 2 eSATA ports in the back)

  
  Is fine. Hard drives do not exceed SATA-II speeds anyways. For more ports get IBM M1015 and cross-flash to IT or IR mode. Google.
  
  

  [*]I have bought an i5 processor. Do you think I am better off spending a few buck now and getting an i7?

  
  No. Or yes if you want to do something proc-intense, like compression. If you want maximum Samba speeds, faster core speeds are key.
  
  

  [*]If I want to upgrade from RAIDZ2 to RAIDZ3 in future, is it easy to do without affecting the stored data?
  [*]

  
  NO IT ISN'T. Build your pool in the basic manner you want it. It is easy to build a 6-disk RAIDZ2 vdev and then add another RAIDZ2 vdev later - that's just adding a device. It's not really possible to change an existing vdev though.

 

[fracai]

• I plan on having a RAIDZ2 using 4 3TB drives initially and grow it. I don't have much data to begin with and I prefer to be safe than sorry, hence RAIDZ2. I envision having no more than 12 3TB hard drives in the array.
• I think 16GB of RAM is a lot for what I will be doing, but I don't plan on enabling deduplication for now just to keep things simple and to take baby steps... possibly in future.

ZFS isn't a Drobo, so you can't grow your array by adding another drive to your your RAIDZ2 4 x 3TB (6TB useable) to get 5 x 3TB (9TB useable). You could add another vdev (of multiple drives) to your pool and eventually have something like 3 x RAIDZ2 4 x 3TB for 18TB total, but you'd be more efficient in having 2 x RAIDZ2 6 x 3 TB for 24TB. There's also the issue of the optimal arrangement of RAIDZ1, RAIDZ2, etc. 4 drives in Z2 is good so you're fine, but keep in mind that it's best to build the system you want, not what you'll start with. At least take the time to consider how you want the pool to eventually look and the path that can get you there. IE. you don't want to end up with one vdev of 4 x 3TB and then 8 more singular drives, your pool would loose all protection.

And you almost certainly do not want dedup. What's your use case for this NAS? Most data won't dedup anyway and you'd be better of just using compression.

Oh, and if you don't mind keeping that SSD, you could use it for ZIL and L2ARC.

 

[paleoN]

Motherboard has 4 SATA III ports and 2 SATA II ports. I plan on using them before I add a SATA PCIe card. Do you think it’s a recommended approach? If not, please suggest a card I should go with. (FYI, Motherboard also has 2 eSATA ports in the back)

Looks like the 2 gray ports are from ASMedia ASM1061. Personally, I would stick to the Intel Z77 brown/black ports, but the others may work just fine.

But for max data protection, yes, ECC.

+1

 

[jeeps]

Hi, jgreco / fracai / PaleoN

Thanks a bunch for the quick responses! You guys are amazing!!

I didn’t get to catch up enough on the FreeNAS / ZFS enablement before I posted my queries, so I wanted to do little bit of that before I respond to you guys, hence the delay. I still have a lot to catch up obviously!

Glad to know that there aren’t any major obstacles in the parts I have purchased. Although I would say that, if I planned ahead, I would have taken the Supermicro chasis route, basically as a server infrastructure. For the money I am spending here, if you really think that’s what I should do, I might keep the Fractal Design case (use it to build a desktop in future), return the rest of the parts and take the server route. Please feel free to say so :)

ELSE, here is what I am thinking, based on your feedback:

• Will use 8GB USB thumb drive for OS. I have an old spare Lenovo Desktop, which I will use as a test machine in the next couple of days.
• Play it safe and go for the i7 processor, instead of i5.
• I am surprised to know that I can’t grow a RAIDZ2 array (again, need to catch up). So, I will go with 6 x 3TB RAIDZ2 and add another vdev later, which won’t be in the foreseeable future in my case. 12 TB is plenty for me.
• Your feedback on not using dedup helps confirm my thoughts on it.
• Yup, you are right! The two gray ports are ASMedia. Now that I am going with a 6 drive array, I will have to use up all the 6 SATA ports.
• SSD for ZIL and L2ARC: I am thinking 240GB SSD is an overkill for the purpose, so I may return it and go for a smaller capacity SSD in future.
• I will certainly heed your advice and go with ECC memory. However, I am finding it difficult to find memory that matches the Asus Sabertooth Z77 motherboard. Although I found the below item on Newegg, no compatible memory stick came up in the searches on the Crucial and Kingston sites for the motherboard. I am a little lost here :( Unless you guys think I should go the Supermicro route, I will call Crucial and Kingston on Monday. Crucial 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1333 (PC3 10600) Desktop Memory Model CT102472BB1339

Appreciate your valuable input!

Thanks & Regards

 

[jeeps]

FYI - If the recommendation is to go with SuperMicro motherboard, I am yet to figure out if I can use Fractal Design full tower case I have or the only choice is to go with a SuperMicro case. If I have to buy a SuperMicro case, I wouldn't mind and I prefer to do the right thing to begin with.

 

[jgreco]

The board you've picked seems to have some nice features (eSATA, USB3) you might have a hard time finding on a server board. I'd be cautious about the registered DDR3 memory, as Sandy/Ivy stuff doesn't support registered (I want to say at all but I'm not sufficiently certain).

That said, you are probably best-served by deciding whether you want to go the "decent high end PC" route (which is IMHO where you are right now, and that is perfectly fine if you want to go that route!) or if you want to go the actual server-grade route, which may not bring a whole lot of value to the table, but the server-grade chassis route may offer a more satisfying way of dealing with drives. I'll also mention there are the "cheap" rackmounts like the Norco, which offer an intermediate option as well, if you really wanted just the features of a rackmount case while retaining some of the nice features of that board, that might be the route.

But if you really want to do ECC and you don't want a lot of frustration, if you truly want "server", you can optimize for that strategy by going server-grade from the ground up. The SuperMicro X9SC* series is a respectable low-end board. Get yourself a mid-range Xeon processor (we like the E3-1230 Sandy's, haven't gotten around to the Ivy's just yet). They're a bit more expensive than the i5's but lower power, and they Do ECC Just Fine. The SuperMicro X9SCL (C202) or X9SCM (C204) both cost substantially less than your board, nearly offsetting the cost of the more expensive Xeon CPU. And you end up with a board whose PCIe slots are not "only compatible with video cards" as ASUS has sometimes claimed. Kingston's KVR1333D3E9SK2/8G kits (2 x 4GB) or KVR1333D3E9SK2/16G kits (2 x 8GB) both work in these boards. My guess is that doing a 16GB Ivy Xeon E3-1240 works out to a similar cost to your platform.

 

[paleoN]

While ASUS may say that board "works with" ECC RAM what does that mean? The board will physically function with ECC DIMMs or does it include the reporting part as well? Then the question is does FreeBSD properly support whatever they are using for reporting?

I was assuming if you went with ECC RAM you would switch to a server class board which also means a different processor. A number of desktop Intel's do not support ECC RAM.

 

[fracai]

• I am surprised to know that I can’t grow a RAIDZ2 array (again, need to catch up). So, I will go with 6 x 3TB RAIDZ2 and add another vdev later, which won’t be in the foreseeable future in my case. 12 TB is plenty for me.

Just to clarify, you can grow the array, just not an individual vdev by adding drives. If you start a vdev with 3x1TB drives, you can grow it by sequentially replacing them with larger drives, waiting for the resilver after each replacement. You just can't change the number of drives in the vdev.

 

[jeeps]

Hi, jgreco

Thanks again for your quick response! After the dust has settled, here is what I am planning to do -- Stick to the “decent high-end PC” as you have put it – 1) just because I already have all the parts I need for the project and I am one of those guys who prefers to return things if I really have to, 2) By going through the pains of researching the server parts I need and exchanging them, the gain isn’t as much. The big gain is probably ECC memory support.

Please note that, I don’t quite need the flexibility offered by the hot swap bays in a case. In fact, I prefer no hot swap bays because I have small children at home – irrespective whether there are keys to the case or not. Moreover, the Fractal Design case I have has 10 nice HD bays and it’s overall a good design and best of all, it’s supposedly very quiet! And, guess what? I got it for just $110 on Newegg on Black Friday.

Having said that, if I were to do it again, server route would be my only choice!

FYI, I came across the below motherboard. Seems like a good choice and it has 10 SATA ports onboard and other bells and whistles.
SUPERMICRO X9SRA Single Socket R (LGA 2011) E5 ATX Workstation/Server Motherboard

I will go ahead and return the 250GB SSD and upgrade the i5 processor to i7:
Intel Core i7-3770 Quad-Core Processor 3.4 GHz 4 Core LGA 1155

The other decision I still need to make is if I wanna spend more money and go for 8 3TB drives, instead of 6 in a RAIDZ2 config (18TB capacity instead of 12TB). I am not into converting DVDs to movies and store them, but a couple of friends offered to let me copy their collection ;) I still don’t see myself crossing 12TB anytime soon, hence the dilemma. I wish the wonderful ZFS offered the flexibility of adding drives!


Hi, PaleoN
I have learnt in the last 48 hours that, there’s less support for ECC memory in Intel desktop line, even compared to AMD. I plan on backing-up the critical data in another desktop with RAID 1 config. Rest will be all movies, which I don’t plan on backing up in another system. To me, it’s not that critical and worth spending so much money to have another backup of those terabytes. I am sharing my mind to see if you guys have any thoughts, although partly it’s a personal choice.

For this system, I have made up my mind to stick to non-ECC memory, if there is no ECC memory available for the Asus motherboard. I plan on calling Asus and Kingston to see if there is ECC support and proper error correction.