30TB FreeNAS Hardware List & Questions

[Mr. Bungle]

Hi everyone - this is my first NAS build and I'm hoping to get some feedback on my hardware choices.

Here's some background info about my usage requirements and patterns:

- My main goal is to consolidate my file storage and provide some redundancy for peace-of-mind. I currently have 9 1.5TB disks in a JBOD configuration (scary, I know), 2 2TB disks in a USB dual external drive, and 2 1.5TB individual 1.5TB USB external drives. My plan is to reuse the 9 1.5TB disks for a separate project, and the rest I can either use for partial backup of my NAS, sell, or use for something else.
- I have ~2TB of widely assorted files that I *really* don't want to lose (music, photos, work-related files, personal files, etc.).
- I have about 16TB (and gradually growing) of video, almost entirely in files of ~3-50GB, which I can reconstruct if absolutely needed - but that would still be a huge time sink so I'm inclined to keep it as safe as I can. I'm not ready to spend the money necessary for a true backup solution, so I'm hoping that RAID-Z2 is sufficient there. I'm aware that RAID != backup, and I think I've gotten a good sense of the tradeoffs and possible pitfalls - but I'm willing to accept a bit of risk with these files to save a good amount of money.
- In most cases, losing an occasional file or two due to corruption, bit rot, etc., wouldn't be the end of the world, but I'd like to try to avoid it, or at least become promptly aware of it.
- The most frequent usage will be Time Machine backups from two Macbooks and a Mac Pro (approx. hourly during the daytime), followed by watching video on a networked HTPC.
- It will usually be accessed by one, but maximum two simultaneous users on a LAN.
- Internet access is a plus, but not required.
- Downtime isn't a big concern (even a couple days is fine).
- I'm hoping to keep power usage very low. The system will be idle much/most of the time, so I'm expecting to rely on Intel or AMD's latest power-saving technologies (e.g., Intel SpeedStep or AMD's EPU) for this.

My plan is to build a ZFS/RAID-Z6 based FreeNAS box, but I'm open to any other suggestions or ideas. I'm on the fence about whether to give special treatment to the ~2TB of more crucial files - i.e., create a separate RAID array for those or the like. I'd appreciate any opinions there.

Considering all of that, I've done a couple weekends'-worth of research and have come up with a lot of questions based on a FreeNAS build:

- Do I need to worry about support for my 3TB Hitachi drives? If I understand correctly, modern 64-bit OSs should support them out of the box as storage drives, and the compatibility issues mainly arise when they're used as boot drives; is that correct? The HBA that I've chosen (HighPoint Rocket 620) was actually bundled with the earliest 3TB drives on the market, so it should have solid support, I think.
- A commenter on another thread suggested that the 512KB sector size of the Hitachi 3TB drives could cause performance issues. Any idea what he is referring to there?
- Is ECC memory advisable (i.e., worth the $50-100 total premium)? I've heard mixed opinions, but the more legitimate sources seem to say it is. I assume registered/buffered is not that important?
- Would I be better off with an Intel-based server board with ECC memory (like http://www.newegg.com/Product/Product.aspx?Item=N82E16813121526), or is an Asus AMD desktop motherboard sufficient? (The latter supports ECC memory.) The main advantage to the Intel that I can see is onboard dual LAN ports, but are there others I'm not aware of? I like the Asus route because they support nice consumer-grade features like dynamic power efficiency, onboard video (for cheap/easy initial setting up and BIOS tweaking), USB 3.0, etc. However, will I be able to use all these features running FreeNAS (primarily the EPU power-saving feature)?
- Is AMD a first-class citizen in the FreeBSD/FreeNAS world, or is Intel a safer bet in terms of stability and driver availability?
- What kind of CPU do I need to ensure that it won't be a performance bottleneck? (Please see my build list below for my best guess.)
- Is 8GB RAM sufficient? Does the need for RAM capacity increase with more storage space?
- Is the commonly-used Realtek 8111E LAN Chipset enough to see the ~100MB/s ideal local network speed, or will I need a good Intel NIC? Are there any other typical bottlenecks to worry about?
- Is Cat6 cable length much of an issue in terms of performance degradation? My longest stretch will be about 30-40 feet.
- I found a write-up (http://lime-technology.com/home/87-for-system-builders) that highly recommends a single-rail PSU, but that wasn't until I'd already bought a dual-rail PSU (Antec EarthWatts EA-380D). Is this likely to be a problem?
- Is there anything else I haven't considered, or may be missing?

Based on my research, current level of understanding, and inclinations, I've put together this build list:

ALREADY OWN:

Antec Three Hundred Illusion Black Steel ATX Mid Tower Computer Case
http://www.newegg.com/Product/Product.aspx?Item=N82E16811129066

Antec EarthWatts Green EA-380D Green 380W Continuous power ATX12V v2.3 / EPS12V 80 PLUS BRONZE Certified Active PFC Power Supply
http://www.newegg.com/Product/Product.aspx?Item=N82E16817371033

(8x) HITACHI Deskstar 0S03230 3TB 5400 RPM 32MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145493

READY TO BUY:

ASUS M4A89GTD PRO/USB3 AM3 AMD 890GX HDMI SATA 6Gb/s USB 3.0 ATX AMD Motherboard
http://www.newegg.com/Product/Product.aspx?Item=N82E16813131631

AMD Athlon II X2 250 Regor 3.0GHz 2 x 1MB L2 Cache Socket AM3 65W Dual-Core Desktop Processor ADX250OCGMBOX
http://www.newegg.com/Product/Product.aspx?Item=N82E16819103903

Kingston 8GB (2 x 4GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) Server Memory Model KVR1333D3E9SK2/8G
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262

HighPoint Rocket 620-OEM PCI-Express 2.0 x1 Low Profile SATA III (6.0Gb/s) Controller Card - OEM
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115099

(2x) HITACHI Deskstar 0S03230 3TB 5400 RPM 32MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive
SATA cables
PATA/CompactFlash adapter and CF card, or USB thumbdrive and internal USB adapter
fan adapter cable(s)
power adapter cable(s) for the extra SATA drives
PSU cable (Antec PSU didn't come with one)
Cat6 cables

How does that look? I'd appreciate any and all feedback - please let me know if I missed anything, or need to provide any more information. Thanks in advance!

 

[golemcito]

Hi,

How are you going to connect 10 HD? you have got 6 sata on the board and 2 on the highpoint controller. You will need 2 additional sata connections...

I have that MOBO, 8Gb RAM, a Phenom 1090T, an intel NIC and only can get 30MB/sec under cifs. I´m a bit dissapointed about the performance.

 

[Mr. Bungle]

Thanks, golemcito - sorry, I should have specified (2x) on the HighPoint Rocket 620. I've mainly been looking at motherboards with 6x onboard SATA ports (the majority these days) and 2 or more PCI-E 2.0 slots so that I can run a pair of those HighPoint HBAs. I've done a lot of comparing prices, and found that 2x SATA HBAs are the most economical if you have PCI-E 2.0 slots to spare.

That's unfortunate to hear that you're having slow performance with that kind of hardware - based on my research it seems like that should be well beyond capable of saturating a gigabit LAN connection (~100+ MB/s). Have you done some troubleshooting to try to isolate the bottleneck? Do your performance metrics (CPU/memory utilization, drive throughput, etc.) give you any clues? What kind of disks are you using, and how are they connected to the motherboard (i.e. onboard ports or via HBA)?

 

[golemcito]

Thanks, golemcito - sorry, I should have specified (2x) on the HighPoint Rocket 620. I've mainly been looking at motherboards with 6x onboard SATA ports (the majority these days) and 2 or more PCI-E 2.0 slots so that I can run a pair of those HighPoint HBAs. I've done a lot of comparing prices, and found that 2x SATA HBAs are the most economical if you have PCI-E 2.0 slots to spare.

That's unfortunate to hear that you're having slow performance with that kind of hardware - based on my research it seems like that should be well beyond capable of saturating a gigabit LAN connection (~100+ MB/s). Have you done some troubleshooting to try to isolate the bottleneck? Do your performance metrics (CPU/memory utilization, drive throughput, etc.) give you any clues? What kind of disks are you using, and how are they connected to the motherboard (i.e. onboard ports or via HBA)?

You can have a look at this card, you can get 8 sata ports easy.

I have tried everything: change router, change switch, change NIC (I bought the intel NIC instead the Realtek), change cifs options under FreeNAS, change NAS version.

The cpu occupation is minimal, the smdb daemon is under 20% always. Doing a dd "test" i get speeds over 100MB/sec reading and writing.
I´m really lost. Maybe FreeNas doesn´t like AMD. I´m waiting for people who are thinking in an Core I3 configuration and see what they get.

I forgot one thing... Under Win7/WHS i get 100MB/sec out of the box.