New Build - Looking For Input/Advice

[hopvision]

Let me start with some background...

I have been running a Windows based file server at home (2008 R2) serving files and media for a few years without issue until last Friday evening (6/22 will forever be known as "Daddy Yelling the F-Word For 16 Hours Day" according to my wife). I have upgraded the hardware several times "around" the RAID controller (Adaptec/ICP 5165BR) and disks without error. I'd simply build the new system, install windows and pop the card in with the drives all connected as they were. Friday, I attempted to replace the motherboard and while on first boot everything appeared to have worked (both of my arrays were optimal), after two reboots, it told me that all of my drives were new, all of the drives from the two arrays were missing, and nothing I tried could resolve the problem. I couldn't even recover the array via configuring an identical array and doing a quick init because somehow the RAID controller had decided that I needed a feature key to create RAID6 arrays for no apparent reason.

I'll pause here to admit that I did not have a backup. With 22TB of storage (16.5TB after parity), I just didn't really have anywhere to back it up to. Some of the loss put my wife, a photographer, in a bad position. Some of the stuff was irreplaceable. So of I went to buy four 2-drive SATA/USB docks, a few 3TB external drives and a license for UFS Explorer RAID recovery. I'm happy to say that I was able to recover the entire first array via UFS Explorer (and a big fan blowing across my drives in the USB docks) and am working on the second as we speak. All of the personal stuff we can't replace is being backed up to the cloud now. The photos of hers that weren't already in the cloud are now.

This brings me to my big rebuild. After building some simple 1-2 drive Freenas setups for friends who wanted the XBMC + Server media setup I have, I've seen a lot that I like. I've decided that building a Freenas 8 server is the right idea for a few reasons, and really could only identify one con.

PROS:

1. Performance. The windows server was never particularly speedy, but I could serve two 1080p MKV files simultaneously while my wife accessed documents without complaint. I think that Freenas (ZFS) will be much quicker, especially with my new hardware.

2. Lack of Hardware Dependence. Knowing that I can put together another Freenas box (same version or higher) and plug my drives into whatever controller I have and move on without having to cross my fingers and pony up $600 for a spare controller is a big positive.

3. Support. These forums and other resources I've found kick ass compared to the helpless feeling I had last friday with the Adaptec controller.

4. Future Expandability. I'm excited to see some of the additional plugins and things planned for future versions. I'd like to consolidate my SABNZBD+/Sickbeard server and the file server at some point now that I have decent hardware.

CONS:

1. Lack of similar RAID recovery software for ZFS if I hit a problem. Because a lot of the storage will be filled with media that I can't reasonably back up anywhere, It's a real pain to lose it. I don't want to rip all of these discs again, that's for sure.


My primary use for this server will be streaming movies and tv shows to up to three XBMC (openelec) media centers around my house (over gigabit Ethernet) and serving RAW/JPG images to my wife's iMac and her local Lightroom library.

Here's the hardware I have (or have coming in the mail this week) and am planning to use:

AMD FX-4100 Quad Core CPU
16GB (4x4GB) DDR3-1333 Memory
GIGABYTE GA-990FXA-UD5 Motherboard (5 PCIe - x16,x16,x8,x8,x4)
Norco RPC-4220 Case
(2) IBM M1015 SATA3 Controllers flashed to the LSI "IT" Firmware (x8,x8)
(8) 750GB Seagate 7200.11 SATA2 Drives (These were my previous RAID6 Array #1)
(8) 2TB Hitachi 0F12117 SATA3 Drives (These were my previous RAID6 Array #2)
Broadcom 5709 Dual-Port PCIe NIC (most of my other equipment are already setup with link aggregation) (x4)
PCIe Video (I have a bunch of crappy old PCIe cards laying around) (x16)
Antec 750W Power Supply

I was initially planning on running Freenas off of a new USB 3.0 memory stick and USB 3.0 internal header to dual port adapter that I bought. I'm not sure now what I'll use for the OS. I'm planning on running two RAIDZ2 arrays with 8 disks each. I will likely upgrade the 8x750GB to 8x3TB sometime in the fall when I'm financially recovered from this whole debacle. I'd like to setup NFS/SMB/AFP for the Linux, Windows and Mac systems spread around the house. Only the drives, case and power supply are being reused. I already had purchased the CPU, Memory and the PCIe NIC. When I decided to get the IBM controllers, I had to exchange the 970A-G45 motherboard I was planning to use for something with more PCIe.

Does this configuration have any "gotchas"? Would I be better off to throw the 16GB memory in my main workstation and put the 8GB from there in the server? Am I wrong in any of my assumptions? Any comment, critique (backups, I know) or suggestions would be greatly appreciated.

 

[Joshua Parker Ruehlig]

I believe the RAM would better suit you in the server then your workstation, if the server runs zfs. I don't know if FreeBSD8.2 uses usb 3.0 speed, I think I recall that being added to 9.0, it will probably just detect it as usb 2.0...

I have streamed to multiple xbmc/openelec boxes before from my lower speced FreeNAS server and have never experienced buffering / stuttering. So, yeah performance should not be an issue.
If you want to get a central mysql database / thumbnails I've had that running great for a while now too.

I wouldn't necessarily agree that zfs is as expandable as other raid systems, mdadm from linux is more expandable in some ways but zfs has alot of pros (like datasets). Just remember when you've created a vdev you CANNOT increase / decrease the amount of drives in that vdev.

 

[survive]

Hi hopvision,

Take a look at this thread, post #4:

http://forums.freenas.org/showthread.php?7591-Best-Recommendations-for-FreeNAS-with-Apple-Xserve

I'm really into the Supermicro boards right now....

-Will

 

[hopvision]

After spending some time in the forums, I'm tempted to just return the GIgabyte board, Non-ECC memory and order a Supermicro board with an E3 Xeon or i3 and some ECC memory. I had no idea how cheap those boards are and how close in price the ECC memory (~$30 more) is. The $100 i3 puts me a little over budget since I was planning to reuse the AMD chip from another PC, but that $100 seems like its worth it. Back to the drawing board...

I'm not worried about expanding arrays or anything. With arrays of this size, it's an exercise in futility anyway. I usually dump the data to something fast an redundant at work, build the new system and copy it back.

My big concern is that I build a solid foundation that will last me 2-3 years performance wise and handle any upgrades I might decide to make. I was hoping to build this weekend, but any change to hardware will likely push it to next week.

I bought everything from Amazon just in case I changed my mind, so returns are easy and it'll only cost me $8-10 in return shipping to send the stuff back.

Also, there appears to be an open-box version of the X9SCM-O Supermicro board on Newegg for $80 if anyone is trying to squeeze one of them into their budget. I think that just made the decision for me. :)

 

[hopvision]

Thanks, Will. It was your posts (that one especially) that were the primary reason for my change of heart. I'm definitely going with the X9SCM-O. I'm just trying to decide whether I should spend the extra $120 and just grab an E3-1230 Xeon. If Freenas isn't going to gain a lot from the extra cores, I won't bother.

I'll post next week when I'm putting it all together. I'm excited.

 

[jgreco]

The E3-1230 Xeon happens to be what I've got on the bench on a FreeNAS box I'm messing with. It's insanely faster than a NAS would normally need. It opens the possibility of doing things like on the fly compression (ZFS supports this). I think the compelling bit here is that if you spend the extra cash, you won't need to be worrying about this for years. However, you can easily do without it.

Bench box is currently sucking in about 800 megabits of iSCSI traffic at 90% idle. I'm kind of trying to melt it. Not sure what the bottleneck is, but it doesn't appear to be CPU.

 

[survive]

Hi hopvision,

I'd be kind of torn between the gear you have now and the Supermicro. I've been running Gigabye "UD" series boards for a while in my "server" platforms, the SM\i3 combo I have is the first Intel platform I've bought in over a decade. I like the amd boards because they seem to be more "willing" to take non-video cards in their 16x slots....you do get a ton of IO on them. In fact you have pretty much what I would be running if

1) I hadn't looked at the X9SCM series.
2) they still had the "GX" chips with the on-board video.

I use my SM board witha baby i3-2100 ($90.00 at Microcenter) and I am so pleased with it....I can max it out with 2 FBSD VM's with dual vcpus if I buildworld on them both but I'm not very tempted to move to a proper Xeon.Personally I wouldn't do a E3-1230 in my filer.....to much chip simply wouldn't be used.

-Will

 

[hopvision]

The things making me want to switch to the Supermicro are:

ECC Memory
Intel NICs
Onboard Video
Quality

The things making me question the move are:

Cost
The fact that everything is already at my house
10xSATA3 Onboard

I'm so confused. ;)

 

[hopvision]

I decided to build with what I have and see what it can do. UPS says that the m1015s are waiting at my door. I got myself a SATA->CF slot adapter and a 4GB card for the OS. I also realized that I have an Intel 1000VT Quad-Port PCIe card that I can use in place of the broadcom. I doubt I'll ever need 4 ports, but I can always take it out if needed for something else and pop in a single/dual Intel card at that time. I'm still moving data around and completing recovery, so it'll be late tomorrow or Sunday until I can get the drives in and test everything, but I'm planning to build everything else tonight.

 

[paleoN]

I'm still moving data around and completing recovery, so it'll be late tomorrow or Sunday until I can get the drives in and test everything, but I'm planning to build everything else tonight.

Well the hardware arrived yesterday and I put most of the pieces together, I just need to go through the hacking the M1015 cards to be in IT mode. I'll be running benchmarks to tweak my performance over the next few days if not week.

Let us know how it turns out. I'm looking to pick up a M1015 myself sometime, so I'm particularly interested.

It's a race!

But seriously keep us posted.

 

[hopvision]

I've got my m1015's flashed to IT and the server assembled, tied down and ready. Burning 8.0.4 to CD now. I'm trying to find two identical capacity sata3 drives to test with. I'm still recovering data from the 8x2TB Hitachis...

 

[hopvision]

Two burns, one to CD and one to DVD and both get to "looking up /boot/loader... Found" with a little spinny thing and constant activity on the optical drive. I let it go for ten minutes and nothing... I need to find another sata optical drive.

 

[jgreco]

You may be able to use a regular PC or laptop to do an install to a USB flash drive. If it's being troublesome, don't forget that options exist to do things like that and then just plug the flash into your NAS.

 

[hopvision]

It was my fault. I had the SATA controller where the DVD-RW drive was attached configured as AHCI. I switched it to IDE and it worked. I got it up and running and everything seems ok so far. I was able to setup:

Single ZFS Volume using 1TB RE3 WD SATA2
3-Disk RAIDZ2 using 3x250GB SATA2
3-Disk RAIDZ2 using 3x320GB SATA2

Copying large files (1.5GB, 4.3GB, 8GB) from a windows desktop over gigabit to any of the CIFS shares, I saw 70MB-100MB/sec. It started off around 100MB/sec and slowly leveled off to about 70MB/sec. Copying large folders of photos (48,000 small files / 130GB), I was only averaging about 18MB/sec. When I get the SATA3 drives in and can do a more realistic test, I'll do so. At that point I can apply some tweaks.

I've got my 8x750GB SATA2 drives in and ready to create array #1 today (once I'm sure that the recovered files from that array have multiple copies across all of my external drives). This recovery/rebuild is long, horrible process, but I got every single file back. UFS Explorer is the best $125 I have ever spent.

 

[hopvision]

I was able to get the 8x750gb SATA2 array up. I'm seeing some SCSI Condition errors on da0:mts:0:0:0:0 which I assume is drive 0 on the first of the m1015s. I'm suspicious that there may be an issue with the SAS cable, which might explain my initial problems that caused the whole switch. I've got one coming from Amazon on Tuesday, but plan to switch with one of the others I have that aren't used currently to see if that clears it up.

Overall, I'm pretty impressed. When transferring large files over via FTP, I get near line speed. Now that the LAGG is set up, I'd like to try transfers from my Poweredge 860 with dual gigabit to see just how fast it is. COFS is ok, comparable to what I saw from the old windows box with the adaptec RAID6.

 

[jgreco]

Remember that in most link aggregation systems, the maximum possible transfer speed for a single stream is still limited to a single physical connection, since all traffic for a single stream gets routed over one connection.

 

[hopvision]

I'm going to try FTP transfers with two simultaneous connections to see what it can do. I was already able to sustain two CIFS transfers each at ~60-70MB/sec. I'm impressed. Performance so far seems to be equal or better than the Dell H800/Powervault MD1220 with SAS 6mbit drives in it.