BUILD First time home build

[pdanders]

Hi all, first time poster.

I recently installed a FreeNAS system at work ($20K monster supermicro based system) and I was really impressed by it. I decided to see if I could build something nice for home use using about 10% of that budget

Here's my hardware list (currently on the way from Newegg):

GIGABYTE GA-F2A85X-D3H FM2 AMD A85X (Hudson D4) SATA 6Gb/s USB 3.0 HDMI ATX AMD Motherboard with UEFI BIOS - http://www.newegg.com/Product/Product.aspx?Item=N82E16813128577

AMD A8-5500 Trinity 3.2GHz (3.7GHz Turbo) Socket FM2 65W Quad-Core Desktop APU (CPU + GPU) with DirectX 11 Graphic AMD Radeon - http://www.newegg.com/Product/Product.aspx?Item=N82E16819113288

CORSAIR Vengeance 32GB (4 x 8GB) 240-Pin DDR3 SDRAM DDR3 1600 Desktop Memory Model CMZ32GX3M4X1600C10 - http://www.newegg.com/Product/Product.aspx?Item=N82E16820233232

8 x Seagate Barracuda 7200.14 ST3000DM001 3TB 7200 RPM SATA 6.0Gb/s 3.5" Internal Hard Drive -Bare Drive - http://www.newegg.com/Product/Product.aspx?Item=N82E16822148844

4 x Intel EXPI9301CTBLK 10/ 100/ 1000Mbps PCI-Express Network Adapter - http://www.newegg.com/Product/Product.aspx?Item=N82E16833106033

NZXT Source 220 CA-SO220-01 Black Steel / Aluminum-like finish ATX Mid Tower Computer Case - http://www.newegg.com/Product/Product.aspx?Item=N82E16811146083

Rosewill CAPSTONE Series CAPSTONE-650-M 650W ATX12V v2.31 & EPS12V v2.92 SLI Ready CrossFire Ready 80 PLUS GOLD Certified ... - http://www.newegg.com/Product/Product.aspx?Item=N82E16817182263

Kingston DataTraveler Micro 8GB USB 2.0 Flash Drive Model DTMCK/8GB - http://www.newegg.com/Product/Product.aspx?Item=N82E16820239113

D-Link DGS-1008G 10/100/1000Mbps Gigabit Desktop Switch - http://www.newegg.com/Product/Product.aspx?Item=N82E16833127325



My plan will be to setup 6 of the 8 disks in a RAIDZ2 for my main file server which is primarily accessed by CIFS/SMB (currently hosted on a raid 5 linux install... which will be rebuilt as a backup server). i'll also have access via sftp/ftp and possibly some other methods, but CIFS/SMB will be primary. I store videos (have a bunch of devices throughout the house that play video directly off the file store), photos and software primarily. I'm going to take the other two drives and mirror them for use as an iSCSI (file extent) datastore for my ESXi server. So I'll have 2 pools... 6 disks in a RAIDZ2 and a 2 disk mirror

I read the forums here pretty extensively prior to my purchase which is why I bought 4 Intel NICs (might be overkill, but hey; they are only 30 bucks ) and it's why I went with 32GB of RAM. My main switch is a Cisco 2970G, which I'm going to connect two of the NICs to in a LAG group for access to the main RAIDZ2 file store. The other two NICs will have jumbo frames enabled and will connect to the DGS-1008G for iSCSI (my ESXi server will also have two Intel Nics that connect to this switch, which is dedicated to iSCSI). The ESXi server will use round robin load balancing to talk to the datastore.

I'd love to hear any general comments on this build and what kind of performance I can expect.

I also have a question; one thing I haven't seen a ton of information on in the forums is compression and the tradeoffs involved. At work our FreeBSD expert told me I should always enabled compression on our systems at work (default compression) because it's basically free (performance wise). However, all the systems we deal with there are dual XEON or better server class boxes.

With the CPU I have selected should I expect any performance difference with compression on and off? (I do realize that might be a question that's hard to answer and I do intend to do some testing). (and I do realize the it's probably silly to enabled compression where I'm storing all my mkvs/jpgs)

Also, does anyone have experience running ESXi against a similar configuration over iSCSI? If so, what kind of performance do you get?

 

[cyberjock]

Enabling compression increases CPU loading. Depending on the type of compression used and the types of files you are saving, it could be a big boon or a big bust.

Every machine I've used it on was a complete bust. It was a major performance killer. If you really want to know if you should use it or not the best advice I can give is to try each way and destroy the zpool in between running dd bench tests and file shares. My general recommendation is to not use it because I've seen the CPU loading was just too much for even modestly designed systems. Also considering how cheap hard drive space is these days its probably not going to make enough of a difference to make your file server have enough free space to put off upgrading later on anyway.

 

[KMR]

With this sort of budget why not run a Supermicro 1155 board and ECC RAM?

 

[pdanders]

I thought about going Xeon and ECC.. but the SAN was only part of my purchase... also building a Vishera based ESXi server with 32GB of RAM that's going to talk to this thing, which ate up the rest of my budget.

Right now I'm running with an ubuntu server with software raid-5 and no ECC (and have been for several years) with only USB drives for backup (which is very intermittent)... I figure this will be a major upgrade simply due to me moving from RAID-5 to RAIDZ2 and being able to do regular automatic backups to my old system

 

[cyberjock]

If you are running Ubuntu and are familiar with Linux you could do ZFS on Linux. I installed it today. So far it seems to be okay.

 

[pdanders]

Bummer 2 out of 8 drives were DOA... at least I can start with the 6 drive RAID-Z... have to wait for the newegg replacement to do the iSCSI... everything worked perfectly other than the dead drives however

 

[grunnsat]

Bummer 2 out of 8 drives were DOA...

Yes, Seagate may have some quality problems. I populated my Readynas with four 1.5TB drives at he time. After some time I had to replace two of hem under warranty. A few years later the third drive died, and since there are no more 1.5B drives available I decided to buy four new 2TB drives. I replaced the defective 1.5TB drive by a new 2TB one. Within a few days the new drive was dead. I repeated replacing the drive until all four new drives were dead. They were replaced by four new drives with the same commercial type number, but with a different part number and firmware. Those drives kept running until after a couple of months one drive started to fail. I have replaced it under warranty. So of the 15 Seagate drives I used, 8 died prematurely.......

 

[cyberjock]

Yeah, I'm seeing the same thing with Seagate. I swore by Seagates starting around 2002 or so. They've been rock solid, but a firmware issue on their end that prevented me from using 16 drives in a RAID turned me off. I still recommended them for desktop boot drives until about a week ago. A friend bought a brand new 1TB Seagate last October. He's on his 4th RMA. I built his system so I can vouch that its not overheating or being mistreated at all. I'm so shocked by this and I can't explain it. If someone else had told me they've had to do 4 RMAs on less than 6 months I'd have dismissed them because I'd have assumed they were doing something wrong. But it is what it is. I'm very sad for Seagate. They used to be the top dog for many years. Now they don't seem to be worth anything.

 

[jgreco]

Every machine I've used it on was a complete bust. It was a major performance killer. If you really want to know if you should use it or not the best advice I can give is to try each way and destroy the zpool in between running dd bench tests and file shares.

w.r.t. "destroy the zpool" ... uh, why?

ZFS compression can be set on a given filesystem. It is applied to files as they're written. You can have a filesystem without it, turn it on, write a file, and then turn it off. Nothing else will be modified, any old files are not compressed.

Decompression is always enabled, so that any files written with compression are transparently available.

# zfs set compression=off pool
# dd if=/dev/urandom of=random bs=65536 count=8192
8192+0 records in
8192+0 records out
536870912 bytes transferred in 7.529419 secs (71303098 bytes/sec)
# dd if=/dev/zero of=zero bs=65536 count=8192
8192+0 records in
8192+0 records out
536870912 bytes transferred in 4.190557 secs (128114452 bytes/sec) ...[SUP]wow /dev/random is slow[/SUP]
# ls -ls random zero
524532 -rw-r--r-- 1 root wheel 536870912 Apr 11 09:39 random
524160 -rw-r--r-- 1 root wheel 536870912 Apr 11 09:39 zero
# zfs set compression=gzip-9 pool
# dd if=/dev/urandom of=random bs=65536 count=8192
8192+0 records in
8192+0 records out
536870912 bytes transferred in 20.065154 secs (26756382 bytes/sec) ... [SUP]compressing uncompressible, 1/3 speed[/SUP]
# dd if=/dev/zero of=zero bs=65536 count=8192
8192+0 records in
8192+0 records out
536870912 bytes transferred in 0.844727 secs (635555495 bytes/sec) ... [SUP]but holy mackerel, look at that[/SUP]
# ls -ls random zero
524532 -rw-r--r-- 1 root wheel 536870912 Apr 11 09:40 random
1 -rw-r--r-- 1 root wheel 536870912 Apr 11 09:40 zero
# zfs set compression=off pool

The main bad thing here is that the file "zero" remains compressed, and as such, will eat a little extra CPU for decompression.

 

[cyberjock]

Correct. I said that only for simplicity because I didn't feel like writing a book about the when/ifs. I didn't want the user to copy 1TB of data compressed, 1TB of data uncompressed and then decide "I'll keep it uncompressed" and then wonder why 1/2 his files are slow.

Call me lazy.. its true!

 

[jgreco]

Ok, lazy.

 

[pdanders]

Doing some initial FTP tests... getting around 100 MB/s on uploads, but only half that on downloads.

SCP is way slower on uploads (like 20-30 MB/s) but downloads are about the same. I am using LACP right now so I may disable that and remove some complexity.

Definitely happy with the upload speeds, not so much with the download speeds.

 

[pdanders]

Funny.... I turned on CIFS and the situation is reversed there. Downloads are super fast (100 MB/s) but loads are slower. Must be some protocol tuning that needs to be done on the FreeNAS side.

Doing testing from a Macbook pro retina with an SSD over a gigE link

 

[cyberjock]

The general rule for LACP is its worthless for home use. LACP works great if you want redundnant links for failed network cards, cables, etc. and large number of workstations/users. For situations where you have just a few workstations and users, its an added complexity that isn't guarateed to work well except on certain expensive commercial grade hardware.

I have 4 NIC ports on my server, each with an different IP address and each mapped to a different machine to prevent bottlenecks.

 

[pdanders]

My core switch is a Cisco 2970G, so I'm fairly confident I can do LACP no problem. You are correct that it's probably not needed... but I'm 99% sure it's working fine.

ZFS really does work the CPU... I'm starting to wonder if I should have gone with a faster APU

 

[cyberjock]

You do realize that LACP doesn't mean you'll get 2Gb/sec, right? That was the whole point of my long winded expanation above.

 

[pdanders]

Yeah I know exactly how LACP works.

I often have 2-3 streams going from different sources at once and do large file transfers... so LACP will help me some (in fact I was just doing 3 large simultaneous transfers from different VMs and I was getting way over 100 MBs...maybe 120-130 )

 

[pdanders]

Ok, score one for autotuning. Ticked the autotune box and rebooted and CIFS write speed has more than doubled.... was only getting like 35 MB/s before, now getting 75 MB/s (CIFS reads max out at 100+ MB/s)

 

[cyberjock]

Yeah I know exactly how LACP works.

I often have 2-3 streams going from different sources at once and do large file transfers... so LACP will help me some (in fact I was just doing 3 large simultaneous transfers from different VMs and I was getting way over 100 MBs...maybe 120-130 )

I get those speeds from any one of my workstations to my server.. and I can do 2 at over 110MB/sec simultaneously. But 3 is where I start to bottleneck elsewhere. :)