BUILD Hardware list and questions.

[millerdc]

I'm looking to spend about $10K on a freenas system to test as NFS storage to a small CentOS 6 based HPC cluster.

Here are some details about the cluster.

This cluster only has 10 nodes and each node will have a dedicated 1Gbit connection for NFS traffic. Each node is connected to a Cisco Nexus2k switch which is connected to our Cisco Nexus 5k router. The Freenas box will be connected directly to the router via 10Gbit SFP+.

The cluster is mainly running old inefficient codes that write out lots of small ascii text files. Not a lot of bandwidth needed but lots of I/O.

I'm looking to use freenas for the users home directories(10GB user quota), and another NFS mount for compute runs and results.

To get the best I/O out of the drives I will go with mirror vdevs in the pool with each vdev consisting of three drive mirror. I'm paranoid and don't like the idea of losing my whole pool because I lost the second disk in the same mirror while the new disk is rebuilding.

One of my questions is related to l2arc and ram usage. For example if I had a 100GB l2arc on SSD how much RAM will that consume if it was filled with data? The system I plan on building has 128GB of RAM. If 100GB of l2arc only consumes a couple GB of RAM then that is not so bad. But if it will use up half the ram then I would rather just forgo the l2arc.

My second question is about the hardware I have been trying to pindown for this NFS server. I'm still uncertain what I should get for the ZIL. I was just going to throw in two satadoms for the ZIL and perhaps two more for a mirrored boot instead of two USB flash drives. Anyway, here is my current hardware list.

SuperChassis 836BE1C-R1K03B
http://www.wiredzone.com/supermicro...ssis-3u-rackmount-cse-836be1c-r1k03b-10024217
$ 1,200

SUPERMICRO MBD-X10SRL-F Server Motherboard LGA 2011 R3
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182927
$270

Intel Xeon E5-1620 v3 Haswell-EP 3.5GHz 4 x 256KB L2 Cache 10MB L3 Cache LGA 2011-3 140W BX80644E51620V3 Server Processor
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117512
$310

Supermicro Add-on Card AOC-S2308L-L8E - storage controller - SATA 6Gb/s / S
https://www.cdw.com/shop/products/S...-storage-controller-SATA-6Gb-s-S/3198646.aspx
$185

Crucial CT16G4RFD4213 16GB 288-Pin DDR4 2133 Server Memory
http://www.newegg.com/Product/Product.aspx?Item=N82E16820148855
$185 x 8 = $1480

WD Red WD60EFRX 6TB IntelliPower 64MB Cache SATA 6.0Gb/s 3.5" NAS Hard Drive Bare Drive
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236737
$260 = 16 = $4160

Intel E10G42BTDA Server Adapter X520-DA2 10Gbps PCI Express 2.0 x8 2 x SFP+
http://www.newegg.com/Product/Product.aspx?Item=N82E16833106044
$436

 

[marbus90]

You can just do 2way mirrors plus having a spare in the chassis. Also go for 7200rpm drives like HGST NAS or Toshiba instead. Since it's NFS, you'll also likely profit from a SLOG device. A P3700 400GB is capable of achieving 250MB/s sync write performance, the Intel 750 400GB (which is the consumer model, seems that it only has lower PBW rating and warranty) should be nearby. For ultimate high-end grab a ZeusRAM or two. SLOG SSDs need the power capacitors, otherwise data is lost in case of power failure. I'd second the motion for mirrored 64GB SATADOMs as bootdevices, especially since it's planned to enable moving the .system dataset to the bootpool, keeping load off the array.

SAS 6Gbps HBAs with SAS 12Gbps backplanes doesn't mix well, also 12Gbps HBAs aren't fully there yet - firmware/driver v8 for 3008/3108 will roll out shortly while SAS 6Gbps started being stable at v11.

There are Samsung 32GB LRDIMMs around, another user posted 4x32GB RAM for $1560 just today at newegg. Gives you some more expansion room.

 

[millerdc]

Ok, after doing some more research I have refined my build to the following.

I swapped out the WD red's for the following.

HGST Deskstar NAS H3IKNAS600012872SN (0S03839) 6TB 7200 RPM Up To 128MB Cache SATA 6.0Gb/s 3.5" High-Performance Hard Drive for Desktop NAS Systems Retail Kit
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145973
$290 x 16 = $4,640

I swapped out the chassis for the following that has a SAS/SATA 6Gbps backplane.

SuperChassis 836BE16-R1K28B
http://www.wiredzone.com/supermicro...ssis-3u-rackmount-cse-836be16-r1k28b-10022136
$1,200

Two SATA DOM's for mirrored OS.

SUPERMICRO SSD-DM016-PHI SATA DOM (SuperDOM) Solutions
http://www.newegg.com/Product/Product.aspx?Item=N82E16816139089
$50 x 2 = $100

I added two Intel data center SSD's with supercaps to be used for the SLOG.

Intel DC S3500 Series SSDSC2BB300G401 2.5" 300GB SATA III MLC Internal Solid State Drive (SSD)
http://www.newegg.com/Product/Product.aspx?Item=N82E16820167165
$280 x 2 = $560

 

[DataKeeper]

I'd go with the 32gb or 64gb DOM over the 16gb. There is a significant read/write increase (see bottom of page) over the 16gb module and the extra space has its advantages.

 

[marbus90]

The write performance on the S3500 is abysmal. There's a reason I listed the S3710 or P3700 or 750. No rating from benches or manufacturer translates into sync write performance. Also I noted the 64GB SATADOMs, not 16 or 32GB. Get 64, much better.