I'm looking to spend about $10K on a freenas system to test as NFS storage to a small CentOS 6 based HPC cluster.
Here are some details about the cluster.
This cluster only has 10 nodes and each node will have a dedicated 1Gbit connection for NFS traffic. Each node is connected to a Cisco Nexus2k switch which is connected to our Cisco Nexus 5k router. The Freenas box will be connected directly to the router via 10Gbit SFP+.
The cluster is mainly running old inefficient codes that write out lots of small ascii text files. Not a lot of bandwidth needed but lots of I/O.
I'm looking to use freenas for the users home directories(10GB user quota), and another NFS mount for compute runs and results.
To get the best I/O out of the drives I will go with mirror vdevs in the pool with each vdev consisting of three drive mirror. I'm paranoid and don't like the idea of losing my whole pool because I lost the second disk in the same mirror while the new disk is rebuilding.
One of my questions is related to l2arc and ram usage. For example if I had a 100GB l2arc on SSD how much RAM will that consume if it was filled with data? The system I plan on building has 128GB of RAM. If 100GB of l2arc only consumes a couple GB of RAM then that is not so bad. But if it will use up half the ram then I would rather just forgo the l2arc.
My second question is about the hardware I have been trying to pindown for this NFS server. I'm still uncertain what I should get for the ZIL. I was just going to throw in two satadoms for the ZIL and perhaps two more for a mirrored boot instead of two USB flash drives. Anyway, here is my current hardware list.
SuperChassis 836BE1C-R1K03B
http://www.wiredzone.com/supermicro...ssis-3u-rackmount-cse-836be1c-r1k03b-10024217
$ 1,200
SUPERMICRO MBD-X10SRL-F Server Motherboard LGA 2011 R3
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182927
$270
Intel Xeon E5-1620 v3 Haswell-EP 3.5GHz 4 x 256KB L2 Cache 10MB L3 Cache LGA 2011-3 140W BX80644E51620V3 Server Processor
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117512
$310
Supermicro Add-on Card AOC-S2308L-L8E - storage controller - SATA 6Gb/s / S
https://www.cdw.com/shop/products/S...-storage-controller-SATA-6Gb-s-S/3198646.aspx
$185
Crucial CT16G4RFD4213 16GB 288-Pin DDR4 2133 Server Memory
http://www.newegg.com/Product/Product.aspx?Item=N82E16820148855
$185 x 8 = $1480
WD Red WD60EFRX 6TB IntelliPower 64MB Cache SATA 6.0Gb/s 3.5" NAS Hard Drive Bare Drive
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236737
$260 = 16 = $4160
Intel E10G42BTDA Server Adapter X520-DA2 10Gbps PCI Express 2.0 x8 2 x SFP+
http://www.newegg.com/Product/Product.aspx?Item=N82E16833106044
$436
Here are some details about the cluster.
This cluster only has 10 nodes and each node will have a dedicated 1Gbit connection for NFS traffic. Each node is connected to a Cisco Nexus2k switch which is connected to our Cisco Nexus 5k router. The Freenas box will be connected directly to the router via 10Gbit SFP+.
The cluster is mainly running old inefficient codes that write out lots of small ascii text files. Not a lot of bandwidth needed but lots of I/O.
I'm looking to use freenas for the users home directories(10GB user quota), and another NFS mount for compute runs and results.
To get the best I/O out of the drives I will go with mirror vdevs in the pool with each vdev consisting of three drive mirror. I'm paranoid and don't like the idea of losing my whole pool because I lost the second disk in the same mirror while the new disk is rebuilding.
One of my questions is related to l2arc and ram usage. For example if I had a 100GB l2arc on SSD how much RAM will that consume if it was filled with data? The system I plan on building has 128GB of RAM. If 100GB of l2arc only consumes a couple GB of RAM then that is not so bad. But if it will use up half the ram then I would rather just forgo the l2arc.
My second question is about the hardware I have been trying to pindown for this NFS server. I'm still uncertain what I should get for the ZIL. I was just going to throw in two satadoms for the ZIL and perhaps two more for a mirrored boot instead of two USB flash drives. Anyway, here is my current hardware list.
SuperChassis 836BE1C-R1K03B
http://www.wiredzone.com/supermicro...ssis-3u-rackmount-cse-836be1c-r1k03b-10024217
$ 1,200
SUPERMICRO MBD-X10SRL-F Server Motherboard LGA 2011 R3
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182927
$270
Intel Xeon E5-1620 v3 Haswell-EP 3.5GHz 4 x 256KB L2 Cache 10MB L3 Cache LGA 2011-3 140W BX80644E51620V3 Server Processor
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117512
$310
Supermicro Add-on Card AOC-S2308L-L8E - storage controller - SATA 6Gb/s / S
https://www.cdw.com/shop/products/S...-storage-controller-SATA-6Gb-s-S/3198646.aspx
$185
Crucial CT16G4RFD4213 16GB 288-Pin DDR4 2133 Server Memory
http://www.newegg.com/Product/Product.aspx?Item=N82E16820148855
$185 x 8 = $1480
WD Red WD60EFRX 6TB IntelliPower 64MB Cache SATA 6.0Gb/s 3.5" NAS Hard Drive Bare Drive
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236737
$260 = 16 = $4160
Intel E10G42BTDA Server Adapter X520-DA2 10Gbps PCI Express 2.0 x8 2 x SFP+
http://www.newegg.com/Product/Product.aspx?Item=N82E16833106044
$436