BUILD 216 TiB Server

[Ericbo]

This will be my second FreeNAS build, which is a great deal larger than my first build. The goal of this build is to start off with ~20TB of storage and slowly expand to ~100TB of storage (over the period of 5 years). Transfer speeds is not a major concern, as long as the Ethernet ports on the motherboard are fully saturated (2x1Gb). Storage space and reliability is the major concern I am trying to address. Lets get started!

Hardware
128 GB RAM 8x(MEM-DR416L-SL02-ER21)
Motherboard Supermicro X10SRL-F
CPU E5-1620 v3
Case + PSU SC847E16-R1K28LPB
RAID Card 2x(LSI SAS9211-8i)
HDD 8x(Seagate NAS 6TB)
UPS APC BR1500G
Boot Media Not yet decided.

At first only 8/36 drive bays will be occupied. I plan on setting up a RAID 10 array, mostly because expanding it would require a smaller payout than using RAID-Z2. I'm hoping to have two SSD's in raid 1 host the FreeNAS OS, however I am unsure if using consumer grade drives would be a good idea (EVO 850). I might end up swapping the RAM sticks from 16GB to 32GB in case more ram is needed to correctly run ZFS with the amount of data I will be storing in the future.

Please let me know what you think of this build and what your suggestions are.

 

[danb35]

There's no need for the second HBA; one will work both backplanes in that chassis. I'm not sure how the prices work, but if 32 GB sticks are comparable in price, I'd use them. A single consumer-grade SSD will be just fine for a boot device--as with any boot device, keep backups of your config file, and if the device fails, install to a new one and upload the saved config. A second SSD would give redundancy, but is probably unnecessary.

For pool configuration, it's really going to depend on how you're planning to buy your disks, both now and in the future. If you're going to buy them in groups of 6 or 8, put them in RAIDZ2--you'll have better redundancy than using striped mirrors, and better capacity available as well.

Note that the Seagate NAS disks run at 7200 RPM, and therefore run at higher temperatures than 5400 RPM disks--mine run about 5 deg C warmer.

 

[Stux]

Note that the Seagate NAS disks run at 7200 RPM

Don't think that's true.

 

[danb35]

Don't think that's true.

Well, it's true of my Seagate NAS 6 TB disks, at least, and they do run about 5 deg C warmer than the other 6 TB disks above, below, and between them. Maybe there are different models of Seagate NAS disks of which this is not true.

 

[nojohnny101]

I have some seagate NAS 3TB and I believe the spec sheet said 5900.

 

[Stux]

It's confusing, but seagtes product sheet for the latest range does mention "low rpm spindle speed" as a feature.

http://www.seagate.com/files/www-content/product-content/nas-fam/nas-hdd/en-us/docs/100724684h.pdf

Also they have a separate range "Enterprise NAS HD" and that lists 7200 rpm as a feature.

 

[danb35]

Code:

[root@freenas2] ~# smartctl -a /dev/da13
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:	 ST6000VN0021-1ZA17Z
Serial Number:	S4D0MJ2B
LU WWN Device Id: 5 000c50 08cd437e4
Firmware Version: SC61
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Wed Nov 16 06:34:02 2016 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

 

[Ericbo]

Thanks for the feedback!

I'm not sure how the prices work, but if 32 GB sticks are comparable in price, I'd use them.

I have opted to go with 4 sticks of 32GB RAM, it actually ended up being a few dollars cheaper.

If you're going to buy them in groups of 6 or 8, put them in RAIDZ2--you'll have better redundancy than using striped mirrors, and better capacity available as well.

I am still undecided if I will take this approach or end up going with RAID 10. It will take a month for the case and the RAM to arrive, so I got plenty of time to figure that out :)

Note that the Seagate NAS disks run at 7200 RPM, and therefore run at higher temperatures than 5400 RPM disks--mine run about 5 deg C warmer.

The increase heat of the hard drives should not be an issue, but something I will definitely keep any eye on.

I will update my post shortly once I purchase everything.

 

[nojohnny101]

Keep in mind one of the reasons people recommend NAS drives is because they spin slower which means less heat and less energy (which matters when keeping a server on 24/7).

For majority of setups, you will not notice a performance difference between 7,200RPM and 5,900/5,400RPM. So why pay for more electricity and possibly have to worry more about cooling?

 

[Ericbo]

Keep in mind one of the reasons people recommend NAS drives is because they spin slower which means less heat and less energy (which matters when keeping a server on 24/7).

For majority of setups, you will not notice a performance difference between 7,200RPM and 5,900/5,400RPM. So why pay for more electricity and possibly have to worry more about cooling?

Good point! Did some quick research and it would seem as though going with something like WD Red drives might be a better option. The Seagate drives seems to use over twice the power at idle speeds. Plus I don't see the extra RPM speed making a difference, especially once my RAID array gets bigger.