BUILD Hardware selection for cold storage servers?

[ilmarmors]

Hello,

I'm looking for a most efficient way how to spend 10000 EUR (including VAT) for 2 cold storage servers. All the components must be purchased in EU. Max budged limit is cut in stone. I want to use 2 identical or almost identical servers. I imagine, there is possibility to play with memory and disks - slightly different configs for primary and secondary server.

Purpose:

• Storing 2nd and 3rd copy of digitized objects master files - images, audio, video. Those files are not accessible to end users.
  
• Files are copied to primary storage server over 1Gbps Internet connection from few places.
• Files from primary storage server are copied to secondary storage server automatically.

Requirements:

• Bit rot safe storage. That's why I'm looking at ZFS.
• Should be able to saturate 1Gbps link.
• Mostly sequential workload. Files once written will stay there forever.
• Must handle at least 10M files in total.
  
• Rack-mountable case. The smaller the better. 1U ideal. Definitely no more than 2U.
• Max 10000 EUR (with VAT) for 2 running servers
• No video trans-coding, no jails or other CPU intensive tasks on servers
• Compression of files are on. Image files are stored mainly as loss-less uncompressed TIFFs, that can be compressed.

I looked around and currently most reasonable hardware selection for specific case I described above in my opinion could be:

SuperMicro SuperStorage Server 5018A-AR12L
http://www.supermicro.com/products/system/1U/5018/SSG-5018A-AR12L.cfm
~1500-1600 EUR

• Intel Atom processor C2750
• 12x 3.5" Hot-swap HDD bays
• Up to 64GB Unbuffered ECC DDR3 1600MHz in 4x DIMM sockets
• Dual GbE LAN ports
• Integrated IPMI 2.0 and KVM with Dedicated LAN
• 400W High-efficiency redundant short depth Power Supplies
• Proprietary form factor motherboard form factor - http://www.supermicro.com/products/motherboard/ATOM/X10/A1SA7-2750F.cfm
• LSI 2116 SW controller for 16x SATA3 (6Gbps) ports; SATA3 or SAS2 with IT mode

There saw also option to use ASRock 1U12LW-C2750
http://www.asrockrack.com/general/productdetail.asp?Model=1U12LW-C2750
~1150 EUR in DE, but I won't be able to buy that cheap in my country, so it would be just ~200 EUR savings, but there are many cons:

• Simple non hot-swappable 250W PSU
• All 12 SATA ports on motherboard are used up, so FreeNAS will need to be run from USB sticks
• Using Marvell ports makes be nervous, if HBA card is added, then savings are even less, if at all.

Memory: 32GB ECC (4x8GB)
~380 EUR
Unfortunatelly tested memory list for Supermicro motherboard is small and 16GB module on that list (or in general?) cost arm and leg > 400 EUR for one module.

HDD options:
12 x WD Red WD40EFRX ~2200 EUR
12 x H3IKNAS40003272SE (7200rpm) ~2200-2300 EUR
12 x WD Red WD50EFRX ~2900 EUR (dangerously close to limit)
12 x HGST IDK Deskstar NAS H3IKNAS500012872SE (probably over the limit)
option for secondary server only:
12 x Seagate Archive 8TB (will require to use 4TB in primary server for sure because of total cost limit) (probably will be available by time servers will actually be bought) (can be interesting option from future perspective - if there are second primary server, then this secondary server can be secondary server for 2 primary servers, which have 2 time smaller disks each)

SATA DOM: Adata SATA III DOM Memory Card 16GB, MLC, 0 to 70C, 7pin, Horizontal (I can get hands on such model, not big selection here for me). If second SATA DOM for OS mirror, then it will need to be sticked into LSI ports + power. SuperMicro has only 1 SATA port with power for SATA DOM on motherboard.

I'm thinking about doing copying from primary to secondary via rsync. And taking snapshots from time to time on secondary too. In theory that should exclude possibility that secondary server ZFS pool can be broken if replication is done by zfs send and ZFS goes nuts on primary server, right?

There is question about most appropriate pool config. Options are:

• one vdev with 12 disk RAID-Z3
• one vdev with 11 disk RAID-Z3 (although "Since FreeNAS utilizes compression by default (and there are 0 cases where it makes sense to change the default!), any attempts to optimize ZFS with the vdev configuration are foiled by the compressor.")
• one vdev with 12 disk RAID-Z2 (feels too risky, although there should be backup copy on other server)
  
• two vdevs with 6 disk RAID-Z2

Any suggestions and comments welcome.

Thanks,
Ilmārs

 

[marbus90]

The Avoton requires unbuffered 16GB DIMMs, these are always more expensive. According to Josh Paetzel you don't need to get crazy on RAM anyway.

Note that the Supermicro powered DOM port only powers them with these specific DOMs: http://www.supermicro.nl/products/nfo/SATADOM.cfm

The ASRock C2750D4I board (which is used in that chassis) is considered as a 6port board in the community since the Marvells aren't always working well. The Supermicro solutions seems far superior. You can grab the Supermicro barebone for ~1450EUR including 19% VAT (it still varies within the EU ;)) right here: https://www.sona.de/.1173794615-Supermicro-Barebone-SuperServer-SSG-5018A-AR12L
However, these 12-in-1U solutions are extremely long with 813mm. It might not fit the usual 1070mm deep racks but requires 1200mm deep ones.

Re vdev config: if you really want your data to be safe, I'd use a 11disk z3. Either only buy 11 disks each or drop in the 12th disk as warm spare to replace the defective drive from the office instead of having to step in the server room.

 

[MoB]

I am also in favor of the 11 Raid Z3 option + spare. One V-DEV is suitable for your way of accessing the servers from few places.

 

[ilmarmors]

I somehow missed that - but SuperMicro SuperStorage Server 5018A-AR12L comes fully pre-built. They came pre-built with all the disks - one with 12 x 4TB Constellation ES.3 ST4000NM0033, second with 12 x 5TB Enterprise Capacity ST5000NM0024 disks. So no WD REDs, but enterprise disks - with higher MTBF, better anti-vibration measures etc. Does that change anyhow 11 Raid Z3 + spare suggestion?