BUILD New near-enterprise build - critiques appreciated

[tvsjr]

First time poster, but I've been lurking for a while. Hopefully I know most of the hot-button items, so I won't get flamed too bad! :D

Purpose: Providing NFS datastores to 3-4 ESXi hosts (30-50 running VMs), plus providing a larger datastore via CIFS for media storage (Plex), etc.

This will be going in the dedicated server room in my new home. Dedicated HVAC, power, the works. So, noise isn't a major consideration. I'm currently providing this with a single Synology DS1512+ 5-drive NAS, but performance is atrocious (and I'm not running all of the VMs all the time, otherwise the poor box would melt). I'm also fortunate to have access to some spare parts, so I want to leverage those where I can (don't worry, I'm not planning to use my old desktop for this!)

So, down to the nuts and bolts:

• Case : Supermicro SC814E16-R1200B (4U, 24 bay) with BPN-SAS2-846EL1 SAS2 expander backplane
  
• Motherboard : Supermicro X8DTE-F
• CPU : Undecided. See below.
  
• Memory : 32GB Crucial ECC Registered RAM (4x8GB, 8 slots free)
  
• Power Supply/UPS/PDU : Included dual/redundant 1200-watt PS, dual APC 5KVA SmartUPS, dual PDUs
  
• Volume 1 : 12x Seagate 15K 450GB SAS, configured as mirrored pairs (2.7TB usable, VMware datastore)
• Volume 2 : 6x WD Red 4TB, RAIDZ2 (16TB usable, media storage)
• Controller : Cross-flashed M1015 or 9211-8i

For the CPU, I will have to see what I can scrounge up. My preference would be one of the 40-watt Westmere-EP chips, like the L5630 (quad-core 2.13GHz, 1.35V). I would start with one, and add a second if the load demands it. I don't intend to add anything in jails, so this will be a purely FreeNAS box. While I know this box will burn some power, I don't want to simply waste power... I don't see a real need to go to a hotter 80/105/130 watt part. I suspect Samba, being single-threaded, will be the biggest issue here.

Memory: I'll have ~18.7TB usable, so this exceeds the 1GB/TB rule. The board requires registered memory to get beyond 48GB - allow for future expansion.

Network: I have a physically isolated storage network. So, the 2 on-board ports bonded via LACP to the "internal" network... and an extra Intel dual-port server NIC bonded via LACP to the storage net.

Boot drives: I will probably snag 2 cheap, small SSDs (Intel) and mirror them for the boot drives. Or would two USB drives be sufficient? The MB has two USB connectors onboard.

ZIL/L2ARC: I'm struggling here. For heavy NFS loads, a ZIL seems to be recommended. Add 2 SSDs for the ZIL? Recommendations on size and model (I always stick with Intel SSDs)? As for the L2ARC, should I just pile on more RAM instead?

Backup: The Synology box will get repopulated with larger drives and used as a backup target. Ultimately, it will go off-site to a location connected via an AirFiber wireless shot.

I hope I've covered all the salient points. Any critiques or recommendations would be appreciated.

 

[tvsjr]

Anyone? Anyone? Bueller? :)

 

[Darren Myers]

see my sig, almost exact same stuff, same chassis, same mobo. Do yourself a favor, get triple channel ram, the 1366 socket xeons are triple channel, you can get 48Gb for cheap , I would indeed get 2 40-80Gb SSD's , i like intel. Someone else can weigh in on the ZIL/L2ARC...but you have far too little RAM, so i'd say no.

EDIT: i didnt see your thread before, otherwise i would of replied sooner. Started new job = no free time :P

 

[tvsjr]

see my sig, almost exact same stuff, same chassis, same mobo. Do yourself a favor, get triple channel ram, the 1366 socket xeons are triple channel, you can get 48Gb for cheap , I would indeed get 2 40-80Gb SSD's , i like intel. Someone else can weigh in on the ZIL/L2ARC...but you have far too little RAM, so i'd say no.

Thanks for the reply - I actually have been through quite a few of your posts, because I found that chassis in your sig. Good advice on the triple channel... can't decide if I should go 6x8 or 3x16... have to run 16GB sticks to max out at 192... decisions, decisions!

Do I have it right from my reading that, basically, one shouldn't spend significant time jacking with ZIL/L2ARC until memory is maxed out?

I always stick with Intels for SSDs... only place I've strayed is Samsung, and that's because it came in a laptop pre-installed. Should the boot drives have any criteria beyond cheap? I know there are certain models that are faster and have higher maximum write limits and are thus recommended for a flash storage pool... but it doesn't seem like that will matter much here.

EDIT: i didnt see your thread before, otherwise i would of replied sooner. Started new job = no free time :p

Aren't you supposed to read the forums and post while you're arsing off at work? :D

Thanks for the response!

 

[cyberjock]

I'm still catching up on forum posts.. but as a 1366 socket owner for my test platform, I'd go 3x16. :)

 

[cyberjock]

And to be blunt.. you might want to go with 6x16 (or more) if you want to run 40-50 VMs.. that's a massive workload.

 

[tvsjr]

Most of them aren't doing thaaaat much - and some are HA pairs in active/standby, so they aren't doing anything besides sitting there until a failover event happens. The most activity will probably be the Zimbra VM and the Percona cluster (3 VMs). But, good point... RAM's cheap.

Keep in mind there are 6 bays free in the chassis... that leaves an opportunity for a 6-drive (3 mirrored vdevs) SSD pool... move the Zimbra datastores, Percona DBs, etc. there and leave the operating system/apps stuff on the spinning rust :D

Since ESXi will force sync writes, I think I will go ahead and throw an SLOG at this. A pair of 40GB drives should be more than sufficient - I want to plan for 10Gbps just in case. Just need to find an Intel drive with super performance and battery backup... that I can afford.

 

[cyberjock]

You can also find 32GB DIMMs for a good price on ebay on occasion. I know someone that dropped 256GB of RAM in a system for $2000.

Why not have each HA pair on a different storage server, diferent esxi host, and on a different network switch so you have true reliability? :P

 

[Robert Trevellyan]

Do I have it right from my reading that, basically, one shouldn't spend significant time jacking with ZIL/L2ARC until memory is maxed out?

That's pretty much how I understand it for L2ARC.

Criteria for a dedicated ZIL device are different.

 

[tvsjr]

Supermicro only rates that board to 192GB... I don't want to do anything that's remotely unstable, and 32GB sticks get rather spendy. My wife is tolerant of my geeky habits (between IT stuff, cybersecurity stuff, and my fire department, she's beyond patient) but I don't want to push my luck too far!

I may ultimately split it up into two storage servers. I figure this is enough for my very first build... get my feet wet with this, then go on from there.