BUILD Hardware check for big FreeNAS box, part 2

[KevinM]

I posted recently about building a couple of 36-drive SAS2 E1R36L Supermicro boxes for FreeNAS, but it does seem like LSI's SAS3 cards are supported in the current FreeNAS release. With that in mind...

• 1 Supermicro 6048-E1CR36L Chassis
• 2 Intel P3700 NVMe (SLOG) 400GB
• 2 Intel S3710 SATA (L2ARC) 400GB
  
• 2 Intel S3710 SATA (OS) 200GB
  
• 2 Chelsio T530-CR 10GB NIC
  
• 1 LSI 3008 3008 HBA AOC-S3008L-L8e
  
• 8 Supermicro 32 GB DDR4 PC4-19200 RAM MEM-DR432L-CL01-ER24
  
• 2 Intel E5-2623 4-Core S-2011-3 CPU
  
• 2 SATA Cable 70 CM CBL-0179L
  
• 1 Y split SATA power extension CBL-0082L
  
• 1 Internal Drive Tray MCP-220-84701-0N
  
• 4 SFP Copper Twinax Cable SFP-H10GB-CU3M=
  
• 36 HGST 8TB SAS 12Gb/s HUH728080AL5204 Hard Drive

Even though I won't be doing a lot of CIFS, I'm thinking the quad-core 2623 3.0 GHZ CPU would be better than having more cores at lower speed, such as the 2620, 6 cores at 2.4 GHZ.

400 GB is overkill for a SLOG device and NVMe is still pretty expensive, but I like the IOPs. I'm assuming it's still best practice to buy these in pairs.

Our 10 GB Ethernet is SFP-based, so I will be getting 2 dual-port add-in cards. Are Chelsio cards still preferred over Intel for 10 GB SFP?

I'm thinking of using a second LSI 3008 HBA for extra redundancy and theoretical performance benefits.

I will be enabling compression from the beginning.

Most important, storage design. The use case will primarily to house datastores for VMware, various kinds of backups and NFS storage for our Linux systems. I used 6 x 6-drive RAIDZ2 for the first FreeNAS boxes I built and they've been great, but I'd like to get more performance as long as I'm not risking reliability. I've been fiddling around with an online ZFS disk space calculator and it looks like a 3-way mirror will get me around 69TiB useable uncompressed, whereas 6 x 6-drive RAIDZ2 will get around 137 TiB uncompressed. I'm just not comfortable with 2-way mirrors. What do you guys think is the best way to carve up this amount of storage?

 

[KevinM]

Also we have a lot of iscsi nimble storage for our vmware environment and all the targets and initiators are on the same vlan. I've read the best practice flamewars^wdiscussions about this, and it used to be that in freenas you could not set up multiple targets on the same vlan. Am I correct in assuming this is still the case in 9.3? It's probably not possible at this point to rearrange our environment.

I've been talking to a local Supermicro reseller and I'm going to be buying soon. I'd be interested in any thoughts about the hardware, like the sizing and type of storage for slog/l2arc, 4-core vs slower 6-core cpu, etc.