BUILD New Build Advice - vSphere Storage Server

[big_sid]

Hi,

Been lurking a while now and doing a lot of reading. I'm about ready to take the plunge and build a proper FreeNAS system, but before I do I'd like to get a sanity check off you knowledgable folk.

Use Case:

The FreeNAS system will primarily be used as an iSCSI storage server for a small vSphere 6 environment consisting of 4 servers in a home-lab environment. When I say four servers, what I actually mean is...

2 x Intel Xeon E3-2603V2, 64GB whiteboxes
1 x AMD 7870K, 32GB whitebox
1 x HP DL585 G5 16 x Opteron, 128GB server

VAAI support is essential.

Additionally, I'll probably be hosting Plex in a jail on the FreeNAS, but its usage will be light. No more than 2 simultaneous streams.

Storage Network:

I'll be using multiple 1Gbps copper connections. Each ESXi host will have 2 x 1Gbps interfaces used solely for iSCSI traffic on different VMKernel interfaces and be configured to use round-robin for load balancing along each path. Everything goes through a HP1920-48G gigabit switch. iSCSI will be on its own dedicated VLAN with 9000 MTU.

Build Spec:

I'm looking at the following kit list right now which incorporates new and existing components I'll be re-purposing from other machines...

Case - 15 HDD bay, 4U X-Case Easystore
PSU - Corsair CSM 650W
Mobo - Supermicro X10SL7-F
CPU - Intel Xeon E3-1231v3
RAM - Crucial CT102472BD160B 8GB DDR3 PC3-12800 Unbuffered ECC x 4 (32GB total)
HDDs - 6 x 3TB Western Digital Red (3 I have already, 3 I will buy)
NICs - 2 onboard (for LAN) + 2 x dual port Qlogic Netxtreme II BCM 5706 with H/W iSCSI + TOE (already have these)
SSDs - 2 x 128GB Plextor SSDs (already have these)

Based on all this I have a few questions...

1) Is the spec and chosen components OK? I think it is.
2) I'm a little concerned about the lack of extensibility of the X10SL7-F as it only has two PCI slots. If I needed to add more NICs or a HBA for example I would be S.O.U.L. Supermicro boards seem difficult to get hold of here in the UK for a reasonable price, but I do like the idea of having IPMI.
3) I don't know what to do with the SSDs. I've read lots about ZILs, SLOGs, L2ARCs and all the pros and cons and ways you can make the performance worse. I've also read that SSDs used for write logs need to ideally be hardwearing and able to survive a power loss (e.g. super capacitor variety). I don't want to fork out more money for additional SSDs, so what could I do with these two that would give me some more benefit without compromising write integrity.
4) How does VMFS work on top of iSCSI? I assume it would have to be file backed iSCSI so a VMFS volume would sit on top of ZFS right?

Thanks!

 

[Mirfster]

RAM - Crucial CT102472BD160B 8GB DDR3 PC3-12800 Unbuffered ECC x 4 (32GB total)

From what I have seen on these forums regarding ISCI, you are going to want to at least double (if not triple or quadruple) the RAM.
Then with that much RAM, you are going to need to think about L2ARC...
But I don't think your motherboard supports more than 32 GB, so you are kinda in a bad place...:(

 

[big_sid]

From what I have seen on these forums regarding ISCI, you are going to want to at least double (if not triple or quadruple) the RAM.
Then with that much RAM, you are going to need to think about L2ARC...
But I don't think your motherboard supports more than 32 GB, so you are kinda in a bad place...:(

Thanks for the reply. I haven't seen anything about needing lots of RAM with iSCSI. Can you point me in the direction of any posts about this? You're right, the board only supports 32GB max, plus my wallet won't support much more RAM to be honest. It doesn't have to be iSCSI, I could use NFS, but I'm after VAAI support and I think that is only available with iSCSI not NFS.

 

[Syris]

4) How does VMFS work on top of iSCSI? I assume it would have to be file backed iSCSI so a VMFS volume would sit on top of ZFS right?

I'll tackle this part at least. I have a home lab(2 nodes) and use iSCSI. You get your choice of file or device extents. I used file extent. When you attach it in vmware, you can vmfs format it, so yes it will be "on top of" ZFS.

 

[Mirfster]

Thanks for the reply. I haven't seen anything about needing lots of RAM with iSCSI. Can you point me in the direction of any posts about this?

Here are a couple examples:
Why iSCSI often requires more resources for the same result
Fatal Trap 12 - In use by another user
Network Services Drop

Whatever you do, don't end up here:
How To Fail ... a guide to things not-to-do.

 

[big_sid]

Here are a couple examples:
Why iSCSI often requires more resources for the same result
Fatal Trap 12 - In use by another user
Network Services Drop

Whatever you do, don't end up here:
How To Fail ... a guide to things not-to-do.

Thanks again. I read some of the comments by jgreco. I see what you mean. He does state in one thread the 32GB would be the minimum recommended RAM for iSCSI but preferably 64GB or more. As you correctly stated, my motherboard will only support 32GB.

So, I suppose my choices are to stick with iSCSI and see how it goes on 32GB of RAM or use NFS and not have VAAI support right?

On the NFS VAAI front, I found posts dating back to 2013 with people discussing if/when it would be supported. What is the situation currently with NFS and VAAI, I'm struggling to find any defnite information.

Thanks

 

[Mirfster]

What is the situation currently with NFS and VAAI, I'm struggling to find any defnite information

Not sure on that, maybe others will chime in.