Network Setup of New Server [Diagram]

[IamSpartacus]

Building out my first FreeNAS server for use as a shared datastore for a 3-node ESXi cluster. I plan to use NFS to mount the datastore even though I've read that iSCSI gives slightly better performance. I like the ease of NFS and the ability to not have to reserve a big chunk of free space like with iSCSI.

Above is a very low level representation of my network. I'm looking for some best practices / advice for how best to configure my physical connections to the FreeNAS server. I've heard mixed things about LACP as well as had mixed results in my own experience so I'm probably going to avoid using it. Without using LACP, how would I go about configuring my dual 10Gb NICs? Also, I'd like to utilize the dual 1Gb connections as well as a failsafe or what have you.

I'm very familiar with vNetworking using vmkernels/vDistributed Switches but I haven't run any bare metal setups in some time so count me as out of the loop on some of this stuff.

 

[MatthewSteinhoff]

LACP for redundant pathing works well and exactly as you would expect. LACP for extra bandwidth rarely works as well as it should because it is misunderstood. In your case, LACP is worth investigating. Unfortunately, in your environment, full redundancy is going to be difficult because your switches are unstacked and from separate vendors. You can LACP into one switch but that won't give you redundancy because, if that switch fails, it isn't going to fail over to the second switch automatically.

Best practice would be to put in a second SG350XG, stack it with the existing SG350XG then LAG/LACP between those two switches. So, FreeNAS would be plugged into Port 1, Unit 1 and Port 1, Unit 2. ESXi01 would be plugged into Port 2, Unit 1 and Port 2, Unit 2. And so on. That way, when everything is up, you've got, more or less, 20G of bandwidth between ESX and FreeNAS. When one switch fails, you've still got 10G without missing a beat.

I would pass on gig networking. It'll add complexity compared with doing 10G in a stacked environment as noted above.

We use NFS for our VMs instead of iSCSI and are happy with the performance and love flexibility.

Before I worried about network redundancy, I'd spin up a second FreeNAS server for snapshot replication.

Cheers,
Matt

 

[IamSpartacus]

LACP for redundant pathing works well and exactly as you would expect. LACP for extra bandwidth rarely works as well as it should because it is misunderstood. In your case, LACP is worth investigating. Unfortunately, in your environment, full redundancy is going to be difficult because your switches are unstacked and from separate vendors. You can LACP into one switch but that won't give you redundancy because, if that switch fails, it isn't going to fail over to the second switch automatically.

Best practice would be to put in a second SG350XG, stack it with the existing SG350XG then LAG/LACP between those two switches. So, FreeNAS would be plugged into Port 1, Unit 1 and Port 1, Unit 2. ESXi01 would be plugged into Port 2, Unit 1 and Port 2, Unit 2. And so on. That way, when everything is up, you've got, more or less, 20G of bandwidth between ESX and FreeNAS. When one switch fails, you've still got 10G without missing a beat.

I would pass on gig networking. It'll add complexity compared with doing 10G in a stacked environment as noted above.

We use NFS for our VMs instead of iSCSI and are happy with the performance and love flexibility.

Before I worried about network redundancy, I'd spin up a second FreeNAS server for snapshot replication.

Cheers,
Matt

Yea I realize I'm not going to have full redundancy with this setup no matter what even if I had another SG350XG (which I can't afford as it's another $2k). Since I only have one FreeNAS box that's another SPOF. I run Veeam backups to my bulk array which in turns mirrors to my second bulk array. On top of that I replicate those Veeam backups to an offsite NAS over a site-to-site VPN so I technically have backups on 3 boxes across 2 different geographic locations.

So since I can't stack my switches, do you see any benefit to configuring the 2 1Gb links in LACP as well in case for some reason my SG350XG is down. Obviously performance would be drastically reduced but at least I'd be up.

 

[MatthewSteinhoff]

do you see any benefit to configuring the 2 1Gb links in LACP as well in case for some reason my SG350XG is down.

You can't LACP across the Dell and the Cisco switches. Your 10G link will have a different IP address than your 1G group. Does it help you to have another link up if the IP address is different than the primary? If so, sure, go for it.

Cheers,
Matt

 

[IamSpartacus]

You can't LACP across the Dell and the Cisco switches. Your 10G link will have a different IP address than your 1G group. Does it help you to have another link up if the IP address is different than the primary? If so, sure, go for it.

Cheers,
Matt

Yes I realize I can't use all 4 connetions (2x1Gb, 2x10Gb) in the same port-channel (LACP group) for a single IP. I was thinking as you said, having a secondary IP as you aluded to.