Question about networking with ESXi/FreeNAS

[jaccovdzaag]

Hi guys,

Yes, I'm here again, with another question. More like a 'need some help on my thoughts'.
Apart from the server, I have a Linksys WRT3200 (and for future trail a EdgeRouter-X) and 2 LGS318 switches. Everything is hooked up on it.

Now, the motherboard (X10SLM+-LN4F) has 4 LAN ports and a IPMI port. The last one is working fine.
I myself, had a thought and came up with this:

LAN1/LAN2: LACP, dedicated FreeNAS with their own virtual switch in ESXi.
LAN3: Use that for a Windows Server VM, on a virtual switch.
LAN4: Sort of a management port. 1 for the ESXi webinterface, and 1 for the FreeNAS GUI. Also it's own virtual switch.

So, create 3 virtual switches for the 3 'services', link it together and that's it.

But I'm thinking, is it possible to have two interfaces on FreeNAS, one pure for 'data' and the other for the interface? Or should it be one LACP with those two LAN ports, and that's it? I know, it's for homeuse, but it can be done, so why not. I like to learn things as well :)

 

[bigphil]

Sounds like you've got an AIO setup going? LACP isn't available for ESXi unless you use a distributed switch, and that requires having vCenter and Enterprise Plus licensing. On another note, you wouldn't need a separate vSwitch for each machine...you'd use a virtual machine port group on one switch. The ESXi management would be on a vmkernel port on the same vSwitch.

 

[jaccovdzaag]

Yes I do. 1 VM is FreeNAS, the other one is Windows Server. Might be coming more, but I just have to get the base stable :)

As for the switches. Okay. Kinda confusing since there is a NIC teaming option in the Web Client. But indeed, now I read the KB's again, I only see vSD.. Aaaand since I'm on a free license and not aiming to rebuild it again, that won't happen. Might be future then.

I will look into that. Yes, I've made port groups for the different VM's and put them on their own switches.

Networking.. I like it, it's interesting, but it's a pain in the arse..

 

[kdragon75]

I would stick with one vSwitch and use VLANS to segregate your "services". Each VLAN on the vSwitch can still be tied to a set NIC(s) on the port group level but you still have the ability to failover. This way if you can even do sudo load balanceing with "Route based on destination MAC". I will warn you that without multiple concurrent connections from different users (or at least MAC address) it wont do anything for you.

This will also allow you to satisfy the "required" redundancy for you management interface.

On the subject of routers, take a look at pfSense. Generally it will provide MUCH more flexibility with VLANS, routing, filtering, monitoring, and more. Also its based on FreeBSD just like FreeNAS ;)

Generally I'll do no vlan for my main LAN, 99 for management interfaces - this covers things like ESXi, FreeNAS SSH and GUI, switch SSH, etc... I have a lot more than that but it doesn't apply to your post. You might like the following:

vSwitch0 LAN 1, 2, 3, and 4
- Management portgroup (i typically use 192.168.99.0/24 with a small DHCP scope to set up new devices.)
-- ESXi
-- FreeNAS NIC1 only used for management, webGUI, SSH, etc..
- VM portgroup (this is DHCP for most but a few pesky things that don't register DNS will get static IPs.)
-- Windows server
-- Some other stuff

This way if you have many clients accessing teh FreeNAS server, ESXi will route the traffic out all 4 ports (under ideal conditions). Keep in mind the NIC on your FreeNAS VM will be a "10gb" interface and the bottle neck will be the ESXi hosts physical ports.

On the switch, set the 4 ports to trunk and if you can only allow the used VLANs.

 

[kdragon75]

Just saw you are using two switches. if there both in the same location as the ESXi host, I would connect two ports to one and two ports to the other. Don't try to use LACP between the host and switches. Let ESXi handle failover and "load balancing". You should use LACP from one switch to the other. Make sure to configure the LACP BEFORE connecting cables just in case the switches aren't using STP/RSTP ;).

 

[Stux]

I benchmarked a vSwitch at 20+gbps on an X10SDV...

 

[kdragon75]

Yeah it depends on the virtual device used, your CPU, and drivers. Generally the VMXNET3 network device will offer the best performance and FreeBSD has the driver baked in. It shows up as a 10gb full duplex link to the system.

Are you getting 20gb bidirectional or just one way? what tools did you use to test? Just curious.

Fond some good reading on VMXNET3 performance testing. Performance Evaluation of VMXNET3 Virtual Network Device

 

[Stux]

Vmxnet3

20gb each way, but 10gbit bidirectional.

Iperf2 test.

 

[kdragon75]

Vmxnet3

20gb each way, but 10gbit bidirectional.

Iperf2 test.

Interesting.