Best use for 2nd NIC

[revengineer]

Hello,
I have one more issue to resolve before replacing my Windows Server with FreeNAS, and I could use some advice. I have a Supermicro X9 motherboard with dual NICs (plus a third for IPMI). It is hard to miss the posts and warnings about connecting more than one NIC to the same network. In my initial ignorance, this is however exactly what I have done. It would be great if someone could suggest how to best make use of the second NIC using Bridge or LAGG or something else for the following use case.

The FreeNAS server will be mainly used for network storage provided by a 6x8TB RAIDZ2 disk array. I do plan on running a Windows VM to host an emby server with media utilities. I also plan on running a jail with urbackup for bare-metal restores of my household PCs. I can probably come up with a few more VMs and jails with a combined total of ~5. There is only one network.

I could imagine that running the Windows emby VM through a dedicated network port may be most beneficial. But I am not sure if this is possible and how to best configure this through the network and VM settings. Your feedback or better solution is greatly appreciated.

Thank you!

 

[Samuel Tai]

Unless your switch has LAGG support, you won't be able to benefit from creating a LAGG.

You could try creating a bridge interface hosting both NICs (which would need their IPs removed), but this can get hairy with spanning tree loops.

Without further info about your network layout, it's hard to say what the best option would be.

 

[revengineer]

Thank you for your response. My switch does not support LAGG, but it is my understanding that I could still use LAGG in failover mode. Is this correct?

This particular net work is very simple. Think 10 machines hooked to a dumb 24 port netgear switch. Also connect to the switch is a pfsense firewall which connects to the internet. The freenas server is also connected to the netgear switch.

I think my question boil down to the following: Can I use the first NIC for freenas itself and the second NIC for VMs?

If this is not possible, the bridging seems to be the only route. In that case I did not understand the comment about "spanning tree loops". If you could explain that would be great.

 

[Samuel Tai]

Thank you for your response. My switch does not support LAGG, but it is my understanding that I could still use LAGG in failover mode. Is this correct?

Yes, this is correct, but doesn't help you achieve your goal, as in failover mode, 1 NIC is active, and the other is inactive standby.

I did not understand the comment about "spanning tree loops". If you could explain that would be great.

An Ethernet network cannot have any physical loops, as broadcasts would then propagate forever, and no device would then be able to transmit. To prevent this, Ethernet uses the spanning tree algorithm to detect loops and disable links to break loops. In your case, using bridging would end up essentially doing the same as active/standby LAGG, as the FreeNAS server would introduce a deliberate loop into your network, and spanning tree would then end up disabling one NIC anyway.

If you want to use the second NIC for VMs, you could do something like this:

Create a 2nd subnet on your pfSense firewall, and connect the FreeNAS 2nd NIC to that. You'll need to configure your pfSense DNS/DHCP to manage the 2nd subnet's VM entries. On the FreeNAS server, the VMs would use the pfSense 2nd subnet IP as their default gateway. You'll also need to create rules on pfSense allowing the PCs to reach the VMs.

In this setup, I recommend you convert the FreeNAS server to static addressing, to prevent multiple default gateways configured on the server itself. The server should have its default gateway on the original subnet. The 2nd subnet should NOT have a default gateway. Only the VMs on that subnet should have a default gateway.

To force all your jails and VMs to the 2nd NIC, you'll need to create a bridge0 interface, and manually assign the 2nd NIC to it. Then any vnet or tap interfaces jails and VMs use will automatically bind to bridge0 and use the 2nd NIC.

 

[revengineer]

Thanks very much again. I now understand the loop issue. The suggested network configuration is what I thought is required but I was hoping for a shortcut. Thanks for setting me straight. This will take some re-engineering. I think for now I will just use the second NIC as failover with the understanding that it will not yield the advantage I was trying to achieve.

 

[Samuel Tai]

As I was returning home from errands, it occurred to me that since pfSense is FreeBSD-based, you could create a pfSense LAGG interface to terminate the FreeNAS LAGG instead of the switch. Then on pfSense, you'd bridge its LAGG interface to the inside LAN, and you'd have a working LAGG on your FreeNAS server with both NICs active.

 

[revengineer]

This is actually a great idea, which I had not thought of. It will require a NIC upgrade for my pfsense to accommodate the freenas server. I was planning for an upgrade anyway for other reasons.