Register for the iXsystems Community to get an ad-free experience

Multiple interfaces, same subnet, different switches

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

virtualdxs

Dabbler
Joined
Nov 19, 2018
Messages
34
Hi. I just got a FreeNAS server up and running, and was disappointed to find out I can't have multiple interfaces on the same subnet. I read in multiple places that the solution is LACP, except that's not possible in this situation. I have 4 10Gb NICs on this server. I also have two 10Gb switches. I have two ports from the server going to each switch. Both switches are on the same subnet, and there is a 20Gb trunk going between them. Every other 10Gb device has one port on each switch. Each port has an IP assigned to the host for it.

Did I design the network badly? My goal was maximum resiliency, so I did it this way so that if device A lost its link to switch 1 and device B lost its link to switch 2, they could still communicate via A -> 1 -> 2 -> B. Am I trying to do something that's not (sanely) possible without a high-speed router?
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,286
Hi Virtualdxs,

Indeed, you designed it wrong...

if device A lost its link to switch 1 and device B lost its link to switch 2, they could still communicate via A -> 1 -> 2 -> B

So if A lost access to switch 1, it can not send it to 1. In the same way, if B looses access to switch 2, how do you think it can receive anything directly from 2 ?

You said that you have an uplink between the 2 switches. Are they coupled in a logical stack and managed as a single switch or are they just 2 independent broadcast domains contaminating each other ?

Are these switches layer 2 ou 3 ?

To access network 2 from network 1 should be done by routing.
You can not do redundancy at layer 2 like you do at layer 3. To have redundancy means to have more than one path for the same source - destination pair.
If you have more than 1 path at layer 3, you are good. But at layer 2, that creates a loop and a layer 2 storm that will kill your network instantly.

For a redundant path at layer 2, you need STP to detect and disable one branch of the loop and re-activate it should the preferred one goes down.

You need to described your entire setup. Not just the FreeNAS server and 10% of your switches. You need to give the complete config of both your switches, all your servers and what is it that you try to achieve.

But from what I can see, you are heading straight for a max of trouble...
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,103
I did, in fact, read that. Nowhere does it cover my scenario of multiple switches.
As you stated earlier your switches are on the same subnet, though you didn't provide enough information to answer this fully. If the switches are all intercommunicating on the same subnet, then you would need a LAGG, though it sounds like a hot mess over there. You should look into getting the network sorted out first.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,085
I can see that you're trying to create redundancy here but the way you're doing it is not the right way.
You have probably created many logical loops inside your network that are actually damaging the performance overall.
 

virtualdxs

Dabbler
Joined
Nov 19, 2018
Messages
34
So if A lost access to switch 1, it can not send it to 1. In the same way, if B looses access to switch 2, how do you think it can receive anything directly from 2?

I apologize, I meant A -> 2 -> 1 -> B.

You said that you have an uplink between the 2 switches. Are they coupled in a logical stack and managed as a single switch or are they just 2 independent broadcast domains contaminating each other?

They are two separate switches with one broadcast domain between them. There's no network loops because I'm running MSTP, and the interconnect between the switches is LAGged.

Are these switches layer 2 or 3?

Layer 2 with some 3 capabilities, but would almost certainly not be able to route 10Gb+ of traffic even without firewall.

To access network 2 from network 1 should be done by routing.

That adds latency (although that's not a big deal in my environment), but more importantly requires a router that can route tens of gigabits of traffic.

You can not do redundancy at layer 2 like you do at layer 3. To have redundancy means to have more than one path for the same source - destination pair.
If you have more than 1 path at layer 3, you are good. But at layer 2, that creates a loop and a layer 2 storm that will kill your network instantly.

If you use STP of some form, then you don't have to worry about loops. Also, since the hosts aren't bridging, that shouldn't be an issue for the case of a host having an uplink to two switches, correct?

For a redundant path at layer 2, you need STP to detect and disable one branch of the loop and re-activate it should the preferred one goes down.

Again, at the switch level, but hosts with multiple uplinks to separate switches should be fine, right?

You need to describe your entire setup. Not just the FreeNAS server and 10% of your switches. You need to give the complete config of both your switches, all your servers and what is it that you try to achieve.

I described the FreeNAS server and 100% of the relevant switches. Other switches exist on other networks, but the storage network only exists on these two switches. Here's a sort-of diagram:

Code:
---------------
|   FreeNAS   |
|  Ports 1&2  |===A
|  Ports 3&4  |===B
---------------
                                 --------------------
---------------                  | Storage Switch A |
| vm server 1 |                  --------------------
|   Port  1   |===A
|   Port  2   |===B
---------------

---------------
| vm server 2 |
|   Port  1   |===A
|   Port  2   |===B
---------------

---------------
| db server 1 |                  --------------------
|   Port  1   |===A              | Storage Switch B |
|   Port  2   |===B              --------------------
---------------

---------------
| db server 2 |
|   Port  1   |===A
|   Port  2   |===B
---------------
 

virtualdxs

Dabbler
Joined
Nov 19, 2018
Messages
34
As you stated earlier your switches are on the same subnet, though you didn't provide enough information to answer this fully. If the switches are all intercommunicating on the same subnet, then you would need a LAGG, though it sounds like a hot mess over there. You should look into getting the network sorted out first.
I'd love to do that. I couldn't find any direction on the Internet for setting up a dual-fabric network like this one, so I put together what made sense in my head. I would graciously accept any help designing this network.
 

virtualdxs

Dabbler
Joined
Nov 19, 2018
Messages
34
I can see that you're trying to create redundancy here but the way you're doing it is not the right way.
You have probably created many logical loops inside your network that are actually damaging the performance overall.
See my previous responses; I'm trying to figure out the best way to do this maximizing resiliency and performance. Loops are not a problem.
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,286
Hi again Virtualdxs,

Thanks for the diagram, that explains your design and goal in a much better way.

So you are clearly running 2 separate networks : Network A and Network B.

Lets start from the beginning :
You create two IP subnets and give one IP in each subnet to each server. You use name service (DNS or Host) to point each names to both IPs. When fail over is needed, the client will timeout on first IP and will move to the second IP after that. Redundancy is achieved by switching from Network A to B, but not instantly. Also, a client in Network A can not reach a server in Network B.

Because you said your router was not that powerful, an option would be to rely on it only during a fail over. The router would be needed only when one server looses Network A and the other, Network B. That very rare condition means lower trafic and the degraded state may be acceptable for that temporary situation. Would that be enough ?

If you really need to do more than that, then you need professional switches that you can stack together to create a single virtual switch made of 2 physical ones. Using a single logical switch, you can do the kind of fault tolerance you seem to try to build. A single virtual switch will let you configure 2 NIC in every server, plug one in each of the physical parts of the virtual switch and do complete, automatic fail over when needed.

To improvise enterprise-class redundancy with cheap home hardware will just not work.

Good luck with your setup,
 

virtualdxs

Dabbler
Joined
Nov 19, 2018
Messages
34
Lets start from the beginning :
You create two IP subnets and give one IP in each subnet to each server. You use name service (DNS or Host) to point each names to both IPs. When fail over is needed, the client will timeout on first IP and will move to the second IP after that. Redundancy is achieved by switching from Network A to B, but not instantly. Also, a client in Network A can not reach a server in Network B.

Alright, so I'll have to settle for this with the two fabrics not talking to each other at layer 2.

Because you said your router was not that powerful, an option would be to rely on it only during a fail over. The router would be needed only when one server loses Network A and the other, Network B. That very rare condition means lower traffic and the degraded state may be acceptable for that temporary situation. Would that be enough?

You make a very good point, but I don't really see a way to tell the hosts to only use a path that crosses networks when all other paths are failed.

If you really need to do more than that, then you need professional switches that you can stack together to create a single virtual switch made of 2 physical ones. Using a single logical switch, you can do the kind of fault tolerance you seem to try to build. A single virtual switch will let you configure 2 NIC in every server, plug one in each of the physical parts of the virtual switch and do complete, automatic fail over when needed.

To improvise enterprise-class redundancy with cheap home hardware will just not work.

Good luck with your setup.

I guess that's fair. It's not cheap home hardware but it's not enterprise either.

Thanks for your help!
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,286
Hi again Virtualdxs,

but I don't really see a way to tell the hosts to only use a path that crosses networks when all other paths are failed.

This is pretty easy actually... Just do plain nothing and you will have it :smile:

When a system tries to reach an IP address, it first looks in its own routing table. Should it find a path to that destination, the shortest path will tell which source IP and which NIC interface to use for the connection. So here, because both the client and server are connected to both A and B, no matter which IP the client receives from the DNS, it will use the same network as the target.

Should that NIC goes down, then the system will have only one NIC to use and it will rely on its default routing table to find a next hop that will let it reach the target.

That is plain TCP / IP basic 101...

Have fun with your setup,
 
Joined
Dec 29, 2014
Messages
1,061
If you want to have them both active, that is more of a challenge if you don't have LACP or something to bond the connections together. If you are just looking for redundancy, you could use a failover LAGG like below.
1553691328324.png

It would always use the first NIC unless it was down. Then it would fail over to the second NIC. That assumes both switch are intended to be part of the same L2/L3 network.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
16,446
So at the end of the day there's a lot of stuff here.

Look at a typical design that's aimed at providing 100% redundancy.

https://extranet.www.sol.net/files/freenas/misc/hypervisor-vswitch-to-physical.jpg

This is a concept diagram I use for training new network engineers. The ugly sideways-H thing is a single hypervisor with two vSwitches. You would normally have a rack full of them, but to keep it clear, there's only one hypervisor shown. The other slightly confusing bit is that this means to show that storage0 runs *through* core0 and storage1 runs *through* core1.

So you have two top-of-rack switches, labeled core0 and core1, that are part of the data center network, which means that they are tied together and have the same set of VLANs available. They are fully redundant switches.

Now, ESXi and iSCSI have a quirk, and that quirk is that when you have multiple physical interfaces on a vSwitch connecting to a network, you need to bind iSCSI and IP through a single physical interface. (Die-hard storage enthusiasts will note that there is now a way to do this with multiple vmkernel interfaces and multiple IP's, but let's keep it simple, as it boils down to the same resolution in this scenario).

So you have some iSCSI storage servers and you want it to never lose connectivity to your ESXi. You use two networks. Simple enough. By running storage0 network (blue) through core0 switch and storage1 (green) through core1, you accomplish the basic idea of full storage redundancy, and in this configuration, it isn't even possible for storage traffic to inadvertently cross between switches.

But here's the other thing. All the other traffic, shown as red lines for networks that exist on vSwitch2 and yellow for vSwitch3, also have redundant paths out and into the data center networks. These networks are set as indicated so that in normal operation, iSCSI is going out one vmnic and other traffic is going out the other vmnic. This gives great flexibility to a 4x10GbE uplink scenario, which is what we do here.
 
Top