Plug NAS Unit into Switch, Whole Network goes Down

[isit.tech01]

Hello, I am fairly new to the TrueNAS Community and the forums. Trying to help my boss with an issue we have going on.

The issue is whenever we plug the NAS unit into a switch via fiber so it's connected to the network, it ends up taking the entire network down. We have checked that here isn't any overlapping vlan issues or any switch loops happening. We were thinking maybe need to change the MAC Address on the physical ethernet port itself.

If you all need anymore information (which I'm sure you'll probably need), I'll be happy to provide it for you.

Thank you,
Darren

 

[Ericloewe]

Start small. Directly connect one other device. Then slowly add complexity until it breaks and work from there.

 

[isit.tech01]

The NAS is a TrueNAS Mini E Compact ZFS Storage Server with 4 Drive Bays, 8GB RAM, Dual Core CPU, Quad 1 Gigabit Network with a fiber card to connect to the switch.

 

[Ericloewe]

I mean, sure, that's not irrelevant information, but you have too many variables at the moment. As I said, try some simpler connections and see how that goes. It will allow you to narrow down the problem.

 

[isit.tech01]

Does anyone know how, when, and why Promiscuous mode occurs?

 

[isit.tech01]

After some digging around it turned out to be a switch bug. But we ran into another problem. We have 5 VLANs setup on the same SPF+ port and only half are Promiscuous mode. Anyone know why this would be like that? What determines if an interface is Promiscuous or not in the FreeNAS Interface?

 

[Ericloewe]

Promiscuous mode is generically used to either do L2 bridging or to sniff packets on the network (e.g. redirected to it by a switch so configured, in a debug setting).

 

[Patrick M. Hausen]

The output of ifconfig on your TrueNAS might give us a starting point. Please do not post screenshots but just the text - thank you.

 

[webdawg]

This is my tech.

We are running up against a strange issue. The only way to fix it was to add "link de:ad:be:ef:ca:fe" to the ifconfig options for the vlan interface.

When we plug the SFP+ link into the Mikrotik switch, the entire 10.0.20.0/24 network would go down, hell it could have been everything.

I think it is a Mikrotik switch bug, and the only difference I can find so far (because I have many sfp+ systems, that access multiple vlans on the same interface) was that on the interface that works I have EVERY vlan checked, and on the interface that would cripple the network, I have just the two vlans checked that the nas unit accesses. I was thinking about testing that theory today with the tech, but who knows.

It is the only thing I can find off the bat. I have a pfsense sfp+ 30 network router that is linked the same way as the nas unit, multiple vlans, same mac address. (Mikrotik has this independent vlan lookup check box that does this: https://forum.mikrotik.com/viewtopic.php?t=64597 and it is checked on all switches)

It is crazy, and there is nothing too crazy about this network design, just that there are a ton of vlans, but when you plug the freenas unit below in without "link de:ad:be:ef:ca:fe" added to the vlan20 interface, it takes down everything.

No duplicate pings.

No duplicate mac addresses.

It is a Chelsio network card, and the replicated nas unit as the same, works with no mac address changes.

The weird part, and I am talking to pfsense about this, is that pfsense doesn't even allow mac address changes per vlan.

Here is the ifconfig output:

root@skf-star0:~ # ifconfig cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:07:43:0a:7d:4a hwaddr 00:07:43:0a:7d:4a nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet 10Gbase-Twinax <full-duplex> status: active cxgb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:07:43:0a:7d:4b hwaddr 00:07:43:0a:7d:4b nd6 options=1<PERFORMNUD> media: Ethernet none status: no carrier igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether d0:50:99:c3:9e:2c hwaddr d0:50:99:c3:9e:2c nd6 options=1<PERFORMNUD> media: Ethernet autoselect status: no carrier igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether d0:50:99:c3:9e:2d hwaddr d0:50:99:c3:9e:2d nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect status: no carrier lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> groups: lo vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: INTERNAL options=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether de:ad:be:ef:ca:fe inet 10.0.20.39 netmask 0xffffff00 broadcast 10.0.20.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet 10Gbase-Twinax <full-duplex> status: active vlan: 20 vlanpcp: 0 parent interface: cxgb0 groups: vlan vlan2702: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: 2702_SON2_MNGD options=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:07:43:0a:7d:4a inet 10.22.99.250 netmask 0xffffff00 broadcast 10.22.99.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet 10Gbase-Twinax <full-duplex> status: active vlan: 2702 vlanpcp: 0 parent interface: cxgb0 groups: vlan bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:f4:9b:04:63:14 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 2000000

 

[jgreco]

You have RSTP enabled without full vlans? Because if you do not have a coherent STP design, the exact thing you're describing is usually the end result, when the switch decides that some OTHER point in your network is a topology loop. You need PVST (per-vlan spanning tree) to be able to selectively bring up vlans in many designs. I don't think there's enough information available here to correctly analyze this, and it doesn't necessarily seem like this SHOULD be an issue with what's posted above, but, sometimes you gotta work back from the symptoms. What are you using for STP on the Mikrotik? And yes I see this "independent learning" thing, but it doesn't really seem to answer the question.

 

[webdawg]

I figured it out. I moved the system to fiber months ago, and never fixed the bridges, or something else is up. Look at this:

bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:1a:9b:73:84:14 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan2702 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000 member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 2000 root@skf-star1:~ # ifconfig deletem vlan2702 ifconfig: interface deletem does not exist root@skf-star1:~ # ifconfig bridge20 deletem vlan2702 root@skf-star1:~ # ifconfig bridge20 bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:1a:9b:73:84:14 nd6 options=9<PERFORMNUD,IFDISABLED> groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 2000

The weird part is that the in the interface the bridge is configured correctly:

I am trying to figure out out to fix this, on reboot, it reverts back to the multi member bridge. Some type of bug, I guess edit the config, and re upload?

 

[webdawg]

I just tried to delete the network interface from the unit vlan2702, and re add it, reboot a bunch of times in between, in the end if vlan2702 exists it adds it. Like it is stuck in the config somewhere.

 

[webdawg]

For now to fix it, I deleted vlan2702 from freenas, and re added it as vlan12702.

There is something, even after I delete the bridge20, vlan2702, and even vlan20 that when I re add stuff with the same name that it sets the bridge back up like it was x long ago.

It must be because I did not delete the network setup when I added a nic card at some point.

This is a bug.

 

[Patrick M. Hausen]

Do you have jails with vnet_default_interface set to anything but "none"?

 

[webdawg]

does not appear so:

 

[Patrick M. Hausen]

vnet_default_interface, not vnet_interfaces ...

 

[webdawg]

Yes then:

 

[webdawg]

Hmm. The freenas file looks like SQLite.

I do not see any leftovers of vlan 2702 so far:

Pretty much the same results on the other config file. I ended up deleting some of the jails, and cleaning up some stuff.

Do you think having it set to auto would have added it to the bridge.

I mean, I was deleting the vlan2702, deleting the bridge, recreating it all AFTER reboot. I guess auto could have added vlan2702 to the bridge?

 

[Patrick M. Hausen]

When vnet_default_interface is set to auto, TrueNAS will dynamically create a bridge if necessary and put into this bridge as a member whichever interface has the default gateway. So that's a great way to create loops without knowing what hit you.

So whenever you are using mutiple VLANs and special bridge assignments it is essential to set this to none.

Down in the Network Properties section of your jail you can then set the interfaces field to e.g. vnet0:bridge20 to assign bridge20/vlan20 to that jail.

 

[isit.tech01]

That's exactly what it was.
Thank you for your help!