Network down after 9.10.1 upgrade?

[friolator]

First, two things happpened:

1) I realized that the IPv4 Default Gateway was the wrong IP address in the FreeNAS GUI, so I changed it to the correct address.
2) Checked for FreeNAS updates and saw 9.10.1, so I installed that

Upon reboot, our 40Gb network isn't working. FreeNAS shows the link to the switch as active, the switch shows everything as ok, and the other machine on that network looks to be fine. That network is on a different subnet (using 10.0.0.2/8 for the FreeNAS server and 10.0.0.4/8 for the one workstation on the network). Our gigabit network (which the FreeNAS admin interface is accessed through), is on 192.168.1.*/24. The Default gateway I specified in step 1 is 192.168.1.1

I don't think they're related, but I figured I'd throw that out there in case I haven't had enough coffee yet.

When I check ifconfig, I see that the 40Gb network interface (cxl0) isn't getting an IP address, and that it's showing the wrong MTU (should be 9614). But the gigabit network (igb0) looks right, and has an IP address. The Admin UI shows the IP for cxl0 as 10.0.0.2/8 with the custom MTU setting.

Code:

[root@freenas] ~# ifconfig
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:07:43:28:a5:e0
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 40Gbase-CR4 <full-duplex>
    status: active
cxl1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:07:43:28:a5:e8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet none
    status: no carrier
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
    ether 0c:c4:7a:ac:26:94
    inet 192.168.1.102 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
    ether 0c:c4:7a:ac:26:95
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Any idea what's going on here? It was working fine before the upgrade/reboot.

 

[friolator]

More info - the web UI keeps giving me "An Error Occurred" warnings at the top of the page, but the console log at the bottom of the page isn't updating. Clicking on the console to expand it just loads a blank window. So something odd is happening here.

I'm rebooting the machine now via SSL, to see if that helps.

 

[CookiesLikeWhoa]

Well the IP address it has is a loopback address.

Are you able to manually configure the port via the GUI?

 

[friolator]

After a reboot, I'm now able to make changes via the GUI again without errors. As a test I changed the subnet mask on the 40gb network to /16 and it stuck. But only in the GUI. I'm still not seeing any IP address at all for the 40gb network via ifconfig.

The address you see in the ifconfig output above:

inet 192.168.1.102 netmask 0xffffff00 broadcast 192.168.1.255

Is for the gigabit network. The IP (192.168.1.102) is the correct one for that network. I am able to change that IP, and ifconfig shows the change. It doesn't work for the 40Gb network, though, even though the GUI does show any changes I make.

So at least for the 40GB the GUI and the OS are out of sync - I'm making changes in the GUI that don't seem to make it to the OS.

 

[friolator]

I'm able to change the settings manually with ifconfig, and they seem to be sticking. So the problem doesn't seem to be something catastrophic with the NIC. I'm running an iperf test now, and it's chugging along just fine.

 

[CookiesLikeWhoa]

Consider submitting a bug report?

Anything out of the norm with the iperf test?

 

[friolator]

iperf looks perfectly normal to me - other than being a bit slow because it's using the default MTU, which had a huge effect on performance. I'm currently seeing about 12Gbps speeds, but with the MTU bumped from 1500 to 9614, I was getting 21Gbps. So, I'll need to do that next.

 

[BigDave]

Does your router give out a static IP for your server based on the NIC's MAC address?

 

[friolator]

Does your router give out a static IP for your server based on the NIC's MAC address?

Nope. All IPs are set on the individual machines. The 10.*.*.* IPs aren't connected to the internet and never touch the router.

 

[friolator]

Consider submitting a bug report?

Not sure I have enough data for that to be useful. I don't know yet if it was something with the 9.10.1 update, or if something happened under the hood over the weekend when i wasn't working on the machine that caused this.

 

[friolator]

Last night when I left the office, the GUI was still not updating the network config at the OS level. When I left, the GUI had a different IP address than the one I set with ifconfig. When I came in this morning, the IP address in ifconfig matched the IP in the GUI. Now, when I make changes in the web interface, they're immediately reflected in ifconfig.

Seriously weird.