WebGUI/SSH/sharing broken after 9.3 minor update

[Bart Kniess]

I've been meaning to reboot my FreeNAS to try to fix the "no entry found" plugins issue I've had for a few weeks now. I saw some updates available, so I pulled those down and applied them, then proceeded to reboot. No major errors that I could see in that process. The NAS came back 'alive' (had a continuous ping going), but I have not been able to reach the GUI or SSH at all since the update. I went from FreeNAS-9.3-STABLE-201602031011 to FreeNAS-9.3-STABLE-201604150515. The only thing jumping out to me are these two items from the most recent dmesg:

Apr 26 01:04:27 HOSTNAME zfsd: DevfsEvent::ReadLabel: /dev/da3p2: Unable to extract pool GUID from vdev label data.
Apr 26 01:04:29 HOSTNAME syslog-ng[15364]: syslog-ng starting up; version='3.5.6'
Apr 26 01:04:29 HOSTNAME kernel: pid 2840 (syslog-ng), uid 0: exited on signal 6 (core dumped)

DA3 is one of my SSD's. zpool status runs clean, so I'm not sure if that error is something cosmetic/red herring in nature.

All mounts look fine, iSCSI is humming along beautifully to both of my ESXi hosts. VM's are fired up and functional.

Jails appear to be fully functional as well.

Pretty much looks like any direct connectivity to the mgmt plane of the NAS is fubar'd. Mgmt itself including HTTP/SSH, CIFS, NFS. But again, ICMP has been rock solid without issues, both within the same network segment as well as outside (not a GW issue).

Network-topology wise...
(4) NICs in use:
(2) in a LAGG for mgmt/CIFS/NFS/FTP/etc. (VLAN tagged)
(2) for iSCSI, separate links and VLANs (no bonding)

Any direction would be appreciated, thanks.

 

[Robert Trevellyan]

More information required.

 

[Bart Kniess]

* FreeNAS version: FreeNAS-9.3-STABLE-201604150515
* Hardware: Supermicro 2U chassis
* Motherboard: Supermicro X8DT6-F
* CPU: E5645
* RAM: 96 GB ECC
* NIC Hardware: Built-in dual-port Intel-based for mgmt/front-end shares
* NIC Hardware: Add-on quad-port Intel-based (EXPI9404PTLBLK) for iSCSI back-end for ESX hosts

For network connectivity, include details on your network hardware/configuration.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
ether 00:1b:21:4e:a2:19
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:4e:a2:18
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em2: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:4e:a2:1b
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
em3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:1b:21:4e:a2:1a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
em4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM>
ether 00:25:90:64:9e:b0
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM>
ether 00:25:90:64:9e:b0
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x10
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWCSUM>
ether 00:25:90:64:9e:b0
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
laggproto roundrobin lagghash l2,l3,l4
laggport: em5 flags=4<ACTIVE>
laggport: em4 flags=4<ACTIVE>
vlan111: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:1b:21:4e:a2:19
inet 192.168.111.52 netmask 0xffffff00 broadcast 192.168.111.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 111 parent interface: em0
vlan112: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:1b:21:4e:a2:18
inet 192.168.112.52 netmask 0xffffff00 broadcast 192.168.112.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlan: 112 parent interface: em1
vlan101: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:25:90:64:9e:b0
inet 192.168.101.52 netmask 0xffffff00 broadcast 192.168.101.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
vlan: 101 parent interface: lagg0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:97:40:d5:15:00
nd6 options=1<PERFORMNUD>
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 27 priority 128 path cost 2000
member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 26 priority 128 path cost 2000
member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 25 priority 128 path cost 2000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 24 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 23 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 22 priority 128 path cost 2000
member: vlan101 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 20 priority 128 path cost 19801
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:49:c0:00:16:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:67:fc:00:17:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:40:50:00:18:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:06:f6:00:19:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair4a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:a4:8f:00:1a:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
epair5a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:1b:ed:00:1b:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active

 

[Robert Trevellyan]

I doubt this is a network issue.

What do you use that SSD for?

 

[Bart Kniess]

There are a pair of SSD's. Mirrored ZIL, remaining used for L2ARC. zpool status shows everything clean, online, 0 errors.

I agree it doesn't seem to be network-related, at least not directly. When I run a port scan, it looks as though FreeNAS is only listening/responding on 25 (smtp) and 80 (www). If I try to telnet to it on port 80, it seems to answer, but a "get /" just gets a connection closed vs. any actual HTML back. Essentially the same as trying to hit it from a browser, just sits there and bombs eventually (ERR_EMPTY_RESPONSE in Chrome). Same with trying SSH... the port scan shows it not responding at all, but I don't get a connection refused on it, just sits for a while then times out.

To me it feels like some critical daemons got jacked up somewhere in the update process. It's just strange to me all the "back-end" functionality is there w/ iSCSI and so forth. If nobody has any ideas, I guess my paths are to either try to roll back or to try 9.10?

 

[Robert Trevellyan]

It seems like FreeBSD is up and running but the FreeNAS middleware is broken. Many times this is a symptom of lack of RAM, but that seems unlikely in your case. It might be as simple as a corrupted boot device, so it might be worth trying to roll back (assuming a simple reboot doesn't fix it). Another strategy is just to do a clean install on a new boot device and then upload your saved config.