Strange network issue

[Stevan44]

After upgrading to 12.0-U3.1. My network access is down, no Web access or shares. My network has not changed so it related to True NAS network stack or something is not loading during the boot process.

I can go to shell and ping the TrueNAS em0 IP address, but not the gateway. I've tried changing the IP address, I've notice that it changes the IP address but it says restarting network failed. I have three network cards (Intel) but none of them can communicate with my network. I believe my storage pools are intact and working.

1) I've tested the NAS switch port and cable by hooking it up to a laptop, it got on the network with DHCP enabled. So it's not the switch or cable.
2) Next I created a USB boot drive with fresh TrueNAS install, the network, tried using static IP and DHCP, also a gateway has been given. It can't ping the gateway just as the existing setup.

What would be the best thing to do in order to fix this? Would resetting the configuration to default fix this issue? I believe I can import or restore my existing pools, so I won't lose my data.

Thanks
Steve

 

[Arwen]

You should try booting to the TrueNAS version prior to the update. That is part of the purpose of alternate boot environments, easy back out plan.

This would help in trouble shooting because if it works prior to the update, then something is broke in the update.

 

[Stevan44]

THnaks for the suggestion. I'm going to do just that. I found an old 11.2 version that I ran off my usb drive. I will restore it to a usb and boot off it. I will post the results soon.

 

[Arwen]

Uh, it has been a while, but you should already have the prior boot environment on your boot disk(s). Unless you deleted them, which is unlikely because of the network access to the GUI won't work, a simple reboot from the console should give you a boot time menu.

This is one strength of ZFS. It is possible to snapshot the existing OS, clone it, patch the clone and then reboot to the clone / patched OS. Solaris 11 does this, and I do this with my home Linux computers. TrueNAS started doing this about the 9.10 version if I remember correctly. Using these "alternate boot environments" has the advantage of more or less instant back out by simply rebooting and selecting the prior boot environment.

 

[Stevan44]

After booting to an older FreeNAS USB, the problem still persist.

At this point it's something to do with NAS network stack. Can't get IP from DHCP or can't ping gateway.

I plugged in my laptop to the switch port works that the NAS uses, got and IP address from DHCP so I know that the port works.
I moved the NAS to a hub and add a laptop as well but they could not see each other.
Switched LAN cable from the switch to NAS port, problem still persists (Host is down). So it's not the original cable.
I have three network (Intel one single port, one dual ports card) on the NAS but, the issues happen to all three ports. It doesn't seem that all the ports could be bad at once.

I've ran out of ideas on how to identify or troubleshoot what the issue is?

 

[Stevan44]

This issue is similar to mines except he could ping the gateway,

Can't connect to WebUI after update/reboot or ping server

I didn't follow read only friday and updated my server to 9.10. Since then, I haven't been able to access my server no matter what I do, it's the weirdest thing. I've tried factory reset and putting the network config back in, I've tried a full reinstall from scratch, I've tried DHCP, I've tried...

www.truenas.com

Since I tried using a new boot disk and it failed, I don't think using prior boot environment boot disk will work either.

 

[Arwen]

Then sorry, I am out of ideas. Perhaps someone else here in the forums can help.

 

[Stevan44]

Thanks for trying. Judging from the other case that I pointed out no one solved that one too. I hoping that someone may had this issue, but I guess not.

 

[Stevan44]

Finally figured it out.

It has something to do with TrueNAS-13.0-U5.2 and single network connection with link aggregation setup for more other 2 LAN cards at the same time. Updates prior to 13 U5.2 worked with the network settings that were currently in use. TrueNAS-13.0-U5.2 did not like those setting and even when I manually tried to set a static IP on my single LAN card while disabling link aggregation it would not work. I guess the network stack hangs up after so many attempts to delete or clear NAS network interfaces.

So I took a break, wake up the next morning and turn on the NAS. The single LAN card got an IP address, it was wrong one. But that was easily fixed by updating the DHCP scope with the new IP. Reload the NAS and it got the correct IP address!

Next was a matter for playing around with link aggregation that was working before the upgrade.

I used these steps to finally get working. In my case I have:
em0 and em1 - dual port network card
em2 - single network card.

1) From the shell I turn off the DHCP setting deleted my em2 card to stop getting an IP address.
2) First setup link Aggregation using em0 and em1 with static IP of your choice
3) Test ping link aggregation IP and make sure it pings continuously so that you can see if it stops while making changes to the following network settings.
4) Made sure em2 is setup using DHCP. Test ping em2 IP (ping it continuously so that you can see if it stops)
5) Play around with turning on and off em2 until it gets it's DHCP IP and both IP's pings.

I hope that this will help someone who maybe having similar issues. You can close this issue out.

 

[Arwen]

Glad you figured it out and it is solved.


One note, FreeBSD / TrueNAS Core does not support multiple NICs on the same sub-net, except in an Aggregation. So if your DHCP & Static IPs are on the same sub-net, that can cause problems. I forgot to ask about that.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com