Bug: default route gets deleted at reboot

[sce91]

My TrueNAS setup (bare metal on an Intel i5-4590, 32 GB non-ECC RAM, 4x 4 TB WD RED) has been running happily for a few years. A few days ago however, after doing a reboot of the system, I was not be able to access the system from outside the network (not via SSH and not on the webui via VPN). From within the same LAN, no problems connecting. From the NAS there was also no internet access. The box is setup with a fixed IP, 1.1.1.1 and 8.8.8.8 as DNS, and my router as default gateway.

It turned out, no default route was setup (netstat -r only listed loopback and my LAN, no default route). Even though a default gateway was setup in Network -> Global configuration, I had to manually add 0.0.0.0/0->my default gateway in Network->Static Routes. I have not touched any of the network configuration, and I would expect a default route to my default gateway to be present by default.

Is this a bug? Has anyone else had this issue? Everything is working again after adding the static route manually, but I feel like there is some setting that I have misconfigured.

 

[foretell.concert]

I just hit the same thing!

Long story short, I used to have a cloud sync task that hit up a FTP server that's only got an `A` record associated with it.
Disabled that task months ago as I didn't need it until recently.
Re-enabled it and found that I couldn't save due to a failure to connect to the remote host.

```shell
$ curl google.com # This would work; connection would be made to the AAAA address
$ curl -4 google.com # This would fail almost instantly with: curl: (7) Failed to connect to google.com port 80 after 4 ms: Couldn't connect to server
```

So that clued me in to an IPv4 related issue.
Popped over to network overview and ... yep, I have exactly ONE default route listed, it's for the `fe80::` link to my router... there should ALSO be a default route for IPv4... but there is none :(.

 

[sce91]

Has no one else encountered this issue or knows a solution? The issue still persists: After a reboot, the default route gets deleted and the Network Overview shows empty under 'static routes'. TrueNAS is unable to connect to anything outside my local LAN. After every reboot, I have to hop over to Global Configuration, change the default gateway to something random, hit save, change the default gateway to my router LAN ip again, hit save, and then the default route is added again and TrueNAS can connect to the internet. Pretty annoying bug when I have to reboot remotely.

 

[Patrick M. Hausen]

Do you have IPv6 autoconfiguration enabled? Are you using only IPv4 and could turn it off?

 

[foretell.concert]

Do you have IPv6 autoconfiguration enabled? Are you using only IPv4 and could turn it off?

I can't speak for @sce91 but I have two interfaces and a fixed DHCPv4 lease for one (static v4 assignment on the other). IPv6 auto-configure enabled on the one interface with a static IPv4 address.
Frustratingly, truenas does not permit ipv6 auto configure on both...

One interface is the same subnet as my persistent storage workloads (k8s mostly) and the other interface is my "everything else" interface (misc workstations, other nfs/smb clients across the network).

I run a dual-stack network as I am in the process of (slowly!) transitioning everything to IPv6 but for now it's a ~50/50 mix of ipv4 and ipv6 traffic.

Oddly enough, there's something _strange_ going on. I went and looked at my settings to confirm a few things before replying and I no longer have a static route configured... despite doing so earlier / since the last reboot.

As of $now, the dashboard reports
Uptime: 80 days, 22:44

But here's the kicker, the last time I posted in this thread was 2024-01-02 where I encountered this issue. $now minus 80 days is 2023-12-10... which is further in the past than when I posted in this thread on january second.

I am at a loss for words; I now have the correct IPv4 and IPv6 default routes and no static route configured...

 

[Patrick M. Hausen]

So you have one interface with IPv4 DHCP? That will of course reset a static default route every time. You cannot have more than one default route and only a single interface configured via DHCP.

TrueNAS does not run a separate IP stack per interface. It's a single stack, single routing table, single everything. If your DHCP server hands out a default gateway this becomes THE DEFAULT ROUTE. Any static ones will be replaced.

 

[foretell.concert]

TrueNAS does not run a separate IP stack per interface. It's a single stack, single routing table, single everything. If your DHCP server hands out a default gateway this becomes THE DEFAULT ROUTE. Any static ones will be replaced.

That's not super intuitive but would explain it. I try to avoid static configurations on _each_ device because every device has a unique interface/steps/quirks/bugs...etc. Much easier to manage from the router's interface which is at least consistent and allows me to see everything at a glance.

Sounds like it's worth doing the entire IPv4 configuration across all interfaces statically in this case :/

 

[sce91]

Do you have IPv6 autoconfiguration enabled? Are you using only IPv4 and could turn it off?

I have ipv6 autoconfig disabled, am only using ipv4.

 

[Patrick M. Hausen]

That's not super intuitive but would explain it.

That's what every Unix host on this planet does unless there is specific network virtualisation and isolation in place.

 

[sce91]

In my case, I have only one NIC, and DHCP turned off. Not sure what causes the reset then.