No IPv6 default route despite SLAAC addresses and received RA's (TrueNAS 13.0-U4)

[ian.m.taylor]

I'm experiencing a problem of no IPv6 default route on my TrueNAS CORE 13.0-U4 system. This problem has been posted on these forums a few times, but with either no resolution or a resolution that is not relevant for me. I apologize if there has been a solution that I just didn't see. Any help would be greatly appreciated, thank you in advance!

Problem Summary

TrueNAS is set up to autoconfigure IPv6. It receives router advertisements (both from rtsold and unsolicited) and autoconfigures an address in every advertised prefix, but does not configure the default route.

The system is running virtual machines, and these virtual machines do not have the same problem as the TrueNAS host. All VMs have default IPv6 routes to the internet and are able to 'ping -6' both LAN and public internet addresses.

Details

• Version: TrueNAS CORE 13.0-U4
• Interface configuration: Physical interface em0, manually created bridge bridge0, with DHCP and Autoconfigure IPv6 selected. Three virtual machines use bridge0 as their parent interface.

• Interface bridge0 has several IPv6 addresses (ULA and two globally routable prefixes, with privacy addresses). The bridge contains members em0 and three virtual machine interfaces. ACCEPT_RTADV is set on the interface.

• rtsold is running

• Router advertisements are being received, with nonzero router lifetime

• No default ipv6 route, even though there is a default ipv4 route and IPv6 on link routes to all the advertised prefixes.

Other relevant threads
Here are a few somewhat relevant threads that I found while looking for a solution

• No default gateway found for IPv6 - original poster just wants to disable IPv6 for a resolution
• IPv6 default route - Again not working - no replies
• No IPv6 default gateway (via SLAAC) - recommendation to manually configure the bridge, which I have already done.

 

[ian.m.taylor]

Update with more information: 'ndp -r' reports the correct default router, but the default route still doesn't appear in the routing table and I can't reach any of the IPv6 internet.

 

[perrin4869]

I am having the exact same problem, did you find a solution? I have no problem using IPv6 in my Linux box but TrueNAS can't configure the route correctly for some reason. ndp is working correctly same as you

 

[ian.m.taylor]

Unfortunately, no.

But for a workaround I have manually specified the default routes in Network > Global Configuration. This has been working for me for some time. I had to specify both the IPv4 and IPv6 routes there - I found that if I just specified the IPv6 default route, then sometimes the IPv4 route would disappear. For IPv6 default gateway, I entered "fe80::xxxx%bridge0", the same address that appears when I run "ndp -r". For IPv4 default gateway, I entered the same one that shows up when I run "netstat -4rn".

Further updates:

• I've been able to replicate this issue in a fresh installation of TrueNAS CORE 13.0-U4 in Virtualbox.
• The issue only appears on reboot. If you boot up TrueNAS with "Autoconfigure IPv6" disabled, then enable it after boot, the default route appears as expected. However if you then reboot TrueNAS, the IPv6 default route will be missing after reboot even though Autoconfigure IPv6 is now enabled. @perrin4869 can you see if you get the same behavior?
• This old Jira ticket appears to be about the same issue: https://ixsystems.atlassian.net/browse/NAS-106239. It's marked as resolved in 11.3-U4, but I'm still having it, so it must be a regression.
• This newer Jira ticket which is about the issue has 13.1-ALPHA as a fix version. So maybe a fix will be coming. https://ixsystems.atlassian.net/browse/NAS-109625

 

[Patrick M. Hausen]

If you are running bridged interfaces because of jails or VMs - did you configure all IP addresses, v4 and v6, on the bridge interface and not on any of the members? Neglecting that breaks multicast and hence SLAAC ...

 

[ian.m.taylor]

No, all IP configuration is on the bridge interface.

Screenshots of interface configuration in web UI:

Show : Screenshots of UI - interface config

ifconfig output for all interfaces:

Show : ifconfig output

VM's NIC are configured to use the bridge (example for one of two):

Show : VM NIC configuration

 

[perrin4869]

I haven't setup a bridge interface manually, it gets setup when I run jails...
In any case, I disabled jail auto-start, reboot the system, and still no default routes...

Here's my network currently:

Code:

% ifconfig
igb0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 3c:ec:ef:45:93:0a
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 3c:ec:ef:45:93:0b
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 3c:ec:ef:45:93:0c
        inet6 fe80::3eec:efff:fe45:930c%igb2 prefixlen 64 scopeid 0x3
        inet6 2001:f70:3460:5c00:3eec:efff:fe45:930c prefixlen 64 autoconf
        inet 192.168.1.108 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igb3: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 3c:ec:ef:45:93:0d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog

Code:

% netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS        igb2
127.0.0.1          link#5             UH          lo0
192.168.1.0/24     link#3             U          igb2
192.168.1.108      link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#5                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2001:f70:3460:5c00::/64           link#3                        U          igb2
2001:f70:3460:5c00:3eec:efff:fe45:930c link#3                   UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%igb2/64                    link#3                        U          igb2
fe80::3eec:efff:fe45:930c%igb2    link#3                        UHS         lo0
fe80::%lo0/64                     link#5                        U           lo0
fe80::1%lo0                       link#5                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

I have set igb2 to auto configure ipv6

 

[perrin4869]

In my case, setting the default routes manually doesn't seem to help with the routing though...

Code:

% ndp -r
fe80::ea9f:80ff:fede:7a32%igb2 if=igb2, flags=O, pref=medium, expire=29m13s
% sudo route -6 add default 'fe80::ea9f:80ff:fede:7a32%igb2'
add net default: gateway fe80::ea9f:80ff:fede:7a32%igb2
% sudo route add default 192.168.1.1
add net default: gateway 192.168.1.1
% netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS        igb2
127.0.0.1          link#5             UHS         lo0
192.168.1.0/24     link#3             U          igb2
192.168.1.108      link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           fe80::ea9f:80ff:fede:7a32%igb2 UGS       igb2
::1                               link#5                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
2001:f70:3460:5c00::/64           link#3                        U          igb2
2001:f70:3460:5c00:3eec:efff:fe45:930c link#3                   UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%igb2/64                    link#3                        U          igb2
fe80::3eec:efff:fe45:930c%igb2    link#3                        UHS         lo0
fe80::%lo0/64                     link#5                        U           lo0
fe80::1%lo0                       link#5                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
% traceroute6 google.com
traceroute6 to google.com (2404:6800:4004:823::200e) from 2001:f70:3460:5c00:3eec:efff:fe45:930c, 64 hops max, 28 byte packets
^C

 

[perrin4869]

Maybe my problem is different from @ian.m.taylor, since I can't even ping my gateway

 

[yusisushi]

Hi, I came here just to say I had the exact same issue.
IPv6 works perfectly on all my devices expect TrueNAS.

I have RA unmanaged configured in pfSense, no DHCP6. So only SLAAC in my network.

Command `ndp -r` shows the ll address of my gateway
Command `netstat -rn inet6` shows no default ipv6 route

To solve routing I had to add both default ipv4 AND default ipv6 in general network config.

I do believe this must be a bug. I spent around 3 hours tonight alone troubleshooting this.

 

[yusisushi]

Maybe my problem is different from @ian.m.taylor, since I can't even ping my gateway

Hi, while pinging your gateway from the ipv6 ll address, did you specifiy the source interface or IP?
For example

Code:

ping6 fe80::xxxx%8

or

Code:

ping6 -S fe80::aaaa fe80::xxxx

Where %8 is the source interface index or -S is the source IP

 

[yusisushi]

Additionally, it seems this is a known bug with the 'not to be fixed' status (???)

TrueNAS - Issues - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[ian.m.taylor]

Thanks for your info @yusisushi .

Just bumping this to restate that the problem has nothing to do with bridge configuration, VMs, or jails; I can replicate on a fresh install of TrueNAS CORE 13.0-U4 without jails, VMs, or a bridge.

Steps to replicate:

1. Install TrueNAS CORE 13.0-U4 on a machine connected to a subnet where an IPv6 router is sending out RA's with SLAAC enabled and an available prefix (i.e., where other machines would be able to get an IPv6 address).
2. In web GUI, confirm that "DHCP" is enabled on the only physical interface (e.g. em0), that the TrueNAS machine has an IPv4 address and internet access.
3. Check "Autoconfigure IPv6" on em0 in the web GUI.
4. Run
   Code:

   ifconfig em0

   Result: see a SLAAC address in the available prefix.
5. Run
   Code:

   ndp -r

   Result: see a default route.
6. Run
   Code:

   netstat -6rn

   Result: see the same default route.
7. Run
   Code:

   ping ipv6.google.com

   Result: successfully reaching an IPv6 host on the internet.
8. Reboot, leaving "Autoconfigure IPv6" enabled.
9. Repeat steps 4-7. Steps 4,5 work as expected, but steps 6,7 fail (no default route in "netstat -6rn" and no connectivity to the IPv6 internet)
10. Wait some time until more unsolicited RA's would have been received from the router.
11. Repeat steps 4-7. Steps 4,5 work as expected, but steps 6,7 fail (no default route in "netstat -6rn" and no connectivity to the IPv6 internet)

IPv6 autoconfiguration works the first time it's enabled, but fails after a reboot. It doesn't fix itself after more RA's are received from the router. The failure is specific to constructing the IPv6 routing table. The machine still receives RA's from the router and partially processes them (as indicated by the SLAAC addresses in the correct prefix and the route reported by "ndp -r") but the route information from the RA's is not being put in the system routing table. I do think this is a bug in TrueNAS.

Any help would be appreciated!

 

[Patrick M. Hausen]

Is rtsold running? ps awux | grep rtsold.

If it isn't, try adding a tunable, type RC, name rtsold_enable, value YES.

 

[ian.m.taylor]

@Patrick M. Hausen Yes, rtsold is running:

Show : ps awux | grep rtsold

 

[Patrick M. Hausen]

Weird. I cannot reproduce this with any of my TrueNAS installations. Typical state:

Code:

root@freenas[~]# netstat -rn|grep default
default            192.168.1.1        UGS     bridge1
default                           fe80::3eec:efff:fe00:5430%bridge1 UG  bridge1

System comes up like this after each reboot.

 

[ian.m.taylor]

That is really weird. Could it be a problem with my router's RA's? Would you mind running

Code:

tcpdump -i bridge0 -v -n "icmp6[0] == 134"

on your TrueNAS machine (replacing with your interface name) to see if the default route information contained is the same as mine?

(addresses partially censored)

Code:

# tcpdump -i bridge0 -v -n "icmp6[0] == 134"
tcpdump: listening on bridge0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:32:50.667553 IP6 (flowlabel 0x554ee, hlim 255, next-header ICMPv6 (58) payload length: 176) fe80::xxxx:xxxx:xxxx:f0c0 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 176
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
          source link-address option (1), length 8 (1): xx:xx:xx:xx:f0:c0
          mtu option (5), length 8 (1):  1480
          prefix info option (3), length 32 (4): 20xx:xxxx:xxxx::/64, Flags [onlink, auto], valid time infinity, pref. time infinity
          prefix info option (3), length 32 (4): fdxx:xxxx:xxxx::/64, Flags [onlink, auto], valid time infinity, pref. time infinity
          route info option (24), length 24 (3):  20xx:xxxx:xxxx::/48, pref=medium, lifetime=1800s
          route info option (24), length 24 (3):  fdxx:xxxx:xxxx::/48, pref=medium, lifetime=1800s
          rdnss option (25), length 24 (3):  lifetime 1800s, addr: fdxx:xxxx:xxxx::51
          advertisement interval option (7), length 8 (1):  600000ms
^C
1 packet captured
622 packets received by filter
0 packets dropped by kernel

 

[Patrick M. Hausen]

Code:

tcpdump: listening on bridge1, link-type EN10MB (Ethernet), capture size 262144 bytes
20:00:47.104224 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::3eec:efff:fe00:5430 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 64
    hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
      prefix info option (3), length 32 (4): 2003:x:yyy:zzzz::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
      mtu option (5), length 8 (1):  1500
      source link-address option (1), length 8 (1): 3c:ec:ef:00:54:30

I notice that you have set the "managed" flag which signalizes to clients to get their information via DHCPv6. I am not quite sure if TrueNAS supports that. I use strictly SLAAC throughout and since everything is dual stack here I am lazy and recursive DNS is all IPv4 only (local client to local server - of course the recursive server reaches out with both protocols).

The RA above serves to set fe80::3eec:efff:fe00:5430 as the default GW - link local for gateways is established best practice.

HTH,
Patrick

 

[ian.m.taylor]

Thanks Patrick. I tried turning off the M and O flags on my router advertisements, but I'm seeing the same behavior. (My router runs OpenWRT and those settings are available under "DHCP Server -> IPv6 RA settings" on the LAN interface.) I confirmed that no flags were set on received RA's with tcpdump, but the problem persists both on my physical TrueNAS machine and the VM with the fresh 13.0-U4 install, even after rebooting. I tested the VM both with a virtual interface set up through Virtualbox, and a USB ethernet dongle with USB passthrough.

This is a real puzzler. I'm tempted to just go with the workaround of manually specifying the default routes in global network settings, and someday maybe sidegrade to SCALE.

 

[redm0nster]

I'm seeing the same behaviour. Here's some info if it helps.

TrueNAS Core version TrueNAS-13.0-U4 on a Pentium Silver J5040 cpu (64-bit) with a single NIC. Network is a single switch behind a pfSense router/firewall, which supports both SLAAC and DHCPv6, and the pfSense "router advertisement" is set to "assisted".

My symptoms look the same as others in this thread: the TrueNAS system gets an IPv6 address with SLAAC but doesn't pick up the default route. A FreeBSD 13.1 virtual machine works just fine and gets a default route as expected, so it doesn't appear to be an issue in the base OS. Windows, Linux and Android systems all use IPv6 fine as well.

If I manually discover the router with "ping6 ff02::2%re0" and add the returned address as a default route, IPv6 works fine (public addresses partly redacted):

Code:

rsh@store:~$ ping6 -c 2 2600::
ping6: UDP connect: No route to host

rsh@store:~$ ping6 -c 2 ff02::2%re0
PING6(56=40+8+8 bytes) fe80::aaa1:59ff:fee0:17f8%re0 --> ff02::2%re0
16 bytes from fe80::20d:b9ff:fe3f:74be%re0, icmp_seq=0 hlim=64 time=0.658 ms
16 bytes from fe80::20d:b9ff:fe3f:74be%re0, icmp_seq=1 hlim=64 time=0.534 ms

--- ff02::2%re0 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.534/0.596/0.658/0.062 ms

rsh@store:~$ sudo route -6 add default fe80::20d:b9ff:fe3f:74be%re0
add net default: gateway fe80::20d:b9ff:fe3f:74be%re0

rsh@store:~$ ping6 -c 2 2600::
PING6(56=40+8+8 bytes) 2001:470:XXXX:XXX:aaa1:59ff:fee0:17f8 --> 2600::
16 bytes from 2600::, icmp_seq=0 hlim=51 time=110.736 ms
16 bytes from 2600::, icmp_seq=1 hlim=51 time=147.067 ms

--- 2600:: ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 110.736/128.902/147.067/18.166 ms

This is what my router advertisments look like in wireshark. You can see that it's advertising a default route:

Edit:

rtsold seems to be running:

Code:

rsh@store:~$ ps awux | grep rtsold
root       17144   0.0  0.0   13036   2776  -  ICs  16:21      0:00.01 /usr/sbin/rtsold -a -i
root       17148   0.0  0.0   13036   2740  -  Is   16:21      0:00.00 rtsold: rtsold.llflags (rtsold)
root       17149   0.0  0.0   13036   2736  -  Is   16:21      0:00.00 rtsold: rtsold.script (rtsold)
root       17151   0.0  0.0   13036   2732  -  Is   16:21      0:00.00 rtsold: rtsold.sendmsg (rtsold)
root       17152   0.0  0.0   13036   2740  -  Is   16:21      0:00.01 rtsold: system.syslog (rtsold)
rsh        18649   0.0  0.0   12924   2724  0  S+   17:46      0:00.00 grep rtsold

Edit2: running rtsold manually shows it sending router solicitation and receiving response, but it doesn't add the default route. This implies that the problem seems to be rtsold not adding the default route even though it is receiving the response from the router.

Code:

# Run rtsold in foreground with debug output in first ssh session. This never returns.
rsh@store:~$ sudo rtsold -i -f -d -D re0
rtsold: checking if re0 is ready...
rtsold: re0 is ready
rtsold: set timer for re0 to 0s
rtsold: timer expiration on re0, state = 1
rtsold: set timer for re0 to 4s
rtsold: New timer is 4s
rtsold: received RA from fe80::20d:b9ff:fe3f:74be on re0, state is 2
rtsold: ManagedConfigFlag on re0 is turned on
rtsold: OtherConfigFlag on re0 is turned on
rtsold: Processing RA
rtsold: ndo = 0x7fffffffe350
rtsold: ndo->nd_opt_type = 3
rtsold: ndo->nd_opt_len = 4
rtsold: ndo = 0x7fffffffe370
rtsold: ndo->nd_opt_type = 24
rtsold: ndo->nd_opt_len = 3
rtsold: ndo = 0x7fffffffe388
rtsold: ndo->nd_opt_type = 31
rtsold: ndo->nd_opt_len = 3
rtsold: labellen = 4
rtsold: labellen = 4
rtsold: dname = home.arpa
rtsold: ndo = 0x7fffffffe3a0
rtsold: ndo->nd_opt_type = 5
rtsold: ndo->nd_opt_len = 1
rtsold: ndo = 0x7fffffffe3a8
rtsold: ndo->nd_opt_type = 1
rtsold: ndo->nd_opt_len = 1
rtsold: rsid = [re0:slaac]
rtsold: write to child = search (7)
rtsold: write to child = home.arpa(9)
rtsold: write to child =  (1)
rtsold: write to child =
(1)
rtsold: script "/sbin/resolvconf" status 0
rtsold: stop timer for re0
rtsold: RA expiration timer: type=31, msg=home.arpa, expire=30m0s
rtsold: there is no timer

# Run tcpdump sniffer in another ssh session showing IPv6 ICMP traffic
rsh@store:~$ sudo tcpdump -i re0 -v icmp6
10:18:12.996501 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::aaa1:59ff:fee0:17f8 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): a8:a1:59:e0:17:f8
10:18:12.998392 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 112) fe80::20d:b9ff:fe3f:74be > fe80::aaa1:59ff:fee0:17f8: [icmp6 sum ok] ICMP6, router advertisement, length 112
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
          prefix info option (3), length 32 (4): 2001:470:1f09:b77::/64, Flags [onlink, auto], valid time 86400s, pref. time 14400s
          route info option (24), length 24 (3):  ::/0, pref=medium, lifetime=1800s
          dnssl option (31), length 24 (3):  lifetime 1800s, domain(s): home.arpa.
          mtu option (5), length 8 (1):  1500
          source link-address option (1), length 8 (1): 00:0d:b9:3f:74:be