SOLVED VNET jails no network, plugins installation failing after nic upgrade.

nas-noob · Oct 8, 2021

Hello friends, I am trying to resolve this problem for hours, but it looks out it's out of my scope.

I've updated my network to 10gbe recently and from that point, my jails lost network access.

I did iocage clean -a, to start fresh. but that didn't help.

I can't make new jail using vnet with working network. I can ping server, but can't ping gateway.

Jails have internet access if they are created without vnet option.

Plugins are failing on installation with message:
Error: plex had a failure Exception: RuntimeError Message: pkg.FreeBSD.org could not be reached via DNS, check plex's network configuration Partial plugin destroyed

I guess it's connected with vnet not working...

Please see my server and jails configuration below, hope somebody more skillful can help me out :) Tnx

On the server ifconfig -a

Code:

igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d0:50:99:d0:06:ee
    media: Ethernet autoselect
    status: no carrier
    nd6 options=1<PERFORMNUD>
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
    ether d0:50:99:d0:06:ef
    media: Ethernet autoselect
    status: no carrier
    nd6 options=1<PERFORMNUD>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:c5:a1:da:99:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: aq0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
    nd6 options=1<PERFORMNUD>
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: test1 as nic: epair0b
    options=8<VLAN_MTU>
    ether f2:2f:74:35:09:16
    hwaddr 02:79:64:79:57:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

On the server netstat -rn

Code:

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS         aq0
127.0.0.1          link#4             UH          lo0
192.168.1.0/24     link#1             U           aq0
192.168.1.10       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#4                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#4                        U           lo0
fe80::1%lo0                       link#4                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

In the jail ifconfig -a

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether f2:2f:74:35:09:17
    hwaddr 02:79:64:79:57:0b
    inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

in the jail netstat -rn

Code:

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS     epair0b
127.0.0.1          link#1             UH          lo0
192.168.1.0/24     link#3             U       epair0b
192.168.1.20       link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#1                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

iocoage get all test1

Code:

CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:0
bpf:0
children_max:0
cloned_release:12.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.1.1
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:test1
host_hostuuid:test1
host_time:1
hostid:00000000-0000-0000-0000-D05099D006EE
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|192.168.1.20/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/test1/data
jail_zfs_mountpoint:none
last_started:2021-10-08 09:29:24
localhost_ip:none
login_flags:-f root
mac_prefix:f22f74
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.2-RELEASE-p10
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:f22f74350916 f22f74350917
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:aq0
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off

nas-noob · Oct 8, 2021

I've reinstalled truenas from the from the scratch, problem persisted.

Now I assume it has something to do with my new 10GbE network.

New hardware is NIC is ASUS XG-C100C using Aquantia AQC-107 chipset and TP Link tl-sx105 10GbE switch.

Internet is working normal from truenas (it can download updates) and 10GbE local network is working at full speed.

Don't know where to go from here...

Samuel Tai · Oct 9, 2021

You'll have to try a different NIC, unfortunately. The Aquantia driver included in TrueNAS is a developer preview alpha. This is the only one available for FreeBSD, on which TrueNAS is based. It's included only for experimentation, not for production use, sorry.

What's tried and true in TrueNAS are the Chelsios for 10Gbps Ethernet.

nas-noob · Oct 9, 2021

Hey Samulel, thank you for your answer.

I started questioning this driver myself, as I connected without new tp-link switch and problem persisted. Only component left is Asus NIC and I knew that this driver is not best, but was thinking it can still work well enough.

From my digging it looks that driver has problem with promiscuous mode and even setting custiom command promisc up doesn't help.

The problem is that 10Gbe NIC choice in Australia is much worse than in US, it's hard to find Chelsios card at normal price.

Only card from supported list I could find at decent price in Australian shops is following:

https://www.scorptec.com.au/product/networking/adapters/64981-x550t2

it has intel x550-at2 chipset, do you are anyone alse has experiance with this chipset as I would not like to burn $500 :)

A saw there is x550-at2 and x550-bt2 chipset, don't how different they are with truenas...

Tnx

Samuel Tai · Oct 9, 2021

Yes, the Intel X550s are supported, and the driver is built into the kernel.

nas-noob · Oct 9, 2021

Thanks again, I've just made my order, tomorrow will share results with x550 NIC.

nas-noob · Oct 10, 2021

X550 is up and running well. All problems solved.

Hope that this can help people experiencing similar issues in the future.

Don't buy ASUS XG-C100C for Turenas! (At least if stable driver is not developed)

Thank you for your help.

Samuel Tai · Oct 11, 2021

You're very welcome. Glad you're up & running.

FrancoisC · Oct 28, 2021

nas-noob said:
X550 is up and running well. All problems solved.

Hope that this can help people experiencing similar issues in the future.

Don't buy ASUS XG-C100C for Turenas! (At least if stable driver is not developed)

Thank you for your help.

Damn, just saw this... Same problem.
Managed to have the GUI to work but not the jails.

Ordering X550...

FrancoisC · Oct 28, 2021

If someone read this, this 10gb adapter does work.

Here is how I did it:

PCIe 10Gb ethernet card setup issue

Good morning all, I've lost few hours trying to find the solution to my problem. I've come to the point where I will need external support. I bought for my NAS an ASUS XG-C100C PCIe card. This is based on Aquantia AQC-107 controller. Starting up the NAS, the card is detected and I can access...

www.truenas.com

Important Announcement for the TrueNAS Community.

SOLVED VNET jails no network, plugins installation failing after nic upgrade.

nas-noob

Cadet

nas-noob

Cadet

Samuel Tai

Never underestimate your own stupidity

nas-noob

Cadet

Samuel Tai

Never underestimate your own stupidity

nas-noob

Cadet

nas-noob

Cadet

Samuel Tai

Never underestimate your own stupidity

FrancoisC

Dabbler

FrancoisC

Dabbler

PCIe 10Gb ethernet card setup issue

Similar threads