Poor Network performance between jails.

[Grinas]

Hey,

Im running multiple iocage jails on a SSD on TrueNAS-12.0-U1 and a very basic network architecture. These jails communicate with devices across my network and i get decent network speeds of 30MBs+ up and down between the jails and my Lan devices.

I have noticed that transfers between the 2 jails never goes above 5MBs. As these jails are running on a NVME SSD so i was expecting even greater speeds than I was seeing when communicating with the devices on my Lan.

The jails in question are running 12.2-RELEASE-p2 and 11.3-RELEASE-p5 and both are running within a VNET and have static IPs. I was seeing this issue when on the Freenas 11 u3 so its not related to TrueNas.


What was done so far.

1. I build the same environment using docker and did not see the same issue. Docker was running on a similar SSD.
2. Updated to Truenas
3. Updated Jails to latest release
4. All tests were carried out with the same file and I tested using multiple different RELEASES

Anyone have any ideas why could be the cause of this or know some way for me to increase network performance between Jails?

 

[sretalla]

What does your Host and jail ifconfig look like?

 

[Grinas]

HOST with all other jails shutdown

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Main
    options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
    ether 00:0c:29:6d:43:17
    inet 192.168.0.11 netmask 0xffffff00 broadcast 192.168.0.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:13:7e:0e:df:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.27 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000
    member: vnet0.23 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=1<PERFORMNUD>
vnet0.23: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: test2 as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:0c:29:81:a4:0e
    hwaddr 02:a7:c0:fb:f4:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.27: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: test3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 02:0c:29:10:4e:75
    hwaddr 02:84:e8:ed:24:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

JAIL

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:0c:29:10:4e:76
    hwaddr 02:84:e8:ed:24:0b
    inet 192.168.0.214 netmask 0xffffff00 broadcast 192.168.0.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

I just done some more testing and i get similar network bandwidth with SCP between the jails so it does not look like its iocage. I though that the app maybe just overloading the Truenas host when its running on 2 jails so i shutdown all other jails and VMs on the ESXI host just have the 2 jails communicating up and i see spikes of over 30MB but still only averaging 15MB which far less than with the docker enviroment or Jail --> LAN

The load on the server did not move when doing the test it was <0.5 with no other jails. I will admit that this server is probably under a load with all jails running as it usually has about a load of 2-5.

Weird thing is that the docker environment is running on this same ESXI host and alll the jails were running when i did my tests with that so obviously docker is far better at managing the resources compared to iocage.

 

[sretalla]

media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

Your jail interface and the vnets are at 10G virtIO, so shouldn't be stopping anything there.

The other NIC is only gigabit, but should be able to handle more than what you're mentioning, so that shouldn't be it either.

 

[Grinas]

Your jail interface and the vnets are at 10G virtIO, so shouldn't be stopping anything there.

The other NIC is only gigabit, but should be able to handle more than what you're mentioning, so that shouldn't be it either.

Do you think it is possible that the traffic is leaving the bridge network and hitting my switch and then going back again to jail2. I need to do some more testing but currently i just thing iocage is not managing the resources as well as docker.

Did the same test on docker this time with all the jails(expect 2) shutdown and VMs down and got 76MB average transfer. Thats 5 times the speed i was getting with the jails and the jail test did not have the docker VM so had more resources.