[TrueNAS-12.0-STABLE] Replication Task and 10GbE very slow

[tanjix]

Hi Guys,

although I've been looking around the forums and found several topics about poor transfer speeds whilst doing replications tasks all these threads did not really help.

First of all, this is my setup:

iSCSI-Server
Supermicro 216 chassis with 24 2,5" slots, where 16 are currently in use by SSDs
X10DRH-IT mainboard, with 2x Intel X540-AT2 10GbE (RJ45) onboard
(and several more cards (10GB SFP+, only used for iSCSI connections), but these are not relevant here)
TrueNAS-12.0-STABLE installed

Backup-Server
Supermicro server with 4x Seagate Ironwolf 10TB hdd and 2x SSD for the TrueNAS installation
X8DTL-iF mainboard with 2x Intel 82745L ethernet controller (1 GBit)
additional Intel X540-T2 with 2x 10GbE RJ45-ports
TrueNAS-12.0-STABLE installed

I want to replicate iSCSI-LUNs from the iSCSI-Server to the Backup-Server. For that purpose, the iscsi host and the backup server are connected directly using the 10GB RJ45 ports. On both TrueNAS installations, ifconfig shows these ports as:

Code:

root@storage[~]# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:3f:90:74
        inet 10.20.10.2 netmask 0xffffff00 broadcast 10.20.10.255
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
root@storage[~]#

Code:

root@iscsi01[~]# ifconfig ix5
ix5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:34:a9:0d
        inet 10.20.10.1 netmask 0xffffff00 broadcast 10.20.10.255
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
root@iscsi01[~]#

SSH connection for these direct link works fine as well as the backup itself. However, transfer speeds for the repliucation task is not more than ~112 MBit/second which is more than painful for a 10GbE-card.

Of course, I did an iperf test on these interfaces, which surprisingly showed this:

Code:

root@iscsi01[~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.20.10.1 port 5001 connected with 10.20.10.2 port 21332
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  10.9 GBytes  9.37 Gbits/sec

Code:

root@storage[~]# iperf -c 10.20.10.1
------------------------------------------------------------
Client connecting to 10.20.10.1, TCP port 5001
TCP window size: 80.8 KByte (default)
------------------------------------------------------------
[  3] local 10.20.10.2 port 21332 connected with 10.20.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.37 Gbits/sec
root@storage[~]#

So, according to that, the hardware itself can handle 10GBit by far. But why do I only have ~100MBit on transfer speeds when doing my backups to another TrueNAS host?

Maybe someone can advise on what I could try or do?

Thanks a lot!

 

[tanjix]

Follow-Up: Enabling Jumbo-Frames (MTU 9000) did not help either.

I also discover the same issue, when using 1GBit ports as well. There is no better speed than around 100 MBit.
Since 00:00am the next backup job started and transfers around 700GB of data. I implemented a grafana setup as well; please see attched the graphs for:

Backup through 1 GBit interface
Backup through 10 GBit interface.

Both show almost the same transfer speeds, which are incredibly low.

Thanks for any input and advise.

 

[nick_]

The next thing I'd try is skipping the network entirely. What speeds do you get with a zfs send of the source dataset to /dev/null? If the source is fast, benchmark a local receive on the destination. Either end being slow can limit the whole thing.

 

[junior466]

Follow-Up: Enabling Jumbo-Frames (MTU 9000) did not help either.

I also discover the same issue, when using 1GBit ports as well. There is no better speed than around 100 MBit.
Since 00:00am the next backup job started and transfers around 700GB of data. I implemented a grafana setup as well; please see attched the graphs for:

Backup through 1 GBit interface
Backup through 10 GBit interface.

Both show almost the same transfer speeds, which are incredibly low.

Thanks for any input and advise.

Sorry to revive this but have you ever found a solution? I am seeing the exact same issue.

 

[tanjix]

Sorry to revive this but have you ever found a solution? I am seeing the exact same issue.

Unfortunately, I haven't found a solution on that until now :(