my 10GbE performance

[katsuwo]

Hello.
I am using FreeNAS 11.3-U4.1.
Recently I upgraded my system's network to 10GbE,
The improvement in transfer rate was below my expectations.

I wrote data of 7.4GByte by using Windows 10 as an initiator with iSCSI.
Data is transferred at a speed of 1.5 GByte/sec for about 3 seconds from the start of copying,
After that, the transfer speed drops and goes back and forth between 10MB-200MB/sec.
I understand that at the time of the first fast transfer, the data is loaded into RAM.

It took 32 seconds to finally finish all data transfer.
That means that data is transferred at an average rate of 232 MByte/Sec.
In the past, when I did the same test using 1GbE, it was 112MB/sec from the beginning to the end of the data transfer.
The current performance is a little disappointing, but is it suitable for the hardware specifications?
Or is there a way for me to get even better throughput?

The network bandwidth test with my iperf3 is as follows.
The results are as follows.
[4] 0.00-10.00 sec 3.58 GBytes 3.07 Gbits/sec sender
[4] 0.00-10.00 sec 3.58 GBytes 3.07 Gbits/sec receiver

Adding -P 10 produces the following results
[SUM] 0.00-10.00 sec 6.36 GBytes 5.47 Gbits/sec sender
[SUM] 0.00-10.00 sec 6.36 GBytes 5.46 Gbits/sec receiver


The results of the dd test are as follows.
# dd if=test.dat of=/dev/null bs=2048k count=16k
8192+0 records in
8192+0 records out
17179869184 bytes transferred in 3.994079 secs (4301334230 bytes/sec)

Write my hardware configuration below
MoterBoard:SuperMicro X8DTN+ (3GBps SATAx6)
CPU: Xeon L5630 x 2
Pool: RAID-Z2 (ST3000NM0033 3TB x 6)
NIC: Intel X520-DA2
Memory: 64GB

 

[Spearfoot]

Hello.
I am using FreeNAS 11.3-U4.1.
Recently I upgraded my system's network to 10GbE,
The improvement in transfer rate was below my expectations.

I wrote data of 7.4GByte by using Windows 10 as an initiator with iSCSI.
Data is transferred at a speed of 1.5 GByte/sec for about 3 seconds from the start of copying,
After that, the transfer speed drops and goes back and forth between 10MB-200MB/sec.
I understand that at the time of the first fast transfer, the data is loaded into RAM.

It took 32 seconds to finally finish all data transfer.
That means that data is transferred at an average rate of 232 MByte/Sec.
In the past, when I did the same test using 1GbE, it was 112MB/sec from the beginning to the end of the data transfer.
The current performance is a little disappointing, but is it suitable for the hardware specifications?
Or is there a way for me to get even better throughput?

The network bandwidth test with my iperf3 is as follows.
The results are as follows.
[4] 0.00-10.00 sec 3.58 GBytes 3.07 Gbits/sec sender
[4] 0.00-10.00 sec 3.58 GBytes 3.07 Gbits/sec receiver

Adding -P 10 produces the following results
[SUM] 0.00-10.00 sec 6.36 GBytes 5.47 Gbits/sec sender
[SUM] 0.00-10.00 sec 6.36 GBytes 5.46 Gbits/sec receiver


The results of the dd test are as follows.
# dd if=test.dat of=/dev/null bs=2048k count=16k
8192+0 records in
8192+0 records out
17179869184 bytes transferred in 3.994079 secs (4301334230 bytes/sec)

Write my hardware configuration below
MoterBoard:SuperMicro X8DTN+ (3GBps SATAx6)
CPU: Xeon L5630 x 2
Pool: RAID-Z2 (ST3000NM0033 3TB x 6)
NIC: Intel X520-DA2
Memory: 64GB

I would say you are getting about as much performance from your system as you can reasonably expect.

Upgrading your network from 1Gb/s to 10Gb/s does not magically increase transfer rates by a factor of 10, it just reveals a different bottleneck in a system -- typically disk I/O. That seems to be your case.

You are getting ~5.46 Gb/s transfer rates with iperf3, nowhere near the line rate. This matches my experience with an older SuperMicro X8SIE-based system. Jumbo frames are frowned upon by many here, but I was able to get line rates by using them. You may want to try using jumbo frames, too.