James Gardiner
Dabbler
- Joined
- Jul 14, 2017
- Messages
- 19
Hi,
I would like to ask a question.
I have a large L2ARC, 2xNVMe 4TB SSD connected to a test 6 drive array, est. 20TB.
When on the local system, doing "dd" commands to test if the L2ARC is working, it does appear so with 2700MB/s reads from files on the share once in the L2ARC.
I then have a nfs share, v4.1 to a server. iperf3 reports 9.4 Gbs or there about, What is expected with a 10-Gbe network interfaces between the FreeNAS server and the debian based client.
If I do test reads from the FreeNAS server using NFS4.1 of files what fit into the main memory cache, I get somewhat the speed I expect, but when I do a read of a larger file say 50gig, even though on the FreeNAS server I can archive 2700MB/s On the client over NFS, it appears to be falling back to sequinous disk speeds the 6 drive array is capable of.
Is this expected? Is there any form of limitation of L2ARC and nfs share?
test/examples follow..
I would like to ask a question.
I have a large L2ARC, 2xNVMe 4TB SSD connected to a test 6 drive array, est. 20TB.
When on the local system, doing "dd" commands to test if the L2ARC is working, it does appear so with 2700MB/s reads from files on the share once in the L2ARC.
I then have a nfs share, v4.1 to a server. iperf3 reports 9.4 Gbs or there about, What is expected with a 10-Gbe network interfaces between the FreeNAS server and the debian based client.
If I do test reads from the FreeNAS server using NFS4.1 of files what fit into the main memory cache, I get somewhat the speed I expect, but when I do a read of a larger file say 50gig, even though on the FreeNAS server I can archive 2700MB/s On the client over NFS, it appears to be falling back to sequinous disk speeds the 6 drive array is capable of.
Is this expected? Is there any form of limitation of L2ARC and nfs share?
test/examples follow..
Code:
-- Server test dd [root@freenastest /mnt/testPool3/nfs_share/LoveSarah_FTR-1]# dd if=LoveSarah_FTR-1_F_EN-XX_AU_51_2K_RIAL_20200306_SIL_IOP_OV_02.mxf of=/dev/null bs=4M status=progress 24335351808 bytes (24 GB, 23 GiB) transferred 9.002s, 2703 MB/s 6377+1 records in 6377+1 records out 26748543781 bytes transferred in 9.974685 secs (2681643078 bytes/sec) --- Client dd if=LoveSarah_FTR-1_F_EN-XX_AU_51_2K_RIAL_20200306_SIL_IOP_OV_02.mxf of=/dev/null bs=4M status=progress 26315063296 bytes (26 GB, 25 GiB) copied, 91 s, 289 MB/s 6377+1 records in 6377+1 records out 26748543781 bytes (27 GB, 25 GiB) copied, 91.9628 s, 291 MB/s --- Client test dd of smaller file.. root@proxmox1:/mnt/LoveSarah_FTR-1# dd if=LoveSarah_FTR-1_F_EN-XX_AU_51_2K_RIAL_20200306_SIL_IOP_OV_07_audio.mxf of=/dev/null bs=4M status=progress 271+1 records in 271+1 records out 1140589994 bytes (1.1 GB, 1.1 GiB) copied, 0.350412 s, 3.3 GB/s --- root@proxmox1:/mnt/LoveSarah_FTR-1# iperf3 -R -c 10.11.2.222 Connecting to host 10.11.2.222, port 5201 Reverse mode, remote host 10.11.2.222 is sending [ 5] local 10.11.2.211 port 54548 connected to 10.11.2.222 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.04 GBytes 8.89 Gbits/sec [ 5] 1.00-2.00 sec 1.09 GBytes 9.35 Gbits/sec [ 5] 2.00-3.00 sec 1.09 GBytes 9.32 Gbits/sec [ 5] 3.00-4.00 sec 1.09 GBytes 9.36 Gbits/sec [ 5] 4.00-5.00 sec 1.09 GBytes 9.34 Gbits/sec [ 5] 5.00-6.00 sec 1.09 GBytes 9.34 Gbits/sec [ 5] 6.00-7.00 sec 1.09 GBytes 9.33 Gbits/sec [ 5] 7.00-8.00 sec 1.09 GBytes 9.34 Gbits/sec [ 5] 8.00-9.00 sec 1.02 GBytes 8.76 Gbits/sec [ 5] 9.00-10.00 sec 1003 MBytes 8.41 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.6 GBytes 9.14 Gbits/sec receiver iperf Done.
