Poor read speeds on multiple different systems

[kurtc]

I have been having poor random and especially poor sequential read performance on 7 different FreeNAS systems. I will pick three to give specs/configs on to hopefully figure out what I am doing wrong! On average for sustained sequential reads of single large files 40-80MBps is what I see. If there is zero activity on the array I might see peaks to 90MBps. Writes aren't great either (compared to traditional RAID configurations I was running on these systems) but at least those will hover from 150-190MBps sustained sequential.

All of these systems are are running pools with multiple 64K block size zvols with sync=standard and lz4 compression. These are presented through iSCSI zvol extents to Windows hosts over dual 10Gb NICs. The physical SAS disks are enterprise grade in most cases capable of 200-220MBps sustained throughput.

System-1
11.2-U6
Dual Intel E5-2670 2.6GHz (32 total cores)
224GB ECC RAM
Broadcom NIC for management
Chelsio T520 10Gb NICs for iSCSI
Avago SAS 9300-8i HBA
POOL RAIDZ2 (ashift 12) of Hitachi Ultrastar Helioseal 10TB SAS drives x 8
(tried with and without Intel 900P optane SLOG and L2ARC)

System-2
11.1-U4
Single Intel E5-2630 2.6GHz
192GB ECC RAM
Broadcom NIC for management
Intel X710 10Gb NICs for iSCSI
Avago SAS 9207-8i HBA
POOL 2 x RAIDZ2 (ashift 12) of Seagate Enterprise ST10000NM0206 10TB SAS drives x 9 each (18 total)
(tried with and without NVMe SLOG and L2ARC)

System-3
11.2-U6
Dual Intel E5-2670 2.6GHz (32 total cores)
128GB ECC RAM
Broadcom NIC for management
Intel X540 10Gb NICs for iSCSI
Avago SAS 9300-8i HBA
POOL 3 x MIRROR of Seagate EXOS X12 12TB SAS drives (6 total)
P4800X NVMe SLOG


Any help would be greatly appreciated, thanks!

 

[MikeyG]

How are you getting these speed results? Have you tested directly on the pool outside a zvol?

If you are using RAIDZ2 this is guaranteed to limit your speeds. If you've got even a couple VMs active, this might limit your performance pretty severely with that vdev type.

 

[kurtc]

How are you getting these speed results? Have you tested directly on the pool outside a zvol?

If you are using RAIDZ2 this is guaranteed to limit your speeds. If you've got even a couple VMs active, this might limit your performance pretty severely with that vdev type.

The throughput is viewing in resource monitor in Windows, and with pure file copies in/out. The only other testing I have done is just an SMB share, which was even worse. I understand the RAIDZ2 limitations, but on those particular systems it is backup storage, which is why I am concerned with sequential throughput. The tests where I was seeing 80-90MBps was with zero activity on the pools. I was also hoping that I would see incremental improvement with a dual Z2 in a single pool and especially with the 3xMIRROR, but I can't even reach single hard drive performance.

 

[Apollo]

Most likely an issue with your Windows system.
Can you perform replication from one system to another?

 

[kurtc]

Most likely an issue with your Windows system.
Can you perform replication from one system to another?

I can definitely try the replication, but we are talking 20-30 different systems talking to these 7 arrays in different datacenters, different OS’s, some running VM’s, others just storing backup data.

 

[kurtc]

So in replication tasks I get the same 40-80MBps rates. 60 seems to be the common denominator in a lot of this. I just had one of the RAIDZ2 arrays need a resilver after a drive was replaced and 58MBps was the average over the 2 days.

 

[kurtc]

Is there a paid option for support? I looked through and it seemed like it was limited to if you are using certified hardware?

Thanks everyone!

 

[Apollo]

Have you tried Netdata ? It might shade some light seeing how the metrics are shown?

 

[MikeyG]

Have you done any testing on the pool itself, not from a client across the network?

 

[kurtc]

Code:

dd if=/dev/zero of=/mnt/POOL01/tmp.000 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 45.309057 secs (2369817197 bytes/sec)

This is from System-2

I am assuming I need to do some tweaking to this test to rule out compression or caching from getting this high of a number. (EDITED WITH RESULTS BELOW FROM A DATASET WITH NO COMPRESSION)

Code:

dd if=/dev/zero of=/mnt/POOL01/TESTING01/tmp.000 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 130.830078 secs (820714810 bytes/sec)

 

[MikeyG]

800MBps seems decent enough. What about iperf results?

 

[kurtc]

800MBps seems decent enough. What about iperf results?

Getting 5.22 to 6.00 Gbits/sec per 10Gb path from the FNAS box to the Windows host. Less than I would want but definitely not my bottleneck. Will have to investigate that if I ever can get decent iSCSI speeds! I know it isn't the switch limiting the throughput because each path has a 40Gb Arista switch where iSCSI traffic to non-FreeNAS targets runs at ~80% line speed.

(EDIT AFTER ANOTHER SET OF TESTS)

I was able to get an aggregate 9.91Gbits/sec if I reach it with 4 simultaneous streams for 30 seconds (-t 30 -P 4)

 

[kurtc]

Bump! Really struggling here to get usable performance out of FreeNAS. Considering the performance I can get out of taking the same hardware and running Windows or Linux arrays with iSCSI target servers, I have to be doing something wrong.

Thanks!

 

[kurtc]

Bump, am I in the wrong forum?

Thanks everyone

 

[Elliot Dierksen]

I don't know if you are using iperf or iperf3. I found that iperf3 gave me some really bad results on my system. It was never able to get over 25Gb between the two 40Gb hosts. iperf (aka iperf2) is able to get 37.7Gb consistently with 8 streams (my physical CPU count) and 4M window. The first thing is to use iperf for synthetic traffic to prove that the network is working correctly. Then you can move on to things like potentially tuning, or hardware adjustments. Prior to doing any tuning, I would strongly suggest the you read the fine 10G network primer https://www.ixsystems.com/community/resources/10-gig-networking-primer.42/ by @jgreco .

 

[kurtc]

I don't know if you are using iperf or iperf3. I found that iperf3 gave me some really bad results on my system. It was never able to get over 25Gb between the two 40Gb hosts. iperf (aka iperf2) is able to get 37.7Gb consistently with 8 streams (my physical CPU count) and 4M window. The first thing is to use iperf for synthetic traffic to prove that the network is working correctly. Then you can move on to things like potentially tuning, or hardware adjustments. Prior to doing any tuning, I would strongly suggest the you read the fine 10G network primer https://www.ixsystems.com/community/resources/10-gig-networking-primer.42/ by @jgreco .

I went through that primer and am good to go on all fronts mentioned. Even one of the systems I am using for this troubleshooting has Chelsio and all are OM4 fiber. My iperf to various systems is between 5-7Gbps per path which is WAY above any iSCSI read or write throughput I am getting. I can't even exceed 1Gbps in actual reads.

Thanks!

 

[kurtc]

I am still having this issue on multiple systems. Today backups being written to the iSCSI block devices are erroring with timeouts and when it is responding to read/writes is peaking at 60MBps. When I look at the reporting the disks in the pool during a backup show the following...

Key: Mean/Max
Busy: 4.41/69.42
R Latency: 1.68/35.2
W Latency: 0.36/3.1
Pending I/O: 0.08/3.89
R Disk I/O: 106KB/9.6MB
W Disk I/O: 2.2MB/37.4MB

The disks don't seem to be running out of gas.

The system referenced above has the following specs...

12.0-U6
Dual Intel E5-2690 2.9GHz (16 total cores)
128GB ECC RAM
Broadcom NIC for management
Intel 10Gb NICs for iSCSI
LSI SAS 9207-8i HBA
LSI SAS 9207-8e HBA
POOL: RAIDZ2 of 6 x Seagate 4TB Exos 7E8 + RAIDZ2 of 6 x Seagate 4TB Exos 7E8

Tried with and without Intel 900P optane SLOG and L2ARC, the L2ARC made the performance worse. Pools are kept below 60-65% full.
Block storage is used due to REFS being needed for backup storage size reduction.

Thanks for any help!

 

[NugentS]

Well your kit list seems good
L2ARC won't help you here I think.
You are using Block iSCSI - mounting the targets from an initiator on another machine?
My guess would have been sync writes - what happens if you do sync=disabled? I am not suggesting this as a solution - purely as a diagnostic step

 

[kurtc]

Well your kit list seems good
L2ARC won't help you here I think.
You are using Block iSCSI - mounting the targets from an initiator on another machine?
My guess would have been sync writes - what happens if you do sync=disabled? I am not suggesting this as a solution - purely as a diagnostic step

I did try (and for now have left) sync=disabled. The write performance went up a bit to roughly where it is with the Optane SLOG, but still not great and the read is still miserable.

Yes, I am mounting the targets on other servers. I have tried various MPIO policies as well and they all perform roughly the same.

Thank you for popping in the thread, I appreciate it!

 

[jgreco]

You're getting 60MBytes/sec on iSCSI on RAIDZ2 with two vdevs? That seems like it could be very reasonable, especially once fragmentation takes hold.

https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

I get about 400-500 megabits per second, it's a 65% full pool (which is where some of the speed loss comes in) with a ZFS-reported 18% fragmentation, with three mirror vdevs. So it's a little different, but given the small amount of available I/O, I think what you're seeing on your hardware is not necessarily bad.