Slow iSCSI Read Performance

[clifford64]

I would consider putting one of the cold-spares in place for the 1M-load-cycle Green.

Lol, unfortunately, all cold spares I have are a WD green or an 8tb that I don't want to put in without another 8tb in case it fails.

 

[HoneyBadger]

Lol, unfortunately, all cold spares I have are a WD green or an 8tb that I don't want to put in without another 8tb in case it fails.

Even a Green with a lower LCC (and ideally with the head parking disabled via wdidle3) would a valid test to see if it's a suspect drive dragging you down.

 

[clifford64]

Even a Green with a lower LCC (and ideally with the head parking disabled via wdidle3) would a valid test to see if it's a suspect drive dragging you down.

I have since re-added the SSD as an L2ARC and it seems to have sped things up a little bit. The latest backup of my plex vm ran last night and transferred 120Gb in about 50 minutes or so. I know I should be getting better performance than this, but at least the L2ARC seems to have helped a little.

 

[clifford64]

First new drive tested and installing today. I am replacing the one in mirror 0.

 

[clifford64]

I have replaced 4 WD Green drives so far. In order of highest parks. One drive in 4 different mirrors. Still no significant increases in performance yet.

 

[mrstevemosher]

We installed an EMC 480. Fixed everything. We just simply gave up.

 

[jgreco]

I have a TrueNAS system that acts as an iSCSI SAN for two ESXi hosts.
iSCSI network is 10Gb and segmented from data network. Have Jumbo Frames turned on. Not sure if it is working properly, I have a Mikrotik CRS305-1G-4S+IN and it is weird to configure jumbo frames.
TrueNAS Specs:
E3-1245 V2 (Hardly ever see it go above 5% usage, not sure if this is normal)
32GB ECC
2TBx12 drives. Drives are setup in mirrors and striped across 6 mirrors.
Truenas version 12.0 U2

ESXi Hosts: running 7.0, Ryzen 2700x, 32GB Ram.

I am getting very slow iSCSI read performance. Trying to perform a backup right now of a single VM that is 6TB, I am getting incredibly slow read performance and trying to do anything on any other vm is impossible.

This also shows when trying to do a storage vmotion of a VM from the iSCSI to a local SSD on either of my two ESXi hosts. Transferring the VM to a local datastore shows reads of only 30-80 MB/sec.

I have about 10 VMs in general. Very low usage in terms of compute. I do have a media server VM, so that does use quite a bit of computer here and there.

I have tried using a 500GB L2ARC drive, but haven't had any really noticeable differences in performance when using it.

When performing crystal disk mark tests on VMs stored on SAN, I get 10GB and saturate the the iSCSI network. Writing VMs to the SAN gets speeds of 300-400MB/sec. But trying to do anything with writes seems to be painfully slow and with the backup going, VMs are unusable. The Backup is only reading the VM at 30-70 MB/sec.

Any ideas on how to troubleshoot this?

4. The pool reports 99% used space on the dashboard. With "zpool list" it shows 10.9T size, 4.48T allocated, 6.39T Free.

Dell Broadcom 57810s all around.

Get rid of jumbo frames.

Quadruple the memory. Ditch the L2ARC until you at least double the memory. If you can get up to 128-256GB RAM, then add L2ARC which will likely help significantly. At 32GB with iSCSI, having L2ARC is sabotaging your ARC.

Get rid of the crappy Broadcom cards. The Intel X520's or Chelsio cards are known to work well.

I would be concerned about fragmentation rates impacting this, but there's so much other crap basically wrong here that it isn't actually the first place I'd look.

You probably have at least two things conspiring against you here, if I were to guess.