TrueNAS 12 RC1 - iSCSI shares very slow

[MikeyG]

I just upgraded from 11.3-U4.1 to 12 RC1. I'm having some issues with iSCSI performance. Usually I can write a large files to my zvol at about 600MBps (the sustained write speed of my SLOG device) but now seems to be at about 14MBps. Reads are ok though at 1GBps.

ZVOL is mounted as an iSCSI disk in Windows 10. Test is done using file copy in explorer. Sync disabled. I have multiple SSD backed pools on different controllers. Same exact result for zvols on each.

SMB Copies to a dataset on the Pool go at 1GBps both directions.

Anyone ideas?

 

[kspare]

I was testing out iscsi and ran into major problems with VMware as well. NFS however is working flawlessly.

 

[MikeyG]

@kspare Could you elaborate a bit more on the problems you had? I'm testing in VMware and Windows 10, but only started setting up zvols and mounting them in Windows because of poor VMware performance.

 

[kspare]

I didn’t troubleshoot it too much... I migrated off and went back to nfs. I tried 3 freenas boxes with iscsi and it got better with a single box vs 3....I dunno....we like nfs works well for us so just sticking with that, I don’t want to waste time on figuring out iscsi

 

[MikeyG]

I can confirm that NFS works much better. Something definitely seems up with iSCSI on 12.

 

[MikeyG]

I take it back. NFS isn't any better. What's strange is that moving a VM to the NFS share was pretty fast, about 400MBps, but then any activity inside the VM is limited to about 25MBps.

 

[Kris Moore]

Are you guys by any chance using a lagg in your network configuration?

 

[Kris Moore]

@mgittelman Talking to one of the iX guys, the said your SMB poor performance was nailed down to some lagg configuration. Can you confirm if this was the case here with iSCSI/NFS as well?

 

[kspare]

Are you guys by any chance using a lagg in your network configuration?

I'm running lagg in my systems yes.

 

[Kris Moore]

Can you please open a bug report at jira.ixsystems.com and include your debug? We'll need to see how your lagg is configured, try to replicate it in-house and determine how to best fix.

 

[kspare]

Can you please open a bug report at jira.ixsystems.com and include your debug? We'll need to see how your lagg is configured, try to replicate it in-house and determine how to best fix.

I switched back to nfs and it's working fine. or do you suspect there is an issue with lagg?

 

[hescominsoon]

I just upgraded from 11.3-U4.1 to 12 RC1. I'm having some issues with iSCSI performance. Usually I can write a large files to my zvol at about 600MBps (the sustained write speed of my SLOG device) but now seems to be at about 14MBps. Reads are ok though at 1GBps.

ZVOL is mounted as an iSCSI disk in Windows 10. Test is done using file copy in explorer. Sync disabled. I have multiple SSD backed pools on different controllers. Same exact result for zvols on each.

SMB Copies to a dataset on the Pool go at 1GBps both directions.

Anyone ideas?

if synch is off the slog isn't doing anything. slog is for synchronous writes..not asynch. I am curious have you tried turning synch writes back on as well?

 

[kspare]

if synch is off the slog isn't doing anything. slog is for synchronous writes..not asynch. I am curious have you tried turning synch writes back on as well?

slog isn't the issue...i'm running rms200 slogs and it was the same poor performance with it on or off...sounds like it's an lagg issue.

 

[Kris Moore]

Either way, please open the bug report with a debug attached. That should provide us enough detail to get to the bottom of what's going on.

 

[hescominsoon]

slog isn't the issue...i'm running rms200 slogs and it was the same poor performance with it on or off...sounds like it's an lagg issue.

roger that..i'm watching this one..

 

[MikeyG]

I was using a lagg with multipath iscsi. Tried configuring as fixed path, and recreated the interface to not use a lagg, still didn't help. Will follow up with a bug report.

 

[kspare]

I was using a lagg with multipath iscsi. Tried configuring as fixed path, and recreated the interface to not use a lagg, still didn't help. Will follow up with a bug report.

So even with no multi path and no lagg it's still bad?

 

[MikeyG]

Yes. I re-assigned IPs to my 1gb intel NICS, instead of the 10gb Chelsio (I think T520) and that actually improved performance a lot.

 

[hescominsoon]

Yes. I re-assigned IPs to my 1gb intel NICS, instead of the 10gb Chelsio (I think T520) and that actually improved performance a lot.

sounds like a driver bug as well...

 

[HoneyBadger]

Were you putting LAGG and MPIO together, or were they just existing on the same system with no overlap?

LAGG and MPIO don't mix, but you can do two interfaces in LAGG for SMB/NFS and two with MPIO for iSCSI.

I'm going to see about getting one of my iSCSI servers to switch trains to 12.0 RC1, I've got a good "before" baseline so I'll know right away if something's awry.