TrueNAS 12 RC1 - iSCSI shares very slow

[kspare]

try this:


hw.cxgbe.fw_install="2" in loader.conf to allow the driver to downgrade the firmware for the chelsios card and downgrade back to the latest v11.

Then change it to "0" to not let it update the firmware and upgrade to 12rc again and see if that helps?? I know v12 does update the firmware.

 

[MikeyG]

@HoneyBadger I was combining LAGG and MPIO together. It has worked perfectly for a couple of years and let me fall back to 1gb if the 10gb NIC went down. Just in case it was a LAGG issue though, I removed the LAGGs and assigned IPs directly. Also changed round robin in vmWare to fixed path. No help. I am now troubleshooting without using LAGG at all.

 

[HoneyBadger]

@HoneyBadger I was combining LAGG and MPIO together. It has worked perfectly for a couple of years and let me fall back to 1gb if the 10gb NIC went down. Just in case it was a LAGG issue though, I removed the LAGGs and assigned IPs directly. Also changed round robin in vmWare to fixed path. No help. I am now troubleshooting without using LAGG at all.

LAGG with two different interface speeds could have separate issues but this seems more like a driver issue on the Chelsio cards. If you want iSCSI failover rather than MPIO I would use vSwitch failover orders instead.

Running my upgrade now although I don't have the same card in the target machine. Let's see what happens on good old fashioned Intel 1GbE.

 

[MikeyG]

@kspare Thanks for the tip. I submitted a bug report and going to give IX a minute to comment on it before I mess with it much more (plus day job to take care of). Just to be clear though on your suggested steps:

1. Add tunable as suggested with value of 2 (and reboot?)
2. Reboot back into 11.3
3. Re-add/change tunable value to 0.
4. Re-run 12-RC1 upgrade again

Sorry if I'm getting that wrong. Just not clear on how tunables survive changing boot environments.

 

[kspare]

@kspare Thanks for the tip. I submitted a bug report and going to give IX a minute to comment on it before I mess with it much more (plus day job to take care of). Just to be clear though on your suggested steps:

1. Add tunable as suggested with value of 2 (and reboot?)
2. Reboot back into 11.3
3. Re-add/change tunable value to 0.
4. Re-run 12-RC1 upgrade again

Sorry if I'm getting that wrong. Just not clear on how tunables survive changing boot environments.

When you boot into 11.3 you will need that tune added, and then 11.3 will change the firmware. Then change it to 0, and upgrade 11.3 to 12rc

this should stop the firmware from updating.

 

[vangoose]

Never run mpio on top of lagg. If you need to run both nfs/smb and iscsi, you need at least 4 interfaces, use lagg for nfs/smb and straight interface for mpio. However, when you have pNFS and smb multi-channel, lagg is no longer required, maybe 1g lagg for management.

 

[MikeyG]

I rebooted back into 11.3, noticed firmware was at 1.24.12.0. Added tunable and restarted again, checked that firmware was now at 1.23.0.0. Tested speeds and all looked good.

Upgraded to 12, and firmware remained at 1.23.0.0. Speed issues still remain. For SMB shares I'm back to the LAGG, but if I remove the Chelsio card from the LAGG, speeds increase from 15MBps to 100MBps.

Seems to still be an issue with the Chelsio, but maybe not with the firmware.

 

[kspare]

I rebooted back into 11.3, noticed firmware was at 1.24.12.0. Added tunable and restarted again, checked that firmware was now at 1.23.0.0. Tested speeds and all looked good.

Upgraded to 12, and firmware remained at 1.23.0.0. Speed issues still remain. For SMB shares I'm back to the LAGG, but if I remove the Chelsio card from the LAGG, speeds increase from 15MBps to 100MBps.

Seems to still be an issue with the Chelsio, but maybe not with the firmware.

Interesting...I have all the boto environments, I wonder if the driver has been changing?

 

[HoneyBadger]

Quick-and-dirty test for before and after.

2x1GbE iSCSI setup using em driver, slight regression in SEQ1M, slight increase in RND4K high queue, but for the most the stats are "margin of error"

Show : Before

Show : After

 

[jenksdrummer]

I was using a lagg with multipath iscsi. Tried configuring as fixed path, and recreated the interface to not use a lagg, still didn't help. Will follow up with a bug report.

That's not following best practices. LAGG and iSCSI don't play well.
Multipath yes ;)

 

[MikeyG]

@jenksdrummer Even for just a failover LAGG setup in TrueNAS (not a managed switch)? In any case, I've removed it for troubleshooting and it doesn't seem to be the issue.

 

[jenksdrummer]

Never run mpio on top of lagg. If you need to run both nfs/smb and iscsi, you need at least 4 interfaces, use lagg for nfs/smb and straight interface for mpio. However, when you have pNFS and smb multi-channel, lagg is no longer required, maybe 1g lagg for management.

I think SMB multichannel is still not supported per SAMBA, but, everything I've tossed around with it seems to work just fine with Windows 10 clients.

I can drop a switch and they keep rollin.

 

[hescominsoon]

That's not following best practices. LAGG and iSCSI don't play well.
Multipath yes ;)

I have ISCSI going over a LAGG but it's not multipath...therein lies the gotcha..:)