Error iSCSI handoff request with ESXi hosts

[cafedangbk]

Hi everybody,
I have a problem that occured twice times in this week with my FreeNAS. It worked OK for a long time, this was the first problem.
Normally, my FreeNAS loaded only about 1% CPU, but at the time when the problem happenned, system load 20% and after 30 min, it was hanged at 60% of CPU. The results was all my VMs (on ESXi hosts) could not access to theirs disk.
I stopped the problem by hard reset FreeNAS server, and it was OK after that.

I saw some logs that logged at the problem time.

Code:

Apr 11 20:30:42 bdkhp18g6sast01 ctld[81669]: 172.16.42.38: exiting due to timeout
Apr 11 20:30:42 bdkhp18g6sast01 ctld[2406]: child process 81669 terminated with exit status 1

Code:

Apr 11 20:34:17 bdkhp18g6sast01 ctld[2406]: child process 82499 terminated with exit status 1
Apr 11 20:34:22 bdkhp18g6sast01 ctld[82781]: 172.16.42.18 (iqn.1998-01.com.vmware:bdkib36m3clsv05-5041dc85): error returned from CTL iSCSI handoff request: cfiscsi_ioctl_handoff: icl_conn_handoff failed with error 54; dropping connection
Apr 11 20:34:22 bdkhp18g6sast01 ctld[2406]: child process 82781 terminated with exit status 1

(pleaser refer the attached log file for details)
https://drive.google.com/drive/folders/1yceasm8hBsJ8I6NhrurABI19UpPSLYcA?usp=sharing
P/S: at the twice times of problem, our ESXi hosts were not overloaded, all disks on FreeNAS loaded at normal value (about 40%)


May someone give me some ideal about the root cause.
Thank a lot.

 

[dlavigne]

Have you figured out the root cause? If not, please create a report at bugs.freenas.org that includes that debug and logs and post the issue number here.

 

[cafedangbk]

Thank your your respond. I asked in FreeNAS Community group on Facebook, one person checked my attached log and told me about there was a hard error on the disk da3. I think this might the root cause, because when I posted this, 2 days later, the problem was repeated again. When it was happening, I saw high read iSCSI traffic unnormally, after 20-30 min, the CPU's load increased, ping dropped and lost all iSCSI connections. I've replaced the error disk yesterday, and the hard error log was not shown again, system CPU is OK and now I'm still keep monitoring them. If it happens again, anything new, I'll update.