Emile.Belcourt
Dabbler
- Joined
- Feb 16, 2021
- Messages
- 23
Hello all,
I've had this issue twice now in the past month and I think I'm going to pre-emptively cycle around some disks as quite a few of my disks are from the same build date, but back to the issue. I first get an alarm that a disk is causing degraded IO performance:
This error (and the one on the previous occasion) preceded an irrecoverable error five minutes later:
In this situation, all service activity continues fine and I prepare to replace the disk, crack out a new one and get it in its sled. I aim to go through the recommended procedure which is to OFFLINE the disk ready for replacement, eject the disk, put in new one, go to replace in the pool for the failed disk, it's replaced and resilvering starts no problem.
However, I press OFFLINE and it just hangs with the twirly gig with no response from the console (directly), via SSH or via the webdmin. I cannot get any response from anything relying on middlewared (which is practically everything) and no zpool status information.
I then notice I start getting Zabbix monitoring outage failures for my VMWare Cluster where VMs are no longer responding to heartbeat messages, I can initiate an RDP session but cannot complete the connection. Then I notice the ESXi disk monitoring showing *Zero* activity meaning the iSCSI link had stopped responding, it was active but nothing was going in or out. The latency briefly spike to ~80,000ms and then nothing:
The fluctuation at 07:00 is when the storage server stated it failed to read SMART values from the failing disk:
The first time this issue happened, I went to the storage server and all but one of the disk activity lights were solid and I could hear every disk running at full pelt but then the disk that had failed was dark. So I broke procedure and yanked the disk, all activity returned to normal, the storage server unlocked itself and all services returned to normal.
This time, I was not in the office and I just had to wait it out and after half an hour i got:
Attached are excerpts of the log data at the time of attempting to OFFLINE the faulting disk:
To summarise all of the above:
Why is OFFLINE-ing a failed/degraded disking causing the iSCSI service to stop responding and subsequently cause issues to our VMWare server? This time, it corrupted a VM and thankfully no changes had been made since the 4am backup so I was able to restore it without any service disruption but it's still highly concerning. It could be I'm asking the wrong question and OFFLINE-ing the disk was not the cause, only part of the symptomatic issues with the disk array going unhealthy, but it's too coincidental that on two occasions a disk degrades, faults, I OFFLINE it and then iSCSI (and the TrueNAS as well) goes unresponsive and causes issues for the virtual host.
Hopefully I've managed to provide enough information to help with analysis! :)
Many thanks,
Emile
I've had this issue twice now in the past month and I think I'm going to pre-emptively cycle around some disks as quite a few of my disks are from the same build date, but back to the issue. I first get an alarm that a disk is causing degraded IO performance:
This error (and the one on the previous occasion) preceded an irrecoverable error five minutes later:
In this situation, all service activity continues fine and I prepare to replace the disk, crack out a new one and get it in its sled. I aim to go through the recommended procedure which is to OFFLINE the disk ready for replacement, eject the disk, put in new one, go to replace in the pool for the failed disk, it's replaced and resilvering starts no problem.
However, I press OFFLINE and it just hangs with the twirly gig with no response from the console (directly), via SSH or via the webdmin. I cannot get any response from anything relying on middlewared (which is practically everything) and no zpool status information.
I then notice I start getting Zabbix monitoring outage failures for my VMWare Cluster where VMs are no longer responding to heartbeat messages, I can initiate an RDP session but cannot complete the connection. Then I notice the ESXi disk monitoring showing *Zero* activity meaning the iSCSI link had stopped responding, it was active but nothing was going in or out. The latency briefly spike to ~80,000ms and then nothing:
The fluctuation at 07:00 is when the storage server stated it failed to read SMART values from the failing disk:
The first time this issue happened, I went to the storage server and all but one of the disk activity lights were solid and I could hear every disk running at full pelt but then the disk that had failed was dark. So I broke procedure and yanked the disk, all activity returned to normal, the storage server unlocked itself and all services returned to normal.
This time, I was not in the office and I just had to wait it out and after half an hour i got:
Services had returned to normal but the UI still said the disk was ONLINE. At this point I left it till I was in the office so I could prepare the replacement disk. I mistakenly detached instead of replaced but either way I was able to get the vDev back into a mirror no problem and it started resilvering.
Attached are excerpts of the log data at the time of attempting to OFFLINE the faulting disk:
- daemon.log - This one starts with making a casefile very early on but after messages exhibits iSCSI errors
- messages - This log is the most damning and very early on the iSCSI service is detecting write errors:
- middlewared.log - doesn't look like anything special except for webmin/socket errors because everything was getting locked up
- X10DRU-i+
- Dual Xeon e5-2620v3
- 128GB DDR4-2166 ECC
- LSI 3008 HBA
- SAS 6gbps backplane (i know, it's a pita)
- 12x ES.3 4TB SAS HDDs in 6 vdev stripe of mirrored pairs
- 16GB Optane SLOG
- 2TB NVMe Cache
- TrueNAS Core 12.0-U8
To summarise all of the above:
Why is OFFLINE-ing a failed/degraded disking causing the iSCSI service to stop responding and subsequently cause issues to our VMWare server? This time, it corrupted a VM and thankfully no changes had been made since the 4am backup so I was able to restore it without any service disruption but it's still highly concerning. It could be I'm asking the wrong question and OFFLINE-ing the disk was not the cause, only part of the symptomatic issues with the disk array going unhealthy, but it's too coincidental that on two occasions a disk degrades, faults, I OFFLINE it and then iSCSI (and the TrueNAS as well) goes unresponsive and causes issues for the virtual host.
Hopefully I've managed to provide enough information to help with analysis! :)
Many thanks,
Emile
