Mystik
Cadet
- Joined
- Aug 26, 2022
- Messages
- 3
Greeting!
After upgrading from 12.0-U8 to 13.0-U1.1, I instantly started having some problems with (I believe) my LSI SAS 9207-8i.
TrueNAS: 13.0-U1.1
CPU: i7-3770K
RAM: 16GB DDR4-1600
Controller: LSI SAS 9207-8i PCI-e 3.0 (IT Mode)
HDDs: 6 x WD Red 4TB (raidz2)
After update, my NAS raised an alert: * Device: /dev/da5 [SAT], not capable of SMART self-check.
Some time after this, it seems like the controller started failing somehow, as if I read the logs correctly all of the drives are being dropped one by one and then the controller is being reset:
After some time the pool is suspended by TrueNAS automatically: Aug 26 21:46:36 TrueNAS-304 Solaris: WARNING: Pool 'holvi' has encountered an uncorrectable I/O failure and has been suspended. (Limited to 3000 characters, so the entire log is readable here: https://f.mstk.eu/NAS_problem.txt)
After this both the Pools and Disks screens just keep showing the loading icon forever. After a restart the whole system works great for a few minutes, until this starts again.
I would revert back to 12.0-U8, but I unfortunately I already upgraded my pools... A big mistake on my part. Is there anything else I should still try? I do not have another controller on hand. Reinstall and restore configuration saved before upgrade maybe?
After upgrading from 12.0-U8 to 13.0-U1.1, I instantly started having some problems with (I believe) my LSI SAS 9207-8i.
TrueNAS: 13.0-U1.1
CPU: i7-3770K
RAM: 16GB DDR4-1600
Controller: LSI SAS 9207-8i PCI-e 3.0 (IT Mode)
HDDs: 6 x WD Red 4TB (raidz2)
After update, my NAS raised an alert: * Device: /dev/da5 [SAT], not capable of SMART self-check.
Some time after this, it seems like the controller started failing somehow, as if I read the logs correctly all of the drives are being dropped one by one and then the controller is being reset:
Code:
Aug 26 21:33:46 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:33:46 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:33:46 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:33:46 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:33:59 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:33:59 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:33:59 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:33:59 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:34:16 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:34:16 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:34:16 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:34:16 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:34:16 TrueNAS-304 (da5:mps0:0:5:0): Invalidating pack Aug 26 21:34:16 TrueNAS-304 da5 at mps0 bus 0 scbus0 target 5 lun 0 Aug 26 21:34:16 TrueNAS-304 da5: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K2KASH4F detached Aug 26 21:34:16 TrueNAS-304 GEOM_MIRROR: Device swap0: provider da5p1 disconnected. Aug 26 21:34:29 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:34:29 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:34:29 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:34:29 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:34:45 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:34:45 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:34:45 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:34:45 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:34:45 TrueNAS-304 (da2:mps0:0:2:0): Invalidating pack Aug 26 21:34:45 TrueNAS-304 da2 at mps0 bus 0 scbus0 target 2 lun 0 Aug 26 21:34:45 TrueNAS-304 da2: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K0SF95HR detached Aug 26 21:34:45 TrueNAS-304 GEOM_MIRROR: Device swap1: provider da2p1 disconnected. Aug 26 21:34:55 TrueNAS-304 (da5:mps0:0:5:0): Periph destroyed Aug 26 21:34:55 TrueNAS-304 (da2:mps0:0:2:0): Periph destroyed Aug 26 21:34:55 TrueNAS-304 da2 at mps0 bus 0 scbus0 target 2 lun 0 Aug 26 21:34:55 TrueNAS-304 da2: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device Aug 26 21:34:55 TrueNAS-304 da2: Serial Number WD-WCC7K0SF95HR Aug 26 21:34:55 TrueNAS-304 da2: 600.000MB/s transfers Aug 26 21:34:55 TrueNAS-304 da2: Command Queueing enabled Aug 26 21:34:55 TrueNAS-304 da2: 3815447MB (7814037168 512 byte sectors) Aug 26 21:34:55 TrueNAS-304 da2: quirks=0x8<4K> Aug 26 21:34:55 TrueNAS-304 da5 at mps0 bus 0 scbus0 target 5 lun 0 Aug 26 21:34:55 TrueNAS-304 da5: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device Aug 26 21:34:55 TrueNAS-304 da5: Serial Number WD-WCC7K2KASH4F Aug 26 21:34:55 TrueNAS-304 da5: 600.000MB/s transfers Aug 26 21:34:55 TrueNAS-304 da5: Command Queueing enabled Aug 26 21:34:55 TrueNAS-304 da5: 3815447MB (7814037168 512 byte sectors) Aug 26 21:34:55 TrueNAS-304 da5: quirks=0x8<4K> Aug 26 21:35:23 TrueNAS-304 GEOM_MIRROR: Device mirror/swap2 launched (3/3). Aug 26 21:35:24 TrueNAS-304 GEOM_ELI: Device mirror/swap2.eli created. Aug 26 21:35:24 TrueNAS-304 GEOM_ELI: Encryption: AES-XTS 128 Aug 26 21:35:24 TrueNAS-304 GEOM_ELI: Crypto: accelerated software Aug 26 21:38:13 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:38:13 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:38:13 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:38:13 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:38:30 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:38:30 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:38:30 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:38:30 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:38:54 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:38:54 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:38:56 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:38:56 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:38:56 TrueNAS-304 (da5:mps0:0:5:0): Invalidating pack Aug 26 21:38:56 TrueNAS-304 da5 at mps0 bus 0 scbus0 target 5 lun 0 Aug 26 21:38:56 TrueNAS-304 da5: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K2KASH4F detached Aug 26 21:38:56 TrueNAS-304 GEOM_MIRROR: Device swap2: provider da5p1 disconnected. Aug 26 21:39:08 TrueNAS-304 (da5:mps0:0:5:0): Periph destroyed Aug 26 21:39:08 TrueNAS-304 da5 at mps0 bus 0 scbus0 target 5 lun 0 Aug 26 21:39:08 TrueNAS-304 da5: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device Aug 26 21:39:08 TrueNAS-304 da5: Serial Number WD-WCC7K2KASH4F Aug 26 21:39:08 TrueNAS-304 da5: 600.000MB/s transfers Aug 26 21:39:08 TrueNAS-304 da5: Command Queueing enabled Aug 26 21:39:08 TrueNAS-304 da5: 3815447MB (7814037168 512 byte sectors) Aug 26 21:39:08 TrueNAS-304 da5: quirks=0x8<4K> Aug 26 21:39:47 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:39:47 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:39:47 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:39:47 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:40:03 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:40:03 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:40:03 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:40:03 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:40:43 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:40:43 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:40:43 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:40:43 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:40:43 TrueNAS-304 (da0:mps0:0:0:0): Invalidating pack Aug 26 21:40:43 TrueNAS-304 da0 at mps0 bus 0 scbus0 target 0 lun 0 Aug 26 21:40:43 TrueNAS-304 da0: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K7EZK9C0 detached Aug 26 21:40:43 TrueNAS-304 GEOM_MIRROR: Device swap1: provider da0p1 disconnected. Aug 26 21:40:51 TrueNAS-304 (da0:mps0:0:0:0): Periph destroyed Aug 26 21:40:51 TrueNAS-304 da0 at mps0 bus 0 scbus0 target 0 lun 0 Aug 26 21:40:51 TrueNAS-304 da0: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device Aug 26 21:40:51 TrueNAS-304 da0: Serial Number WD-WCC7K7EZK9C0 Aug 26 21:40:51 TrueNAS-304 da0: 600.000MB/s transfers Aug 26 21:40:51 TrueNAS-304 da0: Command Queueing enabled Aug 26 21:40:51 TrueNAS-304 da0: 3815447MB (7814037168 512 byte sectors) Aug 26 21:40:51 TrueNAS-304 da0: quirks=0x8<4K> Aug 26 21:40:58 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:40:58 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:40:58 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:40:58 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:40:58 TrueNAS-304 (da2:mps0:0:2:0): Invalidating pack Aug 26 21:40:58 TrueNAS-304 da2 at mps0 bus 0 scbus0 target 2 lun 0 Aug 26 21:40:58 TrueNAS-304 da2: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K0SF95HR detached Aug 26 21:40:58 TrueNAS-304 GEOM_MIRROR: Device swap2: provider da2p1 disconnected. Aug 26 21:41:11 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:41:11 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:41:11 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:41:11 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:41:19 TrueNAS-304 GEOM_ELI: Device mirror/swap2.eli destroyed. Aug 26 21:41:19 TrueNAS-304 GEOM_MIRROR: Device swap2: provider destroyed. Aug 26 21:41:19 TrueNAS-304 GEOM_MIRROR: Device swap2 destroyed. Aug 26 21:41:19 TrueNAS-304 GEOM_ELI: Device mirror/swap1.eli destroyed. Aug 26 21:41:19 TrueNAS-304 GEOM_MIRROR: Device swap1: provider destroyed. Aug 26 21:41:19 TrueNAS-304 GEOM_MIRROR: Device swap1 destroyed. Aug 26 21:41:20 TrueNAS-304 mps0: IOC Fault 0x40000d04, Resetting Aug 26 21:41:20 TrueNAS-304 mps0: Reinitializing controller Aug 26 21:41:21 TrueNAS-304 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Aug 26 21:41:21 TrueNAS-304 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Aug 26 21:41:21 TrueNAS-304 (da3:mps0:0:3:0): Invalidating pack Aug 26 21:41:21 TrueNAS-304 da3 at mps0 bus 0 scbus0 target 3 lun 0 Aug 26 21:41:21 TrueNAS-304 da3: <ATA WDC WD40EFRX-68N 0A82> s/n WD-WCC7K0SF93CX detached Aug 26 21:41:29 TrueNAS-304 GEOM_MIRROR: Request failed (error=6). da3p1[READ(offset=512, length=512)] Aug 26 21:41:29 TrueNAS-304 GEOM_MIRROR: Device swap0: provider da3p1 disconnected. Aug 26 21:41:30 TrueNAS-304 GEOM_ELI: Device mirror/swap0.eli destroyed. Aug 26 21:41:30 TrueNAS-304 GEOM_MIRROR: Device swap0: provider destroyed. Aug 26 21:41:30 TrueNAS-304 GEOM_MIRROR: Device swap0 destroyed. Aug 26 21:41:30 TrueNAS-304 (da2:mps0:0:2:0): Periph destroyed Aug 26 21:41:30 TrueNAS-304 (da3:mps0:0:3:0): Periph destroyed Aug 26 21:41:30 TrueNAS-304 da2 at mps0 bus 0 scbus0 target 2 lun 0 Aug 26 21:41:30 TrueNAS-304 da2: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device Aug 26 21:41:30 TrueNAS-304 da2: Serial Number WD-WCC7K0SF95HR Aug 26 21:41:30 TrueNAS-304 da2: 600.000MB/s transfers Aug 26 21:41:30 TrueNAS-304 da2: Command Queueing enabled Aug 26 21:41:30 TrueNAS-304 da2: 3815447MB (7814037168 512 byte sectors) Aug 26 21:41:30 TrueNAS-304 da2: quirks=0x8<4K> Aug 26 21:41:30 TrueNAS-304 da3 at mps0 bus 0 scbus0 target 3 lun 0 Aug 26 21:41:30 TrueNAS-304 da3: <ATA WDC WD40EFRX-68N 0A82> Fixed Direct Access SPC-4 SCSI device Aug 26 21:41:30 TrueNAS-304 da3: Serial Number WD-WCC7K0SF93CX Aug 26 21:41:30 TrueNAS-304 da3: 600.000MB/s transfers Aug 26 21:41:30 TrueNAS-304 da3: Command Queueing enabled Aug 26 21:41:30 TrueNAS-304 da3: 3815447MB (7814037168 512 byte sectors) Aug 26 21:41:30 TrueNAS-304 da3: quirks=0x8<4K> Aug 26 21:41:30 TrueNAS-304 GEOM_MIRROR: Device mirror/swap0 launched (3/3). Aug 26 21:41:30 TrueNAS-304 GEOM_ELI: Device mirror/swap0.eli created. Aug 26 21:41:30 TrueNAS-304 GEOM_ELI: Encryption: AES-XTS 128 Aug 26 21:41:30 TrueNAS-304 GEOM_ELI: Crypto: accelerated software Aug 26 21:41:31 TrueNAS-304 GEOM_MIRROR: Device mirror/swap1 launched (3/3). Aug 26 21:41:31 TrueNAS-304 GEOM_ELI: Device mirror/swap1.eli created. Aug 26 21:41:31 TrueNAS-304 GEOM_ELI: Encryption: AES-XTS 128 Aug 26 21:41:31 TrueNAS-304 GEOM_ELI: Crypto: accelerated software ...
After some time the pool is suspended by TrueNAS automatically: Aug 26 21:46:36 TrueNAS-304 Solaris: WARNING: Pool 'holvi' has encountered an uncorrectable I/O failure and has been suspended. (Limited to 3000 characters, so the entire log is readable here: https://f.mstk.eu/NAS_problem.txt)
After this both the Pools and Disks screens just keep showing the loading icon forever. After a restart the whole system works great for a few minutes, until this starts again.
I would revert back to 12.0-U8, but I unfortunately I already upgraded my pools... A big mistake on my part. Is there anything else I should still try? I do not have another controller on hand. Reinstall and restore configuration saved before upgrade maybe?