Hello,
I upgraded from 11.0-U4 to 11.1-RELEASE yesterday morning. I did not encounter any problems during the upgrade. I received emails about a pool being degraded, a disk being removed and unexpected shutdown. I didn't find anything in /data/crash and zpool status showed 8MB had been resilvered. I checked the smart status of all the drives and did not find anything with any of the drives. Later, I received more emails about a drive being removed and degraded pool for a different pool than last night. I checked everything again and found no issues, everything was showing healthy and nothing was resilvered. Same thing happened again about 30 minutes later to the same pool from last night and again by the time I was able to check, everything was fine and 17.8MB had been resilvered. I checked out /var/log/messages and found the following
I thought it was da8 but continuing to go through the logs, it includes basically every drive at some point. Sometimes no drives are included, like this from just after the above event.
I restarted and was still encountering this issue. It occurs anywhere from several times a minute to every 10 minutes or so. I rolled back to 11.0-U4 and I'm no longer seeing the events. This server has run without issue for about 18 months now. The hardware is a S2600IP4 with a single E5-2670, 32GB EEC, RES2SV240 expander and H200 flashed to P20 in IT.
Any ideas?
Thank you
I upgraded from 11.0-U4 to 11.1-RELEASE yesterday morning. I did not encounter any problems during the upgrade. I received emails about a pool being degraded, a disk being removed and unexpected shutdown. I didn't find anything in /data/crash and zpool status showed 8MB had been resilvered. I checked the smart status of all the drives and did not find anything with any of the drives. Later, I received more emails about a drive being removed and degraded pool for a different pool than last night. I checked everything again and found no issues, everything was showing healthy and nothing was resilvered. Same thing happened again about 30 minutes later to the same pool from last night and again by the time I was able to check, everything was fine and 17.8MB had been resilvered. I checked out /var/log/messages and found the following
Code:
Dec 28 03:36:11 freenas-1 mps0: IOC Fault 0x40000d04, Resetting Dec 28 03:36:11 freenas-1 mps0: Reinitializing controller, Dec 28 03:36:11 freenas-1 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Dec 28 03:36:11 freenas-1 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Dec 28 03:36:11 freenas-1 mps0: mps_reinit finished sc 0xfffffe0000ecd000 post 4 free 3 Dec 28 03:36:11 freenas-1 (da8:mps0:0:42:0): Invalidating pack Dec 28 03:36:11 freenas-1 da8 at mps0 bus 0 scbus0 target 42 lun 0 Dec 28 03:36:11 freenas-1 da8: <ATA WDC WD80EFAX-68L 0A83> s/n 7SGJEKMC detached Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 384935409bae9a74 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 484b29409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 464a404c9db98e5c Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 49463e409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 494f4a409bae9a74 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 454b304c9db98e5c Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = dd5143409bae9a74 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = d14f3d409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 3d481a4c9db98d7f Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = db5149409bae9a71 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 484f2a409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address for SATA device = 463d364c9db98e5c Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 384935409bae9a74 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 484b29409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 464a404c9db98e5c Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 49463e409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 494f4a409bae9a74 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 454b304c9db98e5c Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = dd5143409bae9a74 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = d14f3d409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 3d481a4c9db98d7f Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = db5149409bae9a71 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 484f2a409bae9a73 Dec 28 03:36:13 freenas-1 mps0: SAS Address from SATA device = 463d364c9db98e5c Dec 28 03:36:19 freenas-1 ZFS: vdev state changed, pool_guid=13824347541065318658 vdev_guid=17157989435552045004 Dec 28 03:36:19 freenas-1 (da8:mps0:0:42:0): Periph destroyed Dec 28 03:36:19 freenas-1 da8 at mps0 bus 0 scbus0 target 42 lun 0 Dec 28 03:36:19 freenas-1 da8: <ATA WDC WD80EFAX-68L 0A83> Fixed Direct Access SPC-4 SCSI device Dec 28 03:36:19 freenas-1 da8: Serial Number 7SGJEKMC Dec 28 03:36:19 freenas-1 da8: 600.000MB/s transfers Dec 28 03:36:19 freenas-1 da8: Command Queueing enabled Dec 28 03:36:19 freenas-1 da8: 7630885MB (15628053168 512 byte sectors) Dec 28 03:36:19 freenas-1 ZFS: vdev state changed, pool_guid=13824347541065318658 vdev_guid=17157989435552045004
I thought it was da8 but continuing to go through the logs, it includes basically every drive at some point. Sometimes no drives are included, like this from just after the above event.
Code:
Dec 28 03:37:19 freenas-1 mps0: IOC Fault 0x40000d04, Resetting Dec 28 03:37:19 freenas-1 mps0: Reinitializing controller, Dec 28 03:37:19 freenas-1 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd Dec 28 03:37:19 freenas-1 mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc> Dec 28 03:37:19 freenas-1 mps0: mps_reinit finished sc 0xfffffe0000ecd000 post 4 free 3 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 384935409bae9a74 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 484b29409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 464a404c9db98e5c Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 49463e409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 494f4a409bae9a74 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 454b304c9db98e5c Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = dd5143409bae9a74 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = d14f3d409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 3d481a4c9db98d7f Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = db5149409bae9a71 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 484f2a409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address for SATA device = 463d364c9db98e5c Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 384935409bae9a74 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 484b29409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 464a404c9db98e5c Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 49463e409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 494f4a409bae9a74 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 454b304c9db98e5c Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = dd5143409bae9a74 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = d14f3d409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 3d481a4c9db98d7f Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = db5149409bae9a71 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 484f2a409bae9a73 Dec 28 03:37:21 freenas-1 mps0: SAS Address from SATA device = 463d364c9db98e5c
I restarted and was still encountering this issue. It occurs anywhere from several times a minute to every 10 minutes or so. I rolled back to 11.0-U4 and I'm no longer seeing the events. This server has run without issue for about 18 months now. The hardware is a S2600IP4 with a single E5-2670, 32GB EEC, RES2SV240 expander and H200 flashed to P20 in IT.
Any ideas?
Thank you