jasonboche
Dabbler
- Joined
- Oct 2, 2018
- Messages
- 25
My newly assembled FreeNAS server gave up tonight and stopped providing storage to my VMware vSphere environment. The server remained powered on but didn't appear to have signs of life at the console and the HTTP UI page wasn't responding.
I powered the box off and then back on. Upon reboot, I was concerned to see hundreds of thousands of these message below fly by on the console. It doesn't appear to be any one single spinning disk.
After the FreeNAS box came back up, my virtual machines seem to be in tact and power on but I'm concerned about the stability as something doesn't seem right. Checking SAS cabling is on my to do list. I'm new to FreeNAS but have spent the better part of the last month reading and testing various build configurations. I thought everything was finally operationally ready but only got about 24 hours out of the box before it locked up tonight. If you have any comments or suggestions, I'm all ears. Thank you in advance.
These errors were also at the tail end of /var/log/messages after FreeNAS had been been successfully started with VMs running.
I powered the box off and then back on. Upon reboot, I was concerned to see hundreds of thousands of these message below fly by on the console. It doesn't appear to be any one single spinning disk.
After the FreeNAS box came back up, my virtual machines seem to be in tact and power on but I'm concerned about the stability as something doesn't seem right. Checking SAS cabling is on my to do list. I'm new to FreeNAS but have spent the better part of the last month reading and testing various build configurations. I thought everything was finally operationally ready but only got about 24 hours out of the box before it locked up tonight. If you have any comments or suggestions, I'm all ears. Thank you in advance.
Code:
Oct 28 20:54:31 freenas1 (da10:mps0:0:22:0): READ(10). CDB: 28 00 00 40 02 a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da10:mps0:0:22:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da10:mps0:0:22:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da10:mps0:0:22:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da10:mps0:0:22:0): Info: 0x400326 Oct 28 20:54:31 freenas1 (da10:mps0:0:22:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da8:mps0:0:20:0): READ(10). CDB: 28 00 00 40 00 a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da8:mps0:0:20:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da8:mps0:0:20:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da8:mps0:0:20:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da8:mps0:0:20:0): Info: 0x40012d Oct 28 20:54:31 freenas1 (da8:mps0:0:20:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da7:mps0:0:15:0): READ(6). CDB: 08 00 00 a0 e0 00 Oct 28 20:54:31 freenas1 (da7:mps0:0:15:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da7:mps0:0:15:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da7:mps0:0:15:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da7:mps0:0:15:0): Info: 0xd3 Oct 28 20:54:31 freenas1 (da7:mps0:0:15:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da5:mps0:0:13:0): READ(10). CDB: 28 00 00 3f fc a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da5:mps0:0:13:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da5:mps0:0:13:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da5:mps0:0:13:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da5:mps0:0:13:0): Info: 0x3ffd6c Oct 28 20:54:31 freenas1 (da5:mps0:0:13:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): READ(10). CDB: 28 00 00 3f fe a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): Info: 0x3fff6d Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): READ(10). CDB: 28 00 00 3f fe a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): Info: 0x3fff7f Oct 28 20:54:31 freenas1 (da3:mps0:0:11:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da2:mps0:0:10:0): READ(10). CDB: 28 00 00 3f fc a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da2:mps0:0:10:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da2:mps0:0:10:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da2:mps0:0:10:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da2:mps0:0:10:0): Info: 0x3ffd6d Oct 28 20:54:31 freenas1 (da2:mps0:0:10:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): READ(10). CDB: 28 00 00 3f fc a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): Info: 0x3ffd7f Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): Retrying command (per sense data) Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): READ(10). CDB: 28 00 00 3f fe a0 00 00 e0 00 Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): CAM status: SCSI Status Error Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): SCSI status: Check Condition Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received) Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): Info: 0x3fff67 Oct 28 20:54:31 freenas1 (da1:mps0:0:9:0): Retrying command (per sense data)
These errors were also at the tail end of /var/log/messages after FreeNAS had been been successfully started with VMs running.
Code:
Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a6 fd 48 00 00 08 00 length 4096 SMID 424 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a6 fd 48 00 00 08 00 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): Retrying command Oct 28 21:22:21 freenas1 (da14:mps0:0:27:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 717 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da14:mps0:0:27:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 Oct 28 21:22:21 freenas1 (da14:mps0:0:27:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da14:mps0:0:27:0): Retrying command Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a6 fe 48 00 00 98 00 length 77824 SMID 243 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a6 fe 48 00 00 98 00 Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): READ(10). CDB: 28 00 16 d5 dc 98 00 00 08 00 length 4096 SMID 1006 terminated ioc 804b loginfo 3112010c sc(da15:mps0:0:28:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): Retrying command Oct 28 21:22:21 freenas1 si 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): READ(10). CDB: 28 00 16 d5 dc 98 00 00 08 00 Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): Retrying command Oct 28 21:22:21 freenas1 (da4:mps0:0:12:0): READ(10). CDB: 28 00 17 31 45 28 00 00 18 00 length 12288 SMID 614 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da4:mps0:0:12:0): READ(10). CDB: 28 00 17 31 45 28 00 00 18 00 Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): READ(10). CDB: 28 00 16 d5 dc 98 00 00 08 00 length 4096 SMID 662 terminated ioc 804b loginfo 3112010c scs(da4:mps0:0:12:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da4:mps0:0:12:0): Retrying command Oct 28 21:22:21 freenas1 i 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): READ(10). CDB: 28 00 16 d5 dc 98 00 00 08 00 Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da5:mps0:0:13:0): Retrying command Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a6 ff 48 00 01 00 00 length 131072 SMID 845 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a6 ff 48 00 01 00 00 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a7 03 48 00 01 00 00 length 131072 SMID 372 terminated ioc 804b loginfo 3112010c(da15:mps0:0:28:0): Retrying command Oct 28 21:22:21 freenas1 scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a7 03 48 00 01 00 00 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): Retrying command Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a7 03 48 00 01 00 00 length 131072 SMID 452 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a7 03 48 00 01 00 00 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): Retrying command Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a7 03 48 00 01 00 00 length 131072 SMID 592 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): WRITE(10). CDB: 2a 00 00 a7 03 48 00 01 00 00 Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da15:mps0:0:28:0): Retrying command Oct 28 21:22:21 freenas1 (da9:mps0:0:21:0): READ(10). CDB: 28 00 2b a0 37 a8 00 00 08 00 length 4096 SMID 585 terminated ioc 804b loginfo 3112010c scsi 0 state c xfer 0 Oct 28 21:22:21 freenas1 (da9:mps0:0:21:0): READ(10). CDB: 28 00 2b a0 37 a8 00 00 08 00 Oct 28 21:22:21 freenas1 (da9:mps0:0:21:0): CAM status: CCB request completed with an error Oct 28 21:22:21 freenas1 (da9:mps0:0:21:0): Retrying command Oct 28 21:23:08 freenas1 (da4:mps0:0:12:0): WRITE(10). CDB: 2a 00 61 5e c5 c8 00 00 08 00 length 4096 SMID 251 Aborting command 0xfffffe0001a8f970 Oct 28 21:23:08 freenas1 mps0: Sending reset from mpssas_send_abort for target ID 12 Oct 28 21:23:08 freenas1 mps0: (da4:mps0:0:12:0): WRITE(10). CDB: 2a 00 61 5e c5 c8 00 00 08 00 Oct 28 21:23:08 freenas1 Unfreezing devq for target ID 12 Oct 28 21:23:08 freenas1 (da4:mps0:0:12:0): CAM status: Command timeout Oct 28 21:23:08 freenas1 (da4:mps0:0:12:0): Retrying command