I purchased a used SuperMicro CSE-836E16-R92JBD 3U LFF 16 BAY 3.5 JBOD BPN-SAS2-836EL1
I think the backplane is bad in my JBOD. I checked the cables, card, backplane jumpers. I just purchased another card, and JBOD, and I have a new spare cable. This is what happens:
------------
I am doing some testing/burn in. I boot into systemrescuecd 7.01, and I run nwipe. I have done this 3-4 times. After about 12 hours I get this:
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 CDB: Write(32)
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:40:52 2021] blk_update_request: critical target error, dev sdf, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354528, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354529, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354530, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354531, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354532, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354533, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354534, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354535, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354536, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354537, lost async page write
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 CDB: Write(32)
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:43:26 2021] blk_update_request: critical target error, dev sdc, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:43:26 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354528, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354529, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354530, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354531, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354532, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354533, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354534, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354535, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354536, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354537, lost async page write
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 CDB: Write(32)
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:48:42 2021] blk_update_request: critical target error, dev sdb, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:48:42 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354528, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354529, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354530, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354531, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354532, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354533, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354534, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354535, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354536, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354537, lost async page write
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 CDB: Write(32)
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:49:04 2021] blk_update_request: critical target error, dev sde, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:49:04 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354528, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354529, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354530, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354531, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354532, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354533, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354534, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354535, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354536, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354537, lost async page write
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 CDB: Write(32)
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:52:08 2021] blk_update_request: critical target error, dev sdd, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:52:08 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354528, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354529, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354530, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354531, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354532, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354533, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354534, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354535, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354536, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354537, lost async page write
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 CDB: Write(32)
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:53:23 2021] blk_update_request: critical target error, dev sda, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:53:23 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354528, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354529, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354530, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354531, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354532, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354533, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354534, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354535, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354536, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354537, lost async page write
I did some testing too after the failure:
[root@sysrescue ~]# dd if=/dev/sdb of=/dev/sda bs=64M status=progress oflag=direct
dd: error writing '/dev/sda': Remote I/O error
1+0 records in
0+0 records out
0 bytes copied, 0.473331 s, 0.0 kB/s
and
dd if=/dev/zero of=/dev/sda bs=64M status=progress oflag=direct
dd: error writing '/dev/sda': Remote I/O error
1+0 records in
0+0 records out
0 bytes copied, 0.447416 s, 0.0 kB/s
I just, while the system was running, switched my 6G External Mini SAS SFF-8088 to SFF-8088 to a different port on the back of the JBOD. It re-detected the drives, and I am going to leave it run to see if it errors out again. I wonder if I have a bad SAS port on the backplane, or SFF-8087 to SFF-8088 cable inside the unit.
[Tue Apr 20 14:09:20 2021] ses 5:0:13:0: Attached Enclosure device
[Tue Apr 20 14:09:20 2021] ses 5:0:13:0: Attached scsi generic sg6 type 13
[Tue Apr 20 14:09:20 2021] sd 5:0:7:0: [sdj] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:8:0: [sdk] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:11:0: [sdn] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:10:0: [sdm] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:9:0: [sdl] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:12:0: [sdo] Attached SCSI disk
[Tue Apr 20 14:09:46 2021] sdi: sdi1 sdi2 sdi3
[Tue Apr 20 14:09:46 2021] sdg: sdg1 sdg2
I am going to keep testing. This is my first time with any external SAS JBOD, and I wanted to talk to the FreeNAS/TrueNAS folks to see if you have an experience you can lend me. Everything is on a APC enterprise UPS that lasts like forever. It's all pretty enterprise. The thing that just struck me is that I was able to move to a different SAS port on the backplane, and the backplane is still working. I have had issues with bad backplanes, all the LEDs go red....etc....but to see this backplane still working.
This is my first time working with these drives, and this type of enclosure. These are IBM drives, ST4000NM0043. There is an error counter log at the end of the smartctl -a report of each drive. Looks like this:
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 3122255856 0 0 3122255856 0 739683.910 0
write: 0 0 0 0 0 705806.176 0
verify: 1675063088 0 0 1675063088 0 14960.613 0
Non-medium error count: 32
I don't have a clue what it means. I was going to log what it is at now for each drive at the beginning of this test, and see if it changes at the end. It just seems like some enterprise stats for internal hard drive workings, then the cable issue errors that you may see with lets say another drive. I don't remember what they look like.
Let me know what you think if you get a chance.
PS, I also checked the LEDs on the backplane, and did not see anything out of the ordinary.
I think the backplane is bad in my JBOD. I checked the cables, card, backplane jumpers. I just purchased another card, and JBOD, and I have a new spare cable. This is what happens:
------------
I am doing some testing/burn in. I boot into systemrescuecd 7.01, and I run nwipe. I have done this 3-4 times. After about 12 hours I get this:
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 CDB: Write(32)
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:40:52 2021] sd 5:0:5:0: [sdf] tag#436 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:40:52 2021] blk_update_request: critical target error, dev sdf, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354528, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354529, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354530, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354531, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354532, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354533, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354534, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354535, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354536, lost async page write
[Tue Apr 20 06:40:52 2021] Buffer I/O error on dev sdf, logical block 970354537, lost async page write
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 CDB: Write(32)
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:43:26 2021] sd 5:0:2:0: [sdc] tag#1605 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:43:26 2021] blk_update_request: critical target error, dev sdc, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:43:26 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354528, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354529, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354530, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354531, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354532, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354533, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354534, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354535, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354536, lost async page write
[Tue Apr 20 06:43:26 2021] Buffer I/O error on dev sdc, logical block 970354537, lost async page write
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 CDB: Write(32)
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:48:42 2021] sd 5:0:1:0: [sdb] tag#1023 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:48:42 2021] blk_update_request: critical target error, dev sdb, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:48:42 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354528, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354529, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354530, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354531, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354532, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354533, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354534, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354535, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354536, lost async page write
[Tue Apr 20 06:48:42 2021] Buffer I/O error on dev sdb, logical block 970354537, lost async page write
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 CDB: Write(32)
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:49:04 2021] sd 5:0:4:0: [sde] tag#1414 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:49:04 2021] blk_update_request: critical target error, dev sde, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:49:04 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354528, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354529, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354530, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354531, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354532, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354533, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354534, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354535, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354536, lost async page write
[Tue Apr 20 06:49:04 2021] Buffer I/O error on dev sde, logical block 970354537, lost async page write
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 CDB: Write(32)
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:52:08 2021] sd 5:0:3:0: [sdd] tag#157 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:52:08 2021] blk_update_request: critical target error, dev sdd, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:52:08 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354528, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354529, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354530, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354531, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354532, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354533, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354534, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354535, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354536, lost async page write
[Tue Apr 20 06:52:08 2021] Buffer I/O error on dev sdd, logical block 970354537, lost async page write
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 Sense Key : Illegal Request [current] [descriptor]
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 Add. Sense: Invalid field in cdb
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 CDB: Write(32)
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 CDB[00]: 7f 00 00 00 00 00 00 18 00 0b 20 00 00 00 00 01
[Tue Apr 20 06:53:23 2021] sd 5:0:0:0: [sda] tag#1619 CDB[10]: ce b3 7b 00 ce b3 7b 00 00 00 00 00 00 00 04 00
[Tue Apr 20 06:53:23 2021] blk_update_request: critical target error, dev sda, sector 7762836224 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 0
[Tue Apr 20 06:53:23 2021] buffer_io_error: 118 callbacks suppressed
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354528, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354529, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354530, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354531, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354532, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354533, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354534, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354535, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354536, lost async page write
[Tue Apr 20 06:53:23 2021] Buffer I/O error on dev sda, logical block 970354537, lost async page write
I did some testing too after the failure:
[root@sysrescue ~]# dd if=/dev/sdb of=/dev/sda bs=64M status=progress oflag=direct
dd: error writing '/dev/sda': Remote I/O error
1+0 records in
0+0 records out
0 bytes copied, 0.473331 s, 0.0 kB/s
and
dd if=/dev/zero of=/dev/sda bs=64M status=progress oflag=direct
dd: error writing '/dev/sda': Remote I/O error
1+0 records in
0+0 records out
0 bytes copied, 0.447416 s, 0.0 kB/s
I just, while the system was running, switched my 6G External Mini SAS SFF-8088 to SFF-8088 to a different port on the back of the JBOD. It re-detected the drives, and I am going to leave it run to see if it errors out again. I wonder if I have a bad SAS port on the backplane, or SFF-8087 to SFF-8088 cable inside the unit.
[Tue Apr 20 14:09:20 2021] ses 5:0:13:0: Attached Enclosure device
[Tue Apr 20 14:09:20 2021] ses 5:0:13:0: Attached scsi generic sg6 type 13
[Tue Apr 20 14:09:20 2021] sd 5:0:7:0: [sdj] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:8:0: [sdk] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:11:0: [sdn] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:10:0: [sdm] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:9:0: [sdl] Attached SCSI disk
[Tue Apr 20 14:09:20 2021] sd 5:0:12:0: [sdo] Attached SCSI disk
[Tue Apr 20 14:09:46 2021] sdi: sdi1 sdi2 sdi3
[Tue Apr 20 14:09:46 2021] sdg: sdg1 sdg2
I am going to keep testing. This is my first time with any external SAS JBOD, and I wanted to talk to the FreeNAS/TrueNAS folks to see if you have an experience you can lend me. Everything is on a APC enterprise UPS that lasts like forever. It's all pretty enterprise. The thing that just struck me is that I was able to move to a different SAS port on the backplane, and the backplane is still working. I have had issues with bad backplanes, all the LEDs go red....etc....but to see this backplane still working.
This is my first time working with these drives, and this type of enclosure. These are IBM drives, ST4000NM0043. There is an error counter log at the end of the smartctl -a report of each drive. Looks like this:
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 3122255856 0 0 3122255856 0 739683.910 0
write: 0 0 0 0 0 705806.176 0
verify: 1675063088 0 0 1675063088 0 14960.613 0
Non-medium error count: 32
I don't have a clue what it means. I was going to log what it is at now for each drive at the beginning of this test, and see if it changes at the end. It just seems like some enterprise stats for internal hard drive workings, then the cable issue errors that you may see with lets say another drive. I don't remember what they look like.
Let me know what you think if you get a chance.
PS, I also checked the LEDs on the backplane, and did not see anything out of the ordinary.
Last edited: