Kernel MPS error messages.

[zanginator]

Hi for the first time,

I am hoping someone can shed some light on an issue I am having. So I have used FreeNAS now for the past year, mainly as a test for ZFS and to see if it was the right thing for me. Well I am happy to say, yes it is.

So recently I decided to bite the bullet on building a bigger machine after I acquired an old Supermicro 24 disk chassis (company upgraded and they were getting rid).
The chassis is a Supermicro SC846E1-R900B (which is a single SA8087 port 24 disk expander).

Hooked up is an LSI 9240-8i cross flashed to a 9211-8i running in IT mode. (Phase 16)

Basically, I can read from any disk no problem. However writing to the disks either stops or is VERY slow (5MB/s). On larger attempted files (larger than 100MB) the system appears to panic and reboots.
As per this thread I tried swapping the SAS cables, however this results in small files only being written and the machine shutting itself down upon any write attempt.
This also happens regardless of any disk I use. So I no reason to believe it maybe because of failing disks.
Currently I was testing under version 9.2 as I still need the UFS to migrate some older data.

Full hardware Spec:

• Supermicro SC846E1-R900B 24 disk Chassis
• Supermicro X8DTN+
• 2x Intel Xeon L5520 (4c/8t) (8c/16t in total)
• 6x 4GB Samsung Registered ECC
• LSI 9240-8i cross-flashed to 9211-8i IT mode (phase 16)

Kernel Log of the Issue: (this repeats a number of times until it either writes the data or crashes)

Code:

Jul 14 14:49:36 freenas kernel: (da0:mps0:0:18:0): Retrying command (per sense data)
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a4 e7 c0 00 00 40 00 length 32768 SMID 335 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 a6 40 00 00 80 00 length 65536 SMID 334 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c8 40 00 01 00 00 length 131072 SMID 317 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 32 c0 00 00 20 00 length 16384 SMID 316 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 32 40 00 00 80 00 length 65536 SMID 315 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c7 40 00 01 00 00 length 131072 SMID 314 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c6 40 00 01 00 00 length 131072 SMID 313 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c5 40 00 01 00 00 length 131072 SMID 312 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c4 40 00 01 00 00 length 131072 SMID 311 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c3 40 00 01 00 00 length 131072 SMID 310 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c2 40 00 01 00 00 length 131072 SMID 309 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c1 40 00 01 00 00 length 131072 SMID 308 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 c0 40 00 01 00 00 length 131072 SMID 307 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 bf 40 00 01 00 00 length 131072 SMID 306 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 be 40 00 01 00 00 length 131072 SMID 305 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 bd 40 00 01 00 00 length 131072 SMID 304 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a7 bc 40 00 01 00 00 length 131072 SMID 303 terminated ioc 804b scsi 0 state c xfer 0
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): WRITE(10). CDB: 2a 00 0a a4 e7 c0 00 00 40 00
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): CAM status: SCSI Status Error
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): SCSI status: Check Condition
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jul 14 14:49:37 freenas kernel: (da0:mps0:0:18:0): Retrying command (per sense data)

And the output of the mps driver and card.

Code:

Jul 16 12:40:01 freenas kernel: mps0: <LSI SAS2008> port 0xd000-0xd0ff mem 0xfbc3c000-0xfbc3ffff,0xfbc40000-0xfbc7ffff irq 16 at device 0.0 on pci3
Jul 16 12:40:01 freenas kernel: mps0: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd
Jul 16 12:40:01 freenas kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

The only conclusion I can draw is that the SAS expander is doing something funny and is possibly failing!

If anyone has any insight or suggestions, I would be most grateful.
If you need any more info, just ask.

-zanginator

 

[Ericloewe]

version 9.2

Which 9.2? P16 was introduced somewhere after 9.2.1, IIRC.

 

[zanginator]

The exact version is 9.2.1.7 Release (embedded)

 

[Ericloewe]

Ok, P16 was introduced with 9.2.1 (just checked), so there goes the easy solution.

I'd proceed by trying to directly attach some drives without the expander and see what happens from there.

Other than that, you can see if there's new firmware for the expander. Those things also have upgradeable firmware, but there's much less information around. Supermicro should have the required tools on their website, though.

 

[zanginator]

Okay, so after sometime away from this. Borrowed a SAS-8087 to 4x SATA cable from work (before the weekend). Turned out it was a reverse cable, whoops. So ordered a Forward type.

Everything works when using it. There are no issues with going directly from the HBA to a drive. So the problem is clearly within the SAS expander.

There is no information (of which I have only quickly searched for) about updating the particular firmware for the chip of the SAS expander. I will drop an email to Supermicro to see if they have any unpublished information.

 

[zanginator]

So after a bit more searching and a message over to Supermicro, it sounds like a bust.

A thread on HardForum from 2012 states that the backplane does not have user upgrade-able firmware nor does it apparently function properly with SATA 3 drives (although I was testing with a mix of SATA 1, 2 and 3)

So shot a quick message over to Supermicro and pretty much had the same response as the guy over on HardForum, saying that it isn't user upgrade-able (requires a specialized adapter) and nor does it support SATA 3 except for the later produced models.

Oh well, guess I shall remove the backplane for now and hook the drives up as I require with the breakout cable. I shall have a look into making my own (non-expander based) backplane :S

 

[Ericloewe]

So after a bit more searching and a message over to Supermicro, it sounds like a bust.

A thread on HardForum from 2012 states that the backplane does not have user upgrade-able firmware nor does it apparently function properly with SATA 3 drives (although I was testing with a mix of SATA 1, 2 and 3)

So shot a quick message over to Supermicro and pretty much had the same response as the guy over on HardForum, saying that it isn't user upgrade-able (requires a specialized adapter) and nor does it support SATA 3 except for the later produced models.

Oh well, guess I shall remove the backplane for now and hook the drives up as I require with the breakout cable. I shall have a look into making my own (non-expander based) backplane :S

That's incredibly lame. Bare LSI SAS expanders are upgradeable in place, via the connected SAS controller.