SCSI Status Error

[RickH]

I've been experimenting with FreeNAS on some older hardware for a while and I finally decided to implement FreeNAS in a production environment. I work for a small family owned service company that scans documents for our clients. The new storage server is going to be used for several roles:

1) Backup location for our main servers (all being backed up using ShadowProtect) over CIFS
2) Working storage for our scanning software - lots of simultaneous users reading and writing small image files (typically 30-60 kb in size) over CIFS
3) ESXi datastore over ISCSI​

Being a small business, money is always an issue, but I've put together the following server to try and meet our needs:

Dell FS12-SC (basically a C2100 that was sold in bulk to big data-centers)

12 hot swap 3.5" trays, dual redundant power supplies, 2 internal 2.5" trays,
and a Newisys SAS/SATA midplane​

2x Xeon L5410 @ 2.33 GHz
24GB ECC ram
2x onboard Intel PRO1000 NIC
Intel PRO1000 4-port PCI-e NIC
LSI SAS9210-8i
Generic 8GB SSD for boot
Intel 313 24GB SSD for Log​

Drives (because of cost concerns I had to re-use some drives, but all passed a full surface SMART test prior to being installed)

11x 2TB drives (Mixture of WD Black, WD Green, and Hitachi)​

I have everything assembled and have been doing some initial testing over the past few days. I have set up an 11 drive RAID Z3 volume (I have ordered a 12th drive to use as a hot spare) using the Intel 313 SSD as a log drive (I have sync=always enabled for the ISCSI Dataset). The NIC's are set up in a LCAP aggregation and overall performance has been excellent. I have been able to max out the Gigabit connections on all of my test clients and have seen excellent performance even with multiple clients simultaneously uploading and downloading files.

My only concern is a 'SCSI Status Error' that keeps showing up in my logs. (see attached files)
I see this error repeated continuously for several of my drives:

"SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)"​

This error seems to only occur on my WD drives (both the Green's and the Black's) and I have yet to see it on any of the 3 Hitachi's. I don't notice any performance issues and a full scrub reveals no data or checksum errors.

I have run and attached the following hoping someone out there has some suggestions.

camcontrol devlist
zpool status
smartctl -a /dev/da## (for each drive)​

You can see that this 'reset' seems to interrupt the SMART short tests I have scheduled, but I don't see any other obvious errors on any of the drives.

NOTE: Please disregard /dev/da11 - this drive isn't part of the pool, it was just a spare I stuck in the chasis to test the midplane and experiment with jails...

The only reference I can find to this error in the forums has been attributed to either a power supply issue or possible cabling issues....
I'm running dual power supplies so I'm fairly confident that isn't the issue. I did have to purchase a SFF-8484 to SFF-8087 cable to connect my SAS9210 to the midplane so there's a small chance that the issue is there (although the fact it's only showing on the WD drives makes me wonder). I have ordered a new cable just to make sure but I'm really hoping someone out there has some experience with this problem.

 

[cyberjock]

My first guess is your firmware version doesn't match the driver version in FreeNAS. Can you post the output of sas2flash -listall?

Edit: Unless you plan to run a single VM that is asleep 99% of the time, you ahve no chance of getting decent performance for *a* VM, let alone multiple VMs, with RAIDZ3. I tried to do that. Cut my wrists and bled out before the VM would work. ;)

You also have nowhere near enough RAM for VMs. Triple your RAM and add an L2ARC and you *might* be able to do VMs with it.

 

[RickH]

Here's the output:

[root@storage01] ~# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 14.00.00.00 (2012.07.04)
Copyright (c) 2008-2012 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2008(B2)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------

0 SAS2008(B2) 14.15.30.00 0e.03.00.05 07.27.00.00 00:06:00:00

Finished Processing Commands Successfully.
Exiting SAS2Flash.

Thanks for the comments on the VM performance. The datastore on the FreeNAS device is only being used as the store location for my VMware DataProtection disks. The DataProtection runs overnight when no other users are on the system, and the backups have been completing pretty quickly (at least when compared to my previous setup). To be clear I'm not currently or planning on running any live VM's from the datastore on the FreeNAS server.

 

[cyberjock]

If you look at that info you see your firmware is 14.15.30. It needs to be on 16.x. So you need to google around on how to update your controller to p16 firmware. Do not flash any other version besides p16.

 

[RickH]

I updated the firmware to P16 (see below) and the SCSI errors persist...


[root@storage01] ~# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 14.00.00.00 (2012.07.04)
Copyright (c) 2008-2012 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2008(B2)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------

0 SAS2008(B2) 16.00.00.00 10.00.00.04 07.31.00.00 00:06:00:00

Finished Processing Commands Successfully.
Exiting SAS2Flash.
[root@storage01] ~#

 

[cyberjock]

Well, I don't have any good ideas at this point. It's kind of back to "power or cables". In your case it could be the backplane itself is bad.

I've never heard of newisys, so I can't tell you if they are a good brand or not. My instinct is "run away very quickly" because I've never heard of them before.

Keep in mind that Dell/HP/etc have random problems and aren't always 100% compatible with FreeBSD/FreeNAS. So you may be stuck with a system that might decide to go bad at some point in the future. :( There's a bunch of reasons why we don't recommend Dell/HP/etc in our hardware recommendations threads. ;)

 

[rs225]

This other thread has a similar scenario, also along the lines of what cyberjock suggests (cables/backplane):
https://forums.freenas.org/index.ph...ves-cable-length-and-scsi-sense-issues.23652/

 

[jkh]

Hmmm. Are you mixing SATA and SAS drives in the same expander, perhaps, and/or doing SATA over SAS? That's also bad, since the converters that map one protocol (ATA) to another (SCSI) often do so with questionable fidelity and hilarity can ensue. A good rule of thumb is "SATA devices only with SATA controllers."


Sent from my iPad using Tapatalk

 

[petr]

I updated the firmware to P16 (see below) and the SCSI errors persist...


[root@storage01] ~# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 14.00.00.00 (2012.07.04)
Copyright (c) 2008-2012 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2008(B2)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------

0 SAS2008(B2) 16.00.00.00 10.00.00.04 07.31.00.00 00:06:00:00

Finished Processing Commands Successfully.
Exiting SAS2Flash.
[root@storage01] ~#

Hi, sorry to hijack the thread - but could you provide short summary what you did to flash to P16 and where did you get the firmware image?

 

[slushieken]

Hi, sorry to hijack the thread - but could you provide short summary what you did to flash to P16 and where did you get the firmware image?

Petr, I just did this myself. If you are still looking for help, I wanted to make a writeup to explain that detailed. Are you still interested?