LSI controller error

[maskeman]

All,
First post on this thread and my first FreeNAS install.

I am trying to re-purpose a 4 year old storage device (formally a Greenbytes GB2000) and I am having problems with my SATA/SAS controllers (4 total)

Here is the hardware spec's:

Super Micro X9SRL-F motherboard
Intel Xeon ES-2670 at 2.60GHz
65483MB Ram
FreeNAS 9.20.0 x64 Release Version
LSI SAS 9201-16i disk controller, 16 ports (4 total)(SAS2116. Flashed fw and bios with P18, latest)
Seagate ST9500530NS 500GB HD (72 drives total, only use 15 for initial testing, no fw updates available)
Six - 32GB SSD, not sure of make and they were not installed for initial testing, will be using them for cache.
Seagate 250gb HD for boot drive.

With no SAS controller cards installed, the FreeNAS install went very well. I attached a single Seagate HD and was able to configure it as a share. WebGUI worked great.

When I added one of the LSI controllers with 15 drives attached I was able to see the drives in the View Disk screen. I was also able to create a raid from those drives.

System console showed the following errors over and over and I would like to clear this up before I continue with installing the other controllers:

Feb 5 07:05:28 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 621 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:29 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 659 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:31 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 605 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:32 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 649 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:33 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 629 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:34 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 642 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:36 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 622 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:37 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 684 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:38 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 670 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:39 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 673 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:41 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 627 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:42 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 643 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:43 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 668 terminated ioc 804b scsi 0 state c xfer 0
Feb 5 07:05:44 lx099fn1 kernel: (probe0:mps0:0:15:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 657 terminated ioc 804b scsi 0 state c xfer 0
I tried searching for these errors but could not find anything specific for my install.
Any help would be appreciate. Thanks in advance for your help

 

[maskeman]

Fixed it, had a bad hard disk. Add all 72 drives and get no errors.

 

[cyberjock]

Few comments:

1. Glad you found the bad drive. I didn't see your post for some reason, of course I didn't read a whole boatload of posts from last week and earlier this week. I spent 2 days trying to catch up and just gave up. (this forum takes up too much of my time as a volunteer). That error didn't say what hard drive was broken(maybe the drive on port 15), but smart output from the drives would have definitely answered the question.

2. Not sure what your intentions are with the SSDs(not sure you even know yet), but please read up on l2arcs and zils before adding them. They do not work how people think, and if you aren't prepared, it can turn into a nightmare for you. I just spent an hour on another thread explaining to them that after 3 months they are fairly clueless. I hate having to tell someone "hey, you're wasting your time... " but it happens. This crap isn't easy. And more than 90% of people get l2arcs and zils wrong, spend weeks or months trying to get it right, then give up and ditch zfs when they can never safely get performance where they want it. So please, please, please, do your homework.

3. 64GB of RAM isn't very much for using a big L2ARC. If you could go to 96GB you'd probably see an explosion in performance if your other design specs are reasonable.

4. Your LSI card should be flashed to the firmware version that matches FreeNAS' driver version. Right now I think 9.2.0 we're on v14, but 9.2.1 is going to be v16(you can verify this by checking dmesg on a system after its booted up). For your own sanity and for your data's safety its strongly recommended you keep those 2 in sync at all times. Someone I know called LSI with a problem and LSI basically said they wouldn't help the guy until he flashed his card to match his driver. He was skiddish about flashing a $500 card for fear of breaking it. Turns out he lost his data because of this oversight. It wasn't a FreeNAS related problem, but data loss is still data loss. We've seen here that having a firmware version of v16 and v17 with driver v14 works fine, but that's not a license to use any firmware version with any driver. I originally flashed the wrong version and instead of taking the chance I reflashed my card with the right version. I'd strongly recommend you do the same.

 

[maskeman]

Cyperjock,
Thanks for the tips. I am currently reviewing your slide show on ZIL and L2ARC, great stuff, thanks for the hard work.
I am currently using this hardware as a trainer where I can build and test and rebuild util I get the optimal system. And along the way learn more and more about FreeNAS.

The unit is maxed out on ram slots (4x16gb) and no money to upgrade ram to 32gb, so I am stuck at 64gb for now. I was going to mirror two 32gb ssd's for the ZIL and use one 32gb ssd for the l2arc.

The hard drive failure was obvious after I noticed the activity lights on it going crazy. I going to put it on another system for more thorough test. I had a spare to use for now.

I am going to leave the LSI bios where it is for now while I am testing. If and when the unit go into production I will then flash it to the current version that matches the operating system I have installed. Where would I find the proper bios/driver versions for FreeNAS version?

Thanks again for your support