LSI 12GB HBA

[kspare]

Just a heads up to anyone thinking about running a 12gb controller, we were running the LSI Controller have found that it keeps dropping out drives that aren't actually bad, so we're downgrading to a 6gb controller.

I doubt it will affect performance anyway but I just wanted to let people know that may be building or something else.

 

[cyberjock]

We already know that. The 12Gb/sec drivers are very alpha. Even the commit to github that added them said something like "let's call these drivers early alpha".

Just responded to someone else just before you that did had the same problem. :P

 

[kspare]

I know, I just thought for some people it may hit home if they hear another user actually try it lol. oh wel I will put the card on the shelf it will be usefull again soon.

 

[mjws00]

Firsthand knowledge is always appreciated.

 

[depasseg]

What firmware version were you using?

 

[fullspeed]

I know, I just thought for some people it may hit home if they hear another user actually try it lol. oh wel I will put the card on the shelf it will be usefull again soon.

Definitely hits home, I'm in the same boat. Do your drives get labelled "too many errors" ?

Let me know if downgrading fixes your issue, i'll probably do the same.

 

[kspare]

downgrading to the a 6gb lsi controller makes a difference. totally stable.

 

[TXAG26]

I'm not using my onboard LSI 3008 controller either. There is an LSI 2308 based card in my new FreeNAS setup temporarily until the 3008 12Gb/s have a little more time to simmer and cook. I'm hoping the jump to BSD 10 or 11 by FreeNAS should get things ironed out.

 

[fullspeed]

Just as a follow up, I downgraded to an LSI 9206-16e 6GB/s card.. booted up and it kernel panicked and rebooted 2x in a row. I switched to a different PCI slot and it finally booted + imported the pool but the logs were full of constant SCSI errors and I ended up reverting back to my LSI 9300 12GB/s card for now.

I did notice there was a firmware/driver mismatch (FW 20 driver 16) so I might dump the 9200 into a spare box and flash the older FW before trying again.

 

[Nicholas M]

Are 12Gbit LSI (9300) cards still unstable in 9.3.1? Im looking to purchase a few and the JBOD is 12Gbit capable, seams a shame to not match it with a 12Gbit HBA.

 

[kspare]

Unless you are multiple 40gb adaptors, and all 12gb drives, there isn't much point. Stick to what works.

 

[Nicholas M]

We are driving 90 8TB SAS drives per JBOD and will have multiple 10Gbit ethernet from the server. Hence why i was keen to use 12Gbit.. LSI have freebsd drivers for 9.3, but i understand FreeNAS had custom drivers for LSI HBAs?

 

[kspare]

How much data transfer do you anticipate? Are you thinking the network cards are going to run at 60% utilization at all times?

 

[jgreco]

Not custom, so much as carefully handpicked. You're still better off with the 6Gbps HBA. They're cheap and they can be replaced later. You can absolutely drive a 12Gbps JBOD with a 6Gbps card.

 

[Nicholas M]

The data will come from several other servers with multiple raid 6 arrays (Nexsan E60s connected over FC to the servers). The ZFS volume will host what we call 'extended retention', i.e. older backup jobs, whereas the latest jobs will stay on the Nexsans. If the ZFS system is fast enough we would look to replace the NexSans with ZFS arrays. It could be pushed quite hard.

The way the financing works here, i might have to buy 6's, use them initially and 12's ready for when the drivers are stable. The system has a 5 year lifecycle, so i want to future proof myself.

 

[depasseg]

I've had no issues with my 12g adapter and matching backplane.

 

[fullspeed]

Are 12Gbit LSI (9300) cards still unstable in 9.3.1? Im looking to purchase a few and the JBOD is 12Gbit capable, seams a shame to not match it with a 12Gbit HBA.

I've never had issues with my 9300E, Currently running the latest version of Freenas. It has 135 x 6TB disks.

It's telling me to upgrade my firmware from version 5 to 9 but I'm not touching it unless I need to.

 

[Bidule0hm]

Did I just read 135 drives?

I'm curious to know the vdev layout :)

 

[depasseg]

I've never had issues with my 9300E, Currently running the latest version of Freenas. It has 135 x 6TB disks.

It's telling me to upgrade my firmware from version 5 to 9 but I'm not touching it unless I need to.

Seeing as it is a recommendation from freenas, freedsd, and LSI, I would upgrade the firmware to v9 to match the driver version.

 

[fullspeed]

Did I just read 135 drives?

I'm curious to know the vdev layout :)

The main server has 150 actually (including spares), replicated to an identical server (I still have to ferry some disks over to it to match).

They are 12 x raidz3
+ 2 x 200GB SSD for log
+ 4 x 520GB SSD for cache

I'd post the whole thing but it would take at least two full pages, basically 12 of these:

raidz3-11 ONLINE 0 0 0
gptid/70aaae63-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/720193f4-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/735ee5ae-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/74b1b8f0-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/75fe27ec-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/77588fac-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/78b1e85e-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/7a087835-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/7b64e6c1-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/7cbc79e1-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/7e156727-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0
gptid/7f7cd0a7-6bf0-11e5-a515-d4ae528f1d5b ONLINE 0 0 0

Seeing as it is a recommendation from freenas, freedsd, and LSI, I would upgrade the firmware to v9 to match the driver version.

If I had more time to deal with the potential fallout I might, Too much other stuff on the go. That server has never had
a single issue (on the 9300) so i'm not excited at the prospect of messing with it.