Latest FreeNAS + LSI Cachecade issue

[Johnnwvs]

I am running into a timeout while booting FreeNAS after turning on CacheCade 2 (SSD Cache) on my LSI 9260-8i controller. I did the Google and found one site that suggested upgrading the Firmware, or that a newer FreeNAS fixes the issue - no dice on either.

The error during boot is (from memory):

mfio: COMMAND 0xfffff(blahblahblah) TIMEOUT after X seconds

If I turn off the CacheCade feature, FreeNAS boots fine.

I even tried to re-install FreeNAS with CacheCade on, and FreeNAS happily installed and allowed me to create a ZFS volume on the LSI, everything was great until I rebooted, then back to the mfio/TIMEOUT errors again.

I am on the latest version of FreeNAS, and the latest firmware upgrade for the LSI card, having updated both this week.

Any ideas?

John

 

[jgreco]

Do NOT use a RAID controller with FreeNAS. See the LSI sticky.

 

[Johnnwvs]

I found the sticky and read up a bit.
I guess I should see if I can return this card.

Thanks.

John

 

[cyberjock]

Oh wow.. you have no idea how lucky you were that it didn't work and you came here. Using it that way WOULD have ended up with a trashed pool....

 

[jgreco]

@cyberjock, sometimes I think you are a bit too negative.

There's really no guarantee that he's end up with a trashed pool. In theory the MegaRAID controller is perfectly competent at providing block storage, and failures that result in drives dropping off-line would definitely be noticeable events.

The real concerns are:

1) An unmonitored RAID controller (i.e. not running MegaRAID Storage Manager or equivalent monitoring) would not necessarily provide sufficient warning of impending failure of a drive, which could be monitored via SMART and log messages on a HBA,

2) There have been several instances of people reporting random trouble with the mfi driver which make me wary of it,

3) The use of CacheCade (in particular) or RAID (less specifically) is a sign that the OP didn't do much research into appropriate hardware for ZFS, since lots of RAM and an L2ARC will be much more awesome than CacheCade,

4) The use of a hardware RAID controller can be a performance impediment .... though - and I know I may have the minority viewpoint here - I think that in specific instances you might be able to leverage the write cache on a RAID controller to gain performance wins, and in an environment where you have an MRSM instance that this would be reasonably safe. However, that is a guru level thing and your typical FreeNAS user lacks the correct infrastructure.

 

[cyberjock]

My reason fo what i said was because those pesky hardware RAID write caches seems to be the bane for many people. In 1/2 the people that have contacted me about data recovery for a non-mounting pool they had a RAID card with the write cache enabled....with no BBU.

Now, I realize that's just stupid to do regardless of whether we're talking JBOD, ZFS, or even some valid hardware RAID configuration. But it seems that far too many people see the manual say "makes stuff go faster" and ignore the warnings or understand what its doing or the consequences.

 

[jgreco]

Hey, I'm certainly with you there. But on the flip side, I think it is fair to recognize that if you wanted a really smokin' fast low latency system, there are ways that that could be tackled with hardware RAID cards each providing access to small numbers of disks with a BBU. I am absolutely certain that this is Not A Beginner Level Project and there are plenty of constraints and gotchas.

I'm just not willing to go so far as to say that any use of a RAID controller *must* result in a trashed pool. I think that is too extreme.