HP dl380e G8 - P822 - HotSwap

[daftu]

Hi,
I'm new in TrueNAS community and trying to get working my first server.
I've HP dl380e G8 with P822 turned into HBA mode.

Attached disk are directly visible in TrueNAS and I can read all disk S.M.A.R.T. informations, create zpools etc.

The problem is with hot swap. After inserting new drive I got following messages in server console:
ciss0: *** Hot-plug drive inserted, Port=5I Box=1 Bay=13 SN=XXXXX ciss0: error detecting logical drive configuration (target status) ciss0: Warning, cannot get physical lin list

Finally the drive is not visible in Disk menu. There is no /dev/daX for a new drive.
This is supported SAS WD drive but no matter what drive I hot swap it's not being detected.

After OS reboot the drive is normally available.

Hot Swap is very important in case of disk failure etc. How can I fix this?

 

[jgreco]

You need to get rid of the RAID controller and replace it with an actual HBA, not just something pretending to be an HBA.

Please visit the following link:

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

which will explain why "HBA mode" is not equivalent to an HBA and why your controller is not useful. And, to be extra clear, be aware that the measure of suitability to the task is NOT "but I got it to make a pool" -- the ciss driver is total crap for demanding applications like ZFS, and we've had a few people who have lost pools because they didn't follow the advice.

 

[daftu]

You need to get rid of the RAID controller and replace it with an actual HBA, not just something pretending to be an HBA.

Could you recommend some controller which fit my server and will support all 14 drives? Or I have to buy actually 2 controllers (avoid using expander)?

 

[jgreco]

https://www.newegg.com/hp-p822-sas/p/N82E16816401945

The NewEgg image suggests your RAID controller has two SFF-8087 ports internally. If this is the case, many of the normal HBA solutions are likely to work for you. If you have more than eight drives being managed by the internal ports right now, this suggests that there is an SAS expander in play, so I would put good odds on being able to just remove the old controller and swap in an LSI HBA.

If you don't know what an SAS expander is, feel free to head on over to the SAS Primer at

https://www.truenas.com/community/t...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

It's basically the equivalent of an ethernet switch, but for SAS.

Some of the Dell and HP vendor gear is funny about "unsupported" cards being added to their hardware manifests. As such, I cannot make promises about any particular selection that will work for you. My go-to selection would be a Dell H310 crossflashed to IT mode, as long as the cables "work out." These are usually about $30-$40 on eBay, often crossflashed for FreeNAS for free by the seller. However, your HP may not appreciate a Dell card.

Do be aware that you need to have IT mode firmware flashed on the HBA. Once there is an LSI HBA with IT firmware present, you will join many tens of thousands of other deployments running them where they just magically work.

 

[daftu]

Thank you for this great answer.
I have possibility to get an old (but officially compatible with my server) HP H220 array controller which has HBA mode.
Isn’t it enough to use HBA or do I need to flash it to IT mode?
Does IT mode support only SAS drives or also SATA? Because I have a few SATA which I would use along with SAS.

 

[daftu]

And the last question (I think :)).
Will my current zpools work (my current controller running in „pseudo” HBA but drives are visible as physical to ZFS) after replacing controller to full HBA? Of course I’ll do a backups before but does it have a chance to work with new controller without recovery or recreating?

 

[jgreco]

Thank you for this great answer.
I have possibility to get an old (but officially compatible with my server) HP H220 array controller which has HBA mode.
Isn’t it enough to use HBA or do I need to flash it to IT mode?
Does IT mode support only SAS drives or also SATA? Because I have a few SATA which I would use along with SAS.

It's not enough to "use HBA". What we really need is for the firmware to be IT. Please read the previous link about RAID vs HBA. The goal here is to get a driver that is known to work 100.000% correctly in all cases. Not all drivers do, which is what seems to have prompted your posts here, so you actually have direct experience with a crappy driver (ciss). The LSI MFI driver, which is what usually powers the LSI 9240 HBA's, is known for being problematic and I think we've just debugged a few of those here on the forum in the past weeks, complete with likely data loss. So, HBA in IT is the winner. Please!

The SAS technology itself supports SATA drives through the use of SATP (SATA tunneling protocol), so yes, unless your vendor has been a jerk and has configured things to forbid SATA, or your cabling is too long (SATA is limited to 1 meter while SAS is several), SATA drives work fine, alone, alongside SAS, etc.

And the last question (I think :)).
Will my current zpools work (my current controller running in „pseudo” HBA but drives are visible as physical to ZFS) after replacing controller to full HBA? Of course I’ll do a backups before but does it have a chance to work with new controller without recovery or recreating?

High chance of it working, but not guaranteed. Most recent generation RAID controllers in "HBA" mode don't do stupid things, but older controllers, especially IIRC some of the ciss and 3Ware controllers, liked to make "virtual disks" for their JBOD modes, which created a proprietary partition on the disk (to identify it as a RAID controller managed disk) and then your "disk" lived inside that, so you had a partition-inside-a-partition thing when you pulled the disk and looked at it on a plain SATA controller. This is a form of vendor lock-in. This is the only reason I can think of that it MIGHT not work.

 

[daftu]

Finally I got HP H220 controller with IT mode firmware.
Disk and zpools are available after controller swap - great.
Hotswap started working - great too!

But now I have still problem with writeback cache on some old SATA WD Black drives.
I tried following commands but it is still off (but says it supports write cache).

Code:

smartctl --set wcache,on /dev/da6
echo "WCE: 1" | camcontrol modepage da6 -m 0x08 -e

This is weird because one of the drives (which is exactly the same model as other one) it works with "Write cache = Enabled", but no others.

Code:

for i in {6..10}; smartctl -ax /dev/da$i | grep -i "Write cache";
Write cache is:   Disabled
Write cache is:   Disabled
Write cache is:   Disabled
Write cache is:   Enabled
Write cache is:   Disabled

I don't think it could be something to set in controller BIOS, could be?

EDIT:
Ofcourse without write cache enabled disk speed is terrible.