Dell PERC H730 Working with SAS, not SATA

[Chris Dill]

I had a working 11.2 Freenas instance, installed on a Dell T430. This has a PERC H730 in HBA mode, and passes through 8 SAS disks which were in a RAIDZ2.

I bought new sata drives, and stuck them in. Same controller, and TrueNAS 12 does not see them at all.

I have factory reset, re-enabled HBA, and presented the disks as non raid. They are identified by bios, and even identified during boot messages in TrueNAS.

Can anyone point me in the right direction?

I have added this to device.hints

 

[Chris Dill]

Crazy followup, and this does not make sense.

No matter what, in HBA mode, no drives appear. I added two additional drives and changed the controller to RAID mode but left all disks as non-raid, and it did not help. I made a RAID1 out of my two additional disks, and it did not help. I made a RAID0 out of my two additional disks, and all of a sudden everything is visible in TrueNAS.

So to get my disks to show, I have the PERC H730 in RAID mode, with 1 RAID0 configured and the rest of the disks set to non-raid. Any other order of operations or configurations fails. Since this is working, I rebooted a bunch of times, and even set up a 6 disk RAIDz2 array and everything is holding.

DA0 is my RAID0 which is 2X 250GB SATA. DA 1-6 are my 6 TB drives in RAIDx2. DA7 is my USB running TrueNAS.

Here is the view in IPMI:

Anyone know how this is possible? As far as I knew, RAID from any controller should not work, with disks configured as non-RAID? Also keep in mind that this system using HBA mode was running a FreeNas 11 system for the past 2 years.

 

[HoneyBadger]

Is the tunable mrsas_load = YES also present? I'm not sure if that's still required in 12, but in 11 you needed to enable and then actually load the mrsas driver.

I'm not entirely sure on the order of events:

H730 in HBA mode running 11.3 with 8x SAS = Worked fine

That's clear, but next:

"I bought new sata drives, and stuck them in. Same controller, and TrueNAS 12 does not see them at all."

Did you buy the new SATA drives and stick them in a system running 11.3, or upgrade to TN12 first (with HBA mode/8x SAS still working) and then swap the drives?

Anyone know how this is possible? As far as I knew, RAID from any controller should not work, with disks configured as non-RAID? Also keep in mind that this system using HBA mode was running a FreeNas 11 system for the past 2 years.

Several controllers in RAID mode, including the newer PERC series, will pass unconfigured disks through to the OS in a "quasi-HBA" mode.

Given the relative value of the cards though I'd say the easiest fix is to sell the H730 and pick up an HBA330 instead (older cards will work too since it's all spinning disks, but you'll need to replace cables if the ports change)

 

[Chris Dill]

So when I got the new drives, I actually just shut down Freenas 11, inserted new drives, and then booted off of the boot media installer for TrueNAS 12, and did a fresh/format install.

Will running the system as above (one RAID0 + 6 Non-RAID disks) pose a problem?

I did not add that tunable, but I will.

 

[jgreco]

The H730 is a RAID controller and is not suitable for use with FreeNAS.

The mfi driver is not suitable for use with FreeNAS.

Useful links for you to read:

https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/

https://www.truenas.com/community/threads/confused-about-that-lsi-card-join-the-crowd.11901/

 

[Chris Dill]

Yeah I read that, I thought that since H730 was on the hardware list AND that I had been successfully running FreeNAS 11 on it for 2 years, it would work. And its working now, but I am nervous based on what you are saying that I discovered some flaw, and it will crash down.

Based on the fact that I have a Dell T430, I should purchase a HBA330 and replace? If so, what cables do I need to service my 6 disks (or 8)?

Here is my current setup with RAID on:

Code:

root@truenas[~]# camcontrol devlist
<DELL PERC H730 Adp 4.24>          at scbus0 target 0 lun 0 (pass0,da0)
<ATA ST6000NM0044 G007>            at scbus1 target 0 lun 0 (pass1,da1)
<ATA ST6000NM0044 G007>            at scbus1 target 1 lun 0 (pass2,da2)
<ATA ST6000NM0044 G007>            at scbus1 target 2 lun 0 (pass3,da3)
<ATA ST6000NM0044 G007>            at scbus1 target 3 lun 0 (pass4,da4)
<ATA ST6000NM0044 G007>            at scbus1 target 4 lun 0 (pass5,da5)
<ATA ST6000NM0044 G007>            at scbus1 target 5 lun 0 (pass6,da6)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus6 target 0 lun 0 (pass7,ses0)
<PLDS DVD-ROM DH-16D8S MD11>       at scbus12 target 0 lun 0 (pass8,cd0)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus13 target 0 lun 0 (pass9,ses1)
<SanDisk Ultra 1.00>               at scbus15 target 0 lun 0 (da7,pass10)

 

[jgreco]

What hardware list is the H730 on? I can probably have that corrected.

It isn't that the controller doesn't work at all. There are a LOT of disk controllers that work with FreeBSD, and therefore by extension FreeNAS. The problem is that even though it might seem like it works under normal conditions, it may not work correctly under high load, or when disks are removed, or because the driver isn't just quite up to snuff, even if it happens to work fine if you're running it on FreeBSD as a web server.

The ZFS and NAS stuff really puts the storage system through the wringer, and the added expectations of being able to do things like disk swapouts and stuff like that tend to find the sharp edges. I can't remember which controller it was that worked just fine until one disk failed in a certain way, at which point the card firmware freaked out and ZFS would drop all the attached drives from the pool thinking them dead.

For better or for worse, we try very hard to guide people onto the golden path that's known to work.

I don't have any more than this right now, no specific knowledge of your hardware, sorry, and I'm trying to get out the door to an appointment I'm already late for.

 

[Chris Dill]

Thanks for the tips. I ordered a SAS9211-8i and I will stop messing with this PERC H730.

 

[Ericloewe]

Based on the fact that I have a Dell T430, I should purchase a HBA330 and replace? If so, what cables do I need to service my 6 disks (or 8)?

Is it a full-size card? If so, no changes to cables if your connectors are in the same general position.

 

[Ericloewe]

Thanks for the tips. I ordered a SAS9211-8i and I will stop messing with this PERC H730.

If you go with a SAS2 card, you'll need to replace the cables with SFF-8087 ones.

 

[Chris Dill]

If you go with a SAS2 card, you'll need to replace the cables with SFF-8087 ones.

Yes sir, thank you. I ordered a SAS9211-8i and snagged two of these cables, and also this SATA power extension cable in case I need it.

 

[HoneyBadger]

Yes sir, thank you. I ordered a SAS9211-8i and snagged two of these cables, and also this SATA power extension cable in case I need it.

Please post an interior photo of your T430 and the current cables in use. I believe you've purchased the wrong cables, those are "reverse breakout" based on the quotes in the linked item.

 

[Chris Dill]

I don't have access to the server right now. Could you shoot me a link to what cables you think I will need? I'll order both and have them on hand for tomorrow.

Edit: I picked up forward breakout as well.

 

[HoneyBadger]

Could you shoot me a link to what cables you think I will need? I'll order both and have them on hand for tomorrow.

I don't think you'll need "breakout" of any stripe, unless you have no backplane at all in your T430. And if you didn't, those SATA cables still wouldn't physically plug into a SAS drive as they have a one-piece connector for power+data vs. the two distinct SATA plugs.

If you have the expected hot-swap capable SAS3 backplane in your PowerEdge, you need something akin to two of these, but the length and necessary right-angles may be different:

https://www.amazon.com/CableCreation-Internal-Cable-SFF-8643-SFF-8087/dp/B013G4FL0A/

SFF-8087 is the SAS2 "flatter and wider" connector, SFF-8643 is the "square" SAS3 connector. The former should be on the LSI 9211-8i, the latter on your backplane.

 

[Chris Dill]

Thanks.

Edit: You are absolutely correct, the backplane needed the SFF-8643, and the cables you linked should work swimmingly. Thanks again.

 

[Chris Dill]

So I got the new card and cables in hooked it up, booted to free dos and flashed it to it mode successfully. I then installed TrueNAS 12 onto two USB sticks.

Inside of the LSI config utility all of the drives show up fine. While booting, the drives are mentioned. While booted up in shell camconteol devlist shows the drives. But the GUI does not.

Right now I rebooted and went into config utility and I am formatting a drive, but it is sitting at 0% after 20m.

What's the trick to get the drives to show? Do I need to add driver to loader?

Right now the LSI is on v20, I read that there is a v 21- do I need that?

 

[Chris Dill]

Nevermind, I rebooted a few times, and this last time took quite a bit of time and the disks showed up.

 

[YesterEve]

Nevermind, I rebooted a few times, and this last time took quite a bit of time and the disks showed up.

Hi Chris. Just curious what PCIe slot do you have the SAS9211-8i installed in? I'm basically copying your setup but I have a H330 installed in PCIe slot 3 that I'm using for my RAID 1 boot to Proxmox and am attempting to pass the SAS9211-8i currently in PCIe slot 4 through to a VM running TrueNAS. How has your stability been with your setup?