TrueNAS 12.2 to 12.6 disks disappeared

[modernpaul]

I am running an older HP storageworks server booting with TrueNAS. I have 12 SAS drives each in a RAID 0 configuration which is working on all my other similar servers. I had no problem with my disks showing up when I was running 12.2. I do have other systems running this same version and did the same upgrade but no issues and they are configured the same way. I was having an issue with AD so I restarted the server and the drives disappeared in TrueNAS. The raid controller shows all 12 disks and so does camcontrol devlist:

<HP RAID 1(1+0) OK> at scbus0 target 0 lun 0 (pass0,da0)
<HP RAID 0 OK> at scbus2 target 0 lun 0 (pass1,da1)
<HP RAID 0 OK> at scbus2 target 1 lun 0 (pass2,da2)
<HP RAID 0 OK> at scbus2 target 2 lun 0 (pass3,da3)
<HP RAID 0 OK> at scbus2 target 3 lun 0 (pass4,da4)
<HP RAID 0 OK> at scbus2 target 4 lun 0 (pass5,da5)
<HP RAID 0 OK> at scbus2 target 5 lun 0 (pass6,da6)
<HP RAID 0 OK> at scbus2 target 6 lun 0 (pass7,da7)
<HP RAID 0 OK> at scbus2 target 7 lun 0 (pass8,da8)
<HP RAID 0 OK> at scbus2 target 8 lun 0 (pass9,da9)
<HP RAID 0 OK> at scbus2 target 9 lun 0 (pass10,da10)
<HP RAID 0 OK> at scbus2 target 10 lun 0 (pass11,da11)
<HP RAID 0 OK> at scbus2 target 11 lun 0 (pass12,da12)

When I am in the GUI under Storage -> Disks though it just shows.

da0
50014380111513D0
279.37 GiB
boot-pool

da12
PACCPID11250FFV
1.82 TiB
N/A

Any ideas how to be able to see the rest of the drives that the system and the OS can see?

 

[NugentS]

You are using a RAID Controller?
These are highly not recommended (something of an understatement)

The only thing I can suggest is ensure you have a copy of the config and rebuild TN from the start - see if it sees the array from scratch

Then get a proper HBA and do things properly

However what does zpool status say?

 

[modernpaul]

Well I think that's what's strange. They pull status shows the two drive raid and just the single drive by itself. Kind of just like it shows with the GUI output. There is no pool currently set up I had just deleted it. So it seems like it was potentially working until I deleted the pool but now I can't see the drives anymore other than the 12th one.

 

[NugentS]

I seriously suggest you get a proper HBA in IT Mode

 

[modernpaul]

We have older hardware that works just fine. You can't blame this on the hardware. It was working and is working in other systems. I have to find out why the TrueNAS software isn't seeing it is the issue. If the OS can see it then TrueNAS should be able to so it is a bug or a configuration issue.

 

[Samuel Tai]

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

@modernpaul, if you're using the RAID controller to present JBOD to TrueNAS, you can expect odd behavior like this. TrueNAS expects it has direct access to the drives. Running a RAID controller in JBOD mode means it interposes itself between the disks and TrueNAS, and performs address translation. If the controller gets confused, you can see disks disappear.

It's not a matter of whether the hardware works or not. This is a hard prerequisite for TrueNAS.

 

[modernpaul]

I believe that the truenas needs to be reset to see the drives. How does one do this? I know I saw the setting when poking around yesterday but can't find it today. I think that since I disconnected the pool that it decided to ignore the drives as well. Now I just need to have it not ignore them anymore. I have not found a way to flash the controller for IT mode. It is a HP212 controller.

Do you know of a cheap hba that one could get with decent performance and I'll just buy one of them. I'm hoping for around $50 would be good. Since you are saying it has to be JBOD I thought you might know. I'll need to buy multiple of them.

 

[Samuel Tai]

I believe that the truenas needs to be reset to see the drives.

Sorry, no. TrueNAS tracks drives by their ZFS partition GUID. The RAID controller is scrambling these or stripping these out.

You can find IT-mode HBAs on eBay for around $50. See https://www.ebay.com/itm/1439678787...rentrq:2e6e8eac17d0ab8477352f93fffded94|iid:1.

 

[jgreco]

I believe that the truenas needs to be reset to see the drives. How does one do this?

Have you tried turning it off and then turning it back on again?



Sorry, had to be done

Not clear what you think needs to be "reset". That's not how servers generally work. They're not crap-grade Windows PC's. However, it is totally possible that whatever compatibility was allowing the NAS to see the disks wasn't particularly robust, and it is not unheard-of for drivers or cards to crash in ways that make your drives/pool vanish. That's basically an unfixable problem which might be temporarily cleared with the classic IT Crowd fix, but it'll just happen again, and the real fix is to replace it with a supported HBA in IT mode.

I know I saw the setting when poking around yesterday but can't find it today. I think that since I disconnected the pool that it decided to ignore the drives as well. Now I just need to have it not ignore them anymore. I have not found a way to flash the controller for IT mode. It is a HP212 controller.

Yeah, some janky old HP SmartArray controller. It's not going to be able to be flashed to IT mode because it isn't an LSI low end RAID controller.

Do you know of a cheap hba that one could get with decent performance and I'll just buy one of them. I'm hoping for around $50 would be good. Since you are saying it has to be JBOD I thought you might know. I'll need to buy multiple of them.

No, we're not saying it has to be "JBOD" (which used to mean "Just a Bunch Of Disks" but RAID card manufacturers coopted the term in a way that makes this super-ambiguous), it needs to be actual direct access to the disks, i.e. without the RAID controller abstracting them. You can tell that the RAID controller is there because it says "HP RAID 0 OK". The article @Samuel Tai linked to explains a lot more.

Basically, the LSI HBA's are the normal solution, and used ones on eBay such as the Dell PERC H200, H310, IBM ServeRAID M1015, etc., start at about $30, but you can find ones already configured with the correct IT mode firmware for $40-$60.

 

[modernpaul]

Update. I tried resetting the settings and that did nothing. I had to go to the HP212 controller and delete each of the raid0 drives and just recreate them. I was able to then see them so I could create a new pool. Somehow with the current identifier TrueNAS had blocked them from view but I don't know why. Everything is working good again.

 

[modernpaul]

Thanks Jgreco, I do know all about RAID controllers and HBAs. I was just looking for recommendations for ones known to work. I didn't see any way to flash the ones I have so I wasn't sure if someone else might have done it before with this one. I don't want to buy one with the HBA idea and it still doesn't work because of compatibility which is why I was requesting a model name of one known to work. I greatly appreciate your input in providing some examples. Yes I did reboot the server multiple times.

In the end it really seems like a bug in the TrueNAS or ZFS software. It was able to see the drives until they are put into a pool. Even if the pool is deleted or detached it still maintains the drives so that it thinks they are in use. By reassigning new drive numbers or identifications through the reRAIDing process on the controller that seemed to allow TrueNAS to see them again.

 

[MutoSan]

Thanks Jgreco, I do know all about RAID controllers and HBAs. I was just looking for recommendations for ones known to work. I didn't see any way to flash the ones I have so I wasn't sure if someone else might have done it before with this one. I don't want to buy one with the HBA idea and it still doesn't work because of compatibility which is why I was requesting a model name of one known to work. I greatly appreciate your input in providing some examples. Yes I did reboot the server multiple times.

In the end it really seems like a bug in the TrueNAS or ZFS software. It was able to see the drives until they are put into a pool. Even if the pool is deleted or detached it still maintains the drives so that it thinks they are in use. By reassigning new drive numbers or identifications through the reRAIDing process on the controller that seemed to allow TrueNAS to see them again.

So you blaming truenas or ZFS for something which is since YEARS not recommendet at all. Dont use Raid when you are using ZFS. Point. Thats not a bug or anything even it worked out for some time for you or even for a longer time. Its not recommendet at all because it does not work well.