Upgrade to TrueNAS-13.0-U5.2 and now my Storage Pool is OFFLINE

[Tong_Po]

Ok, it seems that I am just one of many who have, are or will suffer from this. While I am somewhat familiar with TrueNAS, I am by no means an expert or even an intermediate with it.
Long story short, I performed the upgrade, it rebooted, after that point my pool was Offline. Now when I log into my system, this is what I get:

zpool import: no pools available to import.

I did save the config prior to updating. I imported the saved config and still my pool is OFFLINE.


Any and all help is greatly appreciated at this point. My biggest pain point with this is that if I cant recover or bring the pool back ONLINE, i will lose roughly 3TB of movies for my Emby.

Oh, it also seems that my System Dataset is on the Production Pool which I cant seem to get online.



Thank you,

 

[sretalla]

if zpool import gives you nothing to work with, we're back down to hardware...

How are your disks connected?

Do you have an HBA or are you using the onboard SATA ports? (which Motherboard...)

 

[Tong_Po]

Dell PowerEdge R730xd
TrueNAS Core 13.0-U5.2

RAM: 64GB
Hard Drives (Boot): 2 x Dell 300GB 15k SAS Enterprise
Hard Drives (Storage): 16 x Dell 1.92TB SSD Enterprise
RAID config: raidz2 (if i remember correctly)
HBA: PERC H730PMini (Embedded)
Network: Intel(R) 2P X520/2P I350 rNDC

 

[sretalla]

HBA: PERC H730PMini (Embedded)

Was that flashed to IT mode?

If not, maybe that's the issue.

If it was, was the firmware updated to the latest supported version?

 

[Tong_Po]

IT is in RAID mode.

Why would IT/HBA/RAID mode be an issue now? It was working before the upgrade.

 

[sretalla]

As soon as you mess with anything (and your system dataset is on there, so at least a few changes would happen during an upgrade), there's a chance that there's corruption introduced.

With disks attached via a RAID controller, you're in trouble, I'm afraid...

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[Tong_Po]

Sorry, I forgot to attach the screenshot.

 

[Whattteva]

Never use RAID with ZFS. It works well.... until it doesn't.

Why would IT/HBA/RAID mode be an issue now? It was working before the upgrade.

Working before doesn't mean it would work indefinitely. This is typically the case with people running ZFS using non-recommended setups. It works well for a year until something happens (upgrade, power loss, hardware failure, etc.). Typically, it's the power loss that jump starts everything into no man's land.

 

[jgreco]

Why would IT/HBA/RAID mode be an issue now? It was working before the upgrade.

Because it was only (giant airquotes) "working" before the upgrade. The LSI RAID driver can theoretically talk to an attached HDD via your LSI RAID (which is what the PERC is), as, after all, that is its intended purpose in life. However, as support for new RAID cards or fixes for bugs with existing RAID cards change the behaviour of the system, it could be that the driver no longer behaves as it once did. And it didn't behave correctly before. It's like driving around in a car without seatbelts. The fact that you made the 3000 mile from New York to LA without seat belts and survived to tell the tale does not mean that it is safe to drive without seat belts. It just means you got lucky. Likewise, there were failure modes in your LSI RAID setup that existed before your upgrade. Maybe you never ran into them. Maybe a driver update caused you to run into a new failure mode. Hard to say. But the LSI RAID was never expected to work correctly or to be safe to use. The LSI RAID stuff is fundamentally oriented towards usage with their virtual volumes as implemented via RAID1, RAID5, etc. It may seem to work fine as a ZFS HBA until one day it suddenly doesn't, and then, boom.

 

[Tong_Po]

Thank you both for the info and responses. Much appreciated. At this point, what I am hearing is that I am basically SOL? Sound about right? Would looking into some kind of data recovery service be worth a shot?

 

[HoneyBadger]

What version of TrueNAS did you upgrade from?

You may be able to revert to a previous boot environment from System - Boot and regain access to your data that way, but this doesn't invalidate any of the concerns raised by other posters here regarding the use of a RAID controller with TrueNAS.

 

[Tong_Po]

Upgraded from 13.0-U5.1

 

[jgreco]

Thank you both for the info and responses. Much appreciated. At this point, what I am hearing is that I am basically SOL? Sound about right? Would looking into some kind of data recovery service be worth a shot?

Your data pool is probably intact. You should be able to replace the H730p with a standard HBA such as the LSI 9300-8i or I believe the PERC HBA330. I am not a Dell expert so this is general advice, not a specific recommendation.

 

[Tong_Po]

Does TrueNAS have a hardware compatibility matrix or something similar?

 

[Tong_Po]

Your data pool is probably intact. You should be able to replace the H730p with a standard HBA such as the LSI 9300-8i or I believe the PERC HBA330. I am not a Dell expert so this is general advice, not a specific recommendation.

I would need something to support up to 24 drives, as my 730xd's are fully populated.

 

[jgreco]

No. iXsystems has previously stated a preference to avoid doing this, so here in the forums we meticulously do the work of gathering the stuff that is known to work well. There are lists of recommended hardware under the Resources section, and then for more specific guidance for stuff like HBA's and network controllers, there are also more specific guides such as my 10 Gig Networking Primer and the RAID/HBA document you were previously handed. It is possible that other things may work, and many people have tried. For example, I do not recommend Mellanox ethernet cards due to the complexity for beginners and also some deficiencies in driver support, but you may well be able to get them to work perfectly fine for your particular use case. Or you may find that a PERC H730p seems to work fine. In general, if you're not sure, ask. Forum members are generally quite good about guiding you. Most of us are very aware that Dell sells your R730xd with that Windows-oriented PERC H730p controller. And look, it's a great controller. I have a BUNCH of H740p's in our ESXi hosts. But they're not good for ZFS. We won't blame you for the H730p. We understand how it came to be in your server. But we also cannot help that it is a very bad choice for use with ZFS.

 

[jgreco]

I would need something to support up to 24 drives, as my 730xd's are fully populated.

Again, only general familiarity here. Your 730xd almost certainly has an SAS expander on a drive backplane to handle the 24 drives. This is a device that acts kinda like an ethernet switch, except for SAS, so that many drives can share the eight SAS channels that your H730p provides. Check your sell sheet for the 730xd to see what disk controller options are offered. I believe you may find the PERC HBA330 as one of those options. If you end up going with a generic LSI HBA 9300-8i, be sure it is one that supports 24 devices -- it is possible for these to be firmware locked for a smaller number of devices. Also be aware that some people have had "fan issues" with some OEM server models; I believe this includes the 730xd. Check to see what previous users have arrived at for a solution here on the forums. Search button's in the upper right hand corner. You are best served by their hard-won experience, not my offhand advice. Dell likes doing goofy stuff with fans etc.

 

[HoneyBadger]

Upgraded from 13.0-U5.1

There shouldn't have been any changes between these revisions (eg: drivers) that would have broken things.

If you're at a shell (SSH, preferably) what do you see as a result of zpool import with no options?

I would need something to support up to 24 drives, as my 730xd's are fully populated.

Your R730XD should have a SAS expander in its drive backplane, and only likely two internal SAS cables running back to your H730p. You should be able to swap it like-for-like with an HBA330 - provided that you haven't built a virtual device in the PERC RAID BIOS setup.

Are your drives currently visible through iDRAC or the PERC BIOS?

 

[Tong_Po]

I plan on continuing to use my R730xd but do plan on changing the HBA to something more supported. Right now I am just at the point of trying to get my data back. That way i can then migrate the data off and rectify the server with the proper, recommended hardware.

 

[Tong_Po]

There shouldn't have been any changes between these revisions (eg: drivers) that would have broken things.

If you're at a shell (SSH, preferably) what do you see as a result of zpool import with no options?



Your R730XD should have a SAS expander in its drive backplane, and only likely two internal SAS cables running back to your H730p. You should be able to swap it like-for-like with an HBA330 - provided that you haven't built a virtual device in the PERC RAID BIOS setup.

Are your drives currently visible through iDRAC or the PERC BIOS?

zpool import = no pools available to import.
Yes. The drives are visible in iDRAC. They even show up in the TrueNAS gui, just cant import the pool.