Offline pool

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
Ok well, Here I go again..

Another issue.

One of my pools have decided to go OFFLINE on me now and when I look into the drives area it shows this :
1616027550100.png


BOOT pool is just what it says.. for the OS
and Jukebox5 has data on it.. BUT missing is Jukebox4. It shows like this:
1616027609385.png


I do not know what the MFID is for Jukebox4 so I have NO IDEA what one of these marked "Unused" is the drive.

How do I :
1) Figure out what drive it is that Jukebox4 resides on
2) recover all that data
3) get the pool back online

Thank you all in advance..
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
Sorry for the multi posts.. Just want to give you guys as much info as I can. I ran zpool import and got this :
1616030737862.png


any idea how to fix this issue to recover my data and what caused this data to be corrupt?
I spin up my servers on the weekends and they are powered down during the week as I am not using them
I ALWAYS use the power off function to turn them off and never just press the power button or pull the plug to turn them off
thanks again guys for the help in advance.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Would I be correct in interpreting that you have some sort of LSI RAID controller attaching your disks?

Ditch it.
Indeed ditching the RAID controller is a good idea if you want to continue using TrueNAS.

However, if you were hoping for some kind of chance to recover data, don't do that just yet.

You can try the import with a -F and see if dropping the last few transactions can make the pool importable.

After that, get your data out (if you can) and ditch the RAID for an HBA.
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
How do I "ditch the raid" on a Dell Power Edge server? Its hardware raid. The drives are each set up as RAID-0 in a single drive config.. Thanks again for helping..
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Its hardware raid
That's exactly the part that needs ditching. (see post linked by @jgreco )

You would need an appropriate PCIe card (HBA) to connect the disks/backplane(s) to.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeah, it's a PowerEdge 2950, but the problem is that Dell sells dozens of options for their servers, and the 2950 reached end-of-life years ago, so I can't even pull up a current configurator page to begin to guess at what MIGHT be in your system.

Help us help you. Please review the Forum Rules, conveniently linked at the top of every page in red, which discusses how to create a cogent description of your system and its configuration.

In general, we don't mind a little too much detail, so if you have pictures of the innards and can post about what the lifecycle controller says about the hardware manifest, that's useful stuff.

Dell makes a number of HBA's like the H200 and H310 that work great with FreeNAS once they've been crossflashed to LSI firmware, but Dell also has a horrible history of using embedded controllers in strange places which often mean you need to re-card AND re-cable.
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
My system says it has a PERC 6/i raid card in it. 32 GB ram, and 4 cores of processors
Hope that helps a bit
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Oh dear lord, so that's probably got like SFF8484 connectors on it. You're going to have to evaluate what you need. What's on the backplane?
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
I am not sure, I will have to pull the server out of its rack and get back to you with some pics etc.. I know there is a riser board in there with 2 or 3 PCI slots.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The SCSI (SAS) backplane.

Your RAID controller has some multilane cables that go {somewhere} inside the chassis, probably to a PC board that has a bunch of SAS connectors on it that the drives slide into. The type of connector there is important. And, unfortunately, it is old enough that one of several different things might have been done.

Normally, with an older (but not elderly) 12-drive server that has SFF-8087-based backplane, used for purposes of illustration of the quandary, you have an SFF-8087 cable that goes from an HBA to the backplane with an expander chip on it that handles 12 drives, or two SFF-8087's that go to support up to 8 drives directly and four bays left unconnected, or two SFF-8087's from an HBA and a reverse breakout from the motherboard that go to three SFF-8087's on the backplane, which support up to 12 drives, or two HBA's providing three SFF-8087's to the backplane.

Unfortunately there are also configurations that use standard "SATA" type connectors (the dreaded Supermicro "Q" style chassis being the classic example) instead of SFF-8087 on the backplane, which creates a messier cabling scenario.

But tragically, with elderly servers like yours, SAS wasn't really quite standardized well enough that there is any sort of guarantee as to what they might have done for cabling attachment to the backplane.

I'm sorry this isn't easier.
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
Could not get the box out of the rack today but here is some info i got from the orig config sheet on the unit. Let me know if this
helps

1616110324463.png


1616110351491.png
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Yeuck.

Okay, running the KC411 thru the eBay meat grinder suggests that this is a very short SFF-8484 to SFF-8484 cable, which raised my eyebrows until I saw
it_photo_59830.jpg

this picture which shows a slightly perplexing design where they have the card up front. But you "done good" because this gives us a discussion starting point.

So this server looks like it only has six bays in it. I can only imagine that they were trying to keep slots in the back free, which is the usual reason for the PERC "internal" designs, but usually I've seen those mid-chassis (R510 etc), not sitting up front. I suppose they were trying to keep the space open for "large" cards.

Anyways, the backplane ... If those are actually SFF-8484 on the backplane, then I think you might want to try two cables like these


and replacing that controller. A PERC H200 crossflashed to IT might be able to be cabled into the existing "internal" RAID slot, else I would suggest a PERC H310 crossflashed to IT mode installed in one of the rear slots.

Unfortunately, the way I normally work through these things is just to try them. Because it is totally possible that they only LOOK like 8484.
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
So. Basiclly Freenas and Truenas are not going to work for my server as the software does not play well with hardware raid right and the only way to fix that is to re-wire the server?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
You can interpret it that way, and it's not a misinterpretation.

Due to the way computer hardware works, much of it is unsuitable. FreeNAS is like a semi-truck. It's designed to hook up to and haul a semi-trailer. You can bring your old Ford F350 to the game and say "but it has good towing capacity", and you might even be able to jerry rig something to make it possible to move the semi-trailer without mods to the truck, but in the end that doesn't work out well. You really need to be able to do something like retrofit the F350 to handle a fifth wheel hitch. Which is totally doable, but no longer "stock".

You may actually be in more of a frustrating position than all the people who want to try to bring their Ford Pintos (AMD APU) or Ford GT's (old gaming PC) to the show; your server is at least well in the realm of being possible to mod to be acceptable.
 

thereverend

Dabbler
Joined
Apr 13, 2020
Messages
36
The honesty is refreshing. At least I wont be banging my head against the wall.
Can you tell me how to recover the data on the Corrupt drive?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It's not clear exactly what's happened, and I'm not a ZFS data recovery expert.

You can safely ask FreeBSD what kind of partitions are on a given disk using "gpart list <device>", so for example if you were to try "gpart list mfid4" I would expect you'd probably find a provider, maybe mfid4p2 listed as "type: freebsd-zfs". That corresponds to your "jukebox5" pool. Look at the "rawuuid" and it should correspond to what is listed by "zpool status jukebox5".

This will help you identify which disk might be holding your missing pool.

Now, you can try to force an import, as ZFS indicated to you in your post #3, and as @sretalla discussed in #5. Personally, I wouldn't do this without having a backup disk image, but that may be beyond the scope of this discussion. I am typically very cautious about this stuff.

On a meta-note, it appears that you were building pools out of single disk devices. ZFS isn't doing much for you when you do that. ZFS is intended to build better data stores through redundancy, such that when a disk develops issues, you have redundancy on another disk (or disks). If you are not doing that, there may not be much value to your running FreeNAS, and it might be better to look at other NASware if you aren't interested in replacing the RAID controller. ZFS and FreeNAS are not the right solution for every problem. I definitely do not use it for every storage solution, but it is very good at what it does.
 
Top