Zpool import Status Unavail

[Puaro1]

Hello guys and girls.
So sorry if this questions has been answered before, i was looking and did not find anything as a solution.

Yesterday all of a sudden 2 of my 6 drives stopped working. I mean they don't even get power. Windows can't event initialize them and bios can't see them.

I don't have a backup of the pool and files. Because i never thought that 2 drives can fail at the same time. :(

4 of the drives are healthy and working and i replaced the 2 bad drives.

Now when i try to "zpool import" or "zpool import -f" it says:

Code:

pool: anazone
id: 857141534003576755
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing devices and try again.

config:

anazone -> UNAVAIL insuddicient replicas
raidz1-0 -> UNAVAIL insuddicient replicas
    10464928511143806530 UNAVAIL cannot open
    gptid/4e90a67b-3642-11ec-bb2e-4ccc6ab0e782 ONLINE
    gptid/58a0a38e-638b-11e9-ab06-4ccc6ab0e782 ONLINE
    4677569254766473206 UNAVAIL cannot open
    gptid/678cba-638b-11e9-ab06-4ccc6ab0e782 ONLINE
    gptid/94a0a0ae-0734-11ea-a6bb-4ccc6ab0e782 ONLINE

Can you please help me ?
I don't know what to do and i don't know much of commands.

Thank you in advance.

 

[somethingweird]

If I'm reading this right, its raidz1 -

therefore you allowed only 1 drive to fail.
Since 2 drives failed.. you can't recover -

possible solution - but don't know if it's right.

You could try get any one of the bad drive working - then you can new drive to resilver -
once the new drive done, then replace the bad drive with a new drive.

 

[Puaro1]

Suddenly both of the drives are completely dead and i can't get them to work.
Even the bios can't see them.
Any other trick or help please ?

 

[Whattteva]

It's really unlikely for two drives to fail all at once like that. I'd be more inclined to think it's your SATA controller/HBA that's gone bad, whichever one you're using. You didn't really list your specs, so it's my best guess. Have you been checking your SMART data on the drives?

 

[Puaro1]

So sorry, for the specs.
It has 32GB Ram non ECC, 6 x 4TB Seagate Barracuda Drives and i have them on the motherboard SATA connectors.

Yes i was checking for SMART errors, i had setup emails and all.

I got an email 3 hours before the incident saying:
* Update Train EOL Reached
* Boot Pool Status Is DEGRADED: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state.

I was out of the office though and i said probably its a hard disk and i need to replace him.
Then 1 hour before the complete shutdown i got a second email saying:

* Pool anazone state is UNAVAIL: One or more devices are faulted in response to IO failures.

And that's it. When i arrived, i had to restart the machine and i could not see the pool.
I saw that 2 of the drives where not present event on BIOS, i tried multiple times to powered them up, but nothing happened.

I immediate replaced the drives but i'm getting the same message.

Any thoughts, hacks, tricks to try ?
Even to try to power one of the drives so i can resilver, mount or backup my files ?

 

[Arwen]

Have you tried swapping the bad drives into known good cables and host SATA ports?

Please note that if you can't get 1 of your bad drives working again, it will be full restore time. No if ands or buts. RAID-Z1 only protects from 1 failed disk, not 2.

 

[Puaro1]

I did try to swap sata cables, power cables and even SATA ports. But nothing happened.
Now i'm thinking of purchasing a new barracuda drive so i can at least try to swap the small motherboard on the drives, In case of a power failure that cause something bad to the mobo of the drives.

Something like this can happen only with sudden power loss or something similar.

I don't know any other way to power up the drives or make them work. They are not in warranty anymore.

Any thoughts, hacks, tricks to try ?
Even to try to power one of the drives so i can resilver, mount or backup my files ?

 

[somethingweird]

Shipping it out to hard drive recovery service - that will rebuild the drive - so it can power up? *maybe* the data is there, just won't power up!

Newbie Question - if hard drive recovery service - could extract bit by bit to another drive - is it possible to import it back into the pool?

 

[Puaro1]

I just hang up the phone with 2 recovery professionals, both said that they would try to recover the whole data, so they need the whole array (all 6 drives). The cost can be 4 digits.

I think that if they can somehow manage to power up 1 of the failed drives, the "zpool import" can be completed and i can replace the drives. But i don't know for sure.

Either way, they don't fix the drives, they try to replicated them to new ones and then complete the array. If they succeed then they backup the data.

You pay for the whole service.

I will ship the drives tomorrow morning... let's hope for the best.

Thank you all for your efforts.

P.S.: Both of the professional guys on the phone, told me that the "Seagate Barracuda" models ST4000DM004 & ST4000DM005 and basically all the barracuda models are NOT to be trusted and to be avoided at all costs. Also talked about something like a SMR technology to reduce costs, that hurt write commands.

 

[Whattteva]

* Boot Pool Status Is DEGRADED: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state.

* Pool anazone state is UNAVAIL: One or more devices are faulted in response to IO failures.

Wait, so both your data pool AND your boot pool are degraded/unavailable?

That sounds like a much deeper problem. Somehow all your hard drives are failing, even the ones in the boot pool.
Something in that computer is messing up. I don't know if it's your PSU, your cables. Look at your SMART data. I hope you don't have more power cycle count than power-on count like another guy did a couple of weeks ago.

 

[Davvo]

SMR drives check
RAIDZ1 layout check
non-ECC RAM check
No backup check

It's not a configuration I would trust holding any important data.
How often did your recurrent smart tests and scrubs run?

Device-wise you want to put Seagate's Ironwolf line into your nas. You want CMR, not SMR.

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

Also, when you put the system togheter you should properly burn-in your hardware.

Building, Burn-In, and Testing your FreeNAS system

I've been meaning to post some guidance here for a while now. We frequently see people come to the forums with hardware problems that should have washed out in the system build process, but since many of the users here are DIY'ers without...

www.truenas.com

 

[Puaro1]

Wait, so both your data pool AND your boot pool are degraded/unavailable?

That sounds like a much deeper problem. Somehow all your hard drives are failing, even the ones in the boot pool.
Something in that computer is messing up. I don't know if it's your PSU, your cables. Look at your SMART data. I hope you don't have more power cycle count than power-on count like another guy did a couple of weeks ago.

No not at the same time, but yes at the same day.
3 hours apart. :/

I believe that it was the power.
Someone from the office told me (i was out of town) that we had 3 sudden power outages without completely losing the power.
My guess the drives snapped, since i did not had a UPS on the machine

 

[Puaro1]

SMR drives check
RAIDZ1 layout check
non-ECC RAM check
No backup check

It's not a configuration I would trust holding any important data.
How often did your recurrent smart tests and scrubs run?

Device-wise you want to put Seagate's Ironwolf line into your nas. You want CMR, not SMR.

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.truenas.com

Also, when you put the system togheter you should properly burn-in your hardware.

Building, Burn-In, and Testing your FreeNAS system

I've been meaning to post some guidance here for a while now. We frequently see people come to the forums with hardware problems that should have washed out in the system build process, but since many of the users here are DIY'ers without...

www.truenas.com

I was populating new disks, specifically the IronWolf PRO, but i only managed to install 1.
Now i ordered another 5
Lesson learned :/

----
Does really ECC Ram makes such a great difference, if i'm using Vengeance Pro 32GB DDR4 RAM ?

As for the backup, you are completely true. I'm such an idiot.... :/

 

[somethingweird]

No not at the same time, but yes at the same day.
3 hours apart. :/

I believe that it was the power.
Someone from the office told me (i was out of town) that we had 3 sudden power outages without completely losing the power.
My guess the drives snapped, since i did not had a UPS on the machine

Was it at least plug into a surge protector and not directly to the wall? - just wondering.

 

[Puaro1]

Was it at least plug into a surge protector and not directly to the wall? - just wondering.

Yes 2 of them on the rack.

 

[Davvo]

I was populating new disks, specifically the IronWolf PRO, but i only managed to install 1.
Now i ordered another 5
Lesson learned :/

----
Does really ECC Ram makes such a great difference, if i'm using Vengeance Pro 32GB DDR4 RAM ?

As for the backup, you are completely true. I'm such an idiot.... :/

Wait, so were you resilvering while this event happened? Who could have guessed: SMR drives suck!
Next time please do write all the informations when opening the thread, it does help everyone.
Edit: Didn't notice the post about the power outrages, had understood drives failed during resilvering (but was mainly making fun of SMR drives).
Anyway, I would run a long smart test on that Ironwolf drive if you plan to put it in the new system. Just make sure it's all OK.

About ECC, read here. I personally believe it is mission critical (it's no coincidence I put that article in my signature).

Mistakes are what make us grow, so don't blame yourself too much and use this opportunity to learn as much as you can.

 

[Arwen]

To be clear about un-expected power losses. ZFS was specifically designed to handle random power losses without pool corruption. (Aka, any pre-existing data remains intact, without problems.) Data in flight can be lost, but that applies to ANY file system and RAID scheme.

Of course, this does not help in the case of hardware failure(s).

 

[Puaro1]

Wait, so were you resilvering while this event happened? Who could have guessed: SMR drives suck!
Next time please do write all the informations when opening the thread, it does help everyone.
Edit: Didn't notice the post about the power outrages, had understood drives failed during resilvering (but was mainly making fun of SMR drives).
Anyway, I would run a long smart test on that Ironwolf drive if you plan to put it in the new system. Just make sure it's all OK.

About ECC, read here. I personally believe it is mission critical (it's no coincidence I put that article in my signature).

Mistakes are what make us grow, so don't blame yourself too much and use this opportunity to learn as much as you can.

Yea for sure will learn from my mistakes, now ZFS3 is my number one choice ! I already started planning on a second nas to use as a backup and a third backup option would be to send some specific files to the cloud.

Btw, do you know any automation on TrueNas Core edition that i can use to automated the procedure ?

On the second system i will use ECC Ram for sure. Do you have something to recommend ? DDR4 32GB

I'm waiting for the data recovery specialist to finish the procedure and then clean format all the remaining drives and smart check all of them.

@Arwen yes completely understood that when my 2 drives failed at the same time. Who knows maybe the other ones have some problems also.
I'm checking all my systems and servers right now for Errors and so far none found. It's very weird those 2 Barracuda had that issue.
As i'm told those drives are s....t so i would not use them again. I will repurpose them to a desktop or something.

 

[Davvo]

Yea for sure will learn from my mistakes, now ZFS3 is my number one choice ! I already started planning on a second nas to use as a backup and a third backup option would be to send some specific files to the cloud.

RAIDZ3 means you need to simultaneously lose 4 disks in order to lose your data.
RAIDZ2 is still a valid option for vdevs composed by 5-7 disks.

Btw, do you know any automation on TrueNas Core edition that i can use to automated the procedure ?

You can schedule tasks, incuding ZFS replication or cloud sync, but I am sure there are user scripts out there for more specific needs. It's the strenght of CORE.

On the second system i will use ECC Ram for sure. Do you have something to recommend ? DDR4 32GB

I was told Crucial, Samsung, Micron, and Hynix are among the best in the market.

 

[Whattteva]

I was populating new disks, specifically the IronWolf PRO, but i only managed to install 1.
Now i ordered another 5
Lesson learned :/

I think the lesson learned here is more than just going from RAIDZ1 to RAIDZ2.
I don't think you realize that your data loss is a result of several contributing factors, not just RAIDZ1. The RAIDZ1 itself would've been fine if you didn't have other failures along with it (ie. power loss, no UPS (which more than likely would've prevented the other drives going kaput. And of course, no backups. Because even RAIDZ2 or RAIDZ3 is no substitute for backups.

Also, you still haven't posted your SMART data. Even the specs you posted isn't really "specs". It's just a list of your hard drives. It's almost like you're trying to resist help.

----
Does really ECC Ram makes such a great difference, if i'm using Vengeance Pro 32GB DDR4 RAM ?

As for the backup, you are completely true. I'm such an idiot.... :/

I wouldn't say it makes that much of a difference. Plenty have run successfully without it, but I do consider it essential on data you will weep about losing or getting corrupted. Whether you deem it worth it or not depends on you. Considering you're mourning your data loss now, I'm guessing it would be worth it.

Yes 2 of them on the rack.

You actually have enough money to build a rack setup, but skimping on the actual components that actually run the server. It's a trend I have noticed with a lot of forum members that end up losing their data. I really don't understand the rationale. A lot of people splurge for things like large capacity hard drives and other things that aren't too relevant to ensuring data integrity and they put lots of important data on it and just trust it without backups. I don't get it. I do not get it, dear readers.