Hello from Tororo, Uganda (East Africa)

[coolcor]

Hello to all FreeNas forum members.
I've been a freeNas 4Tb user for around 5 years now and have been using it to provide network file-storage for our Heart4Children.org organization without any trouble, until now that is :-(

One disk started warning two weeks ago so I asked a visitor from Germany to bring me two new (slightly bigger) drives (one array in mirror formation) which I received. By the time they left after 5 days I wanted to change my drives (one by one) but by now I could not see my sftp share anymore...
By now BOTH drives have given up!!!
So there you have it, that's why I'm here...

Now when I boot, it tries to do:
Beginning ZFS volume imports
importing (long number)...
cannot import (long number) : no such pool available
No idea how I'm going to retrieve my files now

I have learned system admin over time as a missionary here in Uganda and we are using Ubuntu all over here so I'm not a stranger to troubleshooting and trying stuff on the command line but with this ZFS... it's all new to me and I dont know where to start.

attached are the output of the following commands I found somewhere on the forum:
gpart list, gpart show, zpool import, smartctl -a -q noserial /dev/ada3, camcontrol devlist, glabel status

Any help is really appreciated here

Blessings!

Cor

 

[Ericloewe]

zpool import that there's a pool ready to import that should import fine. The pool is made up of ada3 and ada4, the HGST 4 TB drives.

ada4 seems to be in pretty poor shape, 2000+ bad sectors and holy crap, has it really seen 91 degrees Celsius!? And ada3 96 degrees!?
The SMART output for ada2 seems to be mislabeled and actually be that of ada3.

First of all, do whatever you can to keep those drives cooler, they're above the recommended maximum of 40 degrees Celsius as it is right now. Once that is out of the way, the next step is to try and import the pool with zpool import h4cLive.

SMART data for the other disks would also be useful.

 

[Chris Moore]

These drives should normally last five, or more, years but you must find a way to keep the system cool.
Perhaps the organization can arrange to build a basement?

 

[mopcrust]

You may also might want to consider going with SSDs in your next upgrade cycle. They use less electricity, run cooler and potentially have greater longevity than spinning disks. They also tolerate heat better. If your data are compressible and can fit within one 2TB SSD you can mirror two drives and have a great deal of flexibility.

 

[coolcor]

Hi Y'all, thanks very much for all replies!
Surely the disks have overheated, when I found the machine failing, I went to poweroff and found them really hot. The reason is that one fan completely failed due to the African dust and the one in the power-supply was blocked due to a filter being full of dust. So bad!
@Ericloewe, I tried to do

Code:

zpool import h4cLive

but it gives me a lot of errors:

Dec 20 15:17:52 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:17:52 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:17:52 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:17:52 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:17:52 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:17:55 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:17:55 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:17:55 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:17:55 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:17:55 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:17:59 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:17:59 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:17:59 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:17:59 nas (ada4:ahcich13:0:0:0): RES: 51 40 a0 10 46 40 f4 00 00 38 00
Dec 20 15:17:59 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:18:02 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:02 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:02 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:02 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:02 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:18:05 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:05 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:05 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:05 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:05 nas (ada4:ahcich13:0:0:0): Error 5, Retries exhausted
Dec 20 15:18:08 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:08 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:08 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:08 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:08 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:18:11 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:11 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:11 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:11 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:11 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:18:14 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:14 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:14 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:14 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:14 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:18:17 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:17 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:17 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:17 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:17 nas (ada4:ahcich13:0:0:0): Retrying command
Dec 20 15:18:19 nas (ada4:ahcich13:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 d8 0f 46 40 f4 00 00 01 00 00
Dec 20 15:18:19 nas (ada4:ahcich13:0:0:0): CAM status: ATA Status Error
Dec 20 15:18:19 nas (ada4:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
Dec 20 15:18:19 nas (ada4:ahcich13:0:0:0): RES: 51 40 98 10 46 40 f4 00 00 40 00
Dec 20 15:18:19 nas (ada4:ahcich13:0:0:0): Error 5, Retries exhausted

And now I wonder if the -f option and -o rdonly would help to force a recovery?..
Please advise?
I'm scared I may leave all data useless...

 

[Ericloewe]

It sounds to me like the disk isn't even getting as far as ZFS needs to try to import them. Try unplugging ada4 and importing the pool with only ada3 attached.

That still leaves the question of the other two disks, can you show us the SMART data for those?

 

[coolcor]

It sounds to me like the disk isn't even getting as far as ZFS needs to try to import them. Try unplugging ada4 and importing the pool with only ada3 attached.

That still leaves the question of the other two disks, can you show us the SMART data for those?

Ohhh my dear Eric, you made it work!!!
Just unplugging ada4 made it mount the volume on boot.
Thank you toooo much, for sure I thought all our files were burned

Those ada1 & 2 are the brand new drives of 8Tb each which I added as a new volume in mirror-raid (not sure I said that correctly)
Right now I'm pumping all the data of the cooked 4Tb drive to the new 8Tb volume

Blessings to you!!

Cor