OMG! what happened to my dataset?

[berrick]

Hi All,

I can't access my main dataset which consists of 3 x 2Tb disks configured as raidz1-0.

Looking at the logs 1 disk started logging

Code:

Jun 19 03:12:11 NAS smartd[3828]: Device: /dev/ada4, 48 Currently unreadable (pending) sectors
Jun 19 03:12:11 NAS smartd[3828]: Device: /dev/ada4, 48 Offline uncorrectable sectors

But showed nothing was wrong in the gui. Some posts I read state this is normal behaviour but seems a little dangerous to me as this issue is easily missed especially if you dont have alerts configured to send to email. More so if another disk starts flagging the same kind of errors then following occurs!

Code:

	Jun 19 20:39:42 NAS kernel: (ada2:ata2:0:0:0): WRITE_DMA48. ACB: 35 00 70 3f 22 40 a1 00 00 00 00 01
	Jun 19 20:39:42 NAS kernel: (ada2:ata2:0:0:0): CAM status: Command timeout
	Jun 19 20:39:42 NAS kernel: (ada2:ata2:0:0:0): Retrying command
	Jun 19 20:39:42 NAS kernel: (ada3:ata2:0:1:0): lost device
	Jun 19 20:39:42 NAS kernel: (ada2:ata2:0:0:0): lost device
	Jun 19 20:39:42 NAS kernel: (ada3:ata2:0:1:0): removing device entry
	Jun 19 20:39:42 NAS kernel: (ada2:ata2:0:0:0): removing device entry

This all happened whilst i'm away :( 2 of the 3 disks appear failed and the other isn't looking good. So; a couple of questions

1. does FreeNAS 9.10 still not show in gui if a disk starts logging smart uncorrectable sector errors etc
2. As a learning exercise, might the dataset be recovered somehow or am i'm going to have to restore from backup?

Kind regards

 

[Jailer]

If you lost 2 disks from a 3 disk RAIDZ1 pool then your data is gone. You had ample time and warning to mitigate this, why did you ignore the email warnings you were getting?

Can you post the output of zpool status to confirm this is the case.

 

[nojohnny101]

What @Jailer said, looks like you will have to restore from backup. zpool status will confirm.

 

[rs225]

Give zpool status

Check your cables, any power splitters, and the power supply quality.

Your data is probably not gone if any of the drives can be resurrected, imaged, or sent for a data recovery. But if you have backups, this is why you make them.

 

[danb35]

this issue is easily missed especially if you dont have alerts configured to send to email.

...but you should always have alerts configured to send to email.

does FreeNAS 9.10 still not show in gui if a disk starts logging smart uncorrectable sector errors etc

In my experience, yes, it does (and at least as far back as 9.3 does too)--the status light turns red and flashes.

As a learning exercise, might the dataset be recovered somehow

What dataset? You haven't said anything about a dataset. Datasets don't consist of disks or vdevs; I think you mean your pool.

 

[berrick]

Hi Sorry for the delayed reply but as I said I was away when this happened and shutdown the nas to prevent things getting worse. When I got back been mad busy.

So;

You had ample time and warning to mitigate this

If i had been monitoring my emails whilst away, agreed.

you should always have alerts configured to send to email

I do but as above

yes, it does (and at least as far back as 9.3 does too)--the status light turns red and flashes.

On this occasion it didn't turn red. It was green. Clicking on it stated all was OK yet the console output in the footer was showing the smartd warnings

Other weirdness with the NAS when this was happening is that this nas also has a mirror but on the view disk page there was absolutely no mention of these disks. Also never receive any email notification about one of the disks flagging these smartd messages.

So with dread I powered on the nas, it came up and the output from zpool status showed both pools were online. I was also able to access the data of the pool with the dodgy disk!

Disks have been replaced.