zpool disappeared, disk still visible

Status
Not open for further replies.

Erd

Cadet
Joined
May 30, 2016
Messages
5
Hey guys I've been running a FreeNAS box casually for about 2 years, and the past 3 or 4 months I've been building up my server and having it do more and more. About a month ago I decided my FreeNAS configs were borked up and decided to start fresh (I was running 9.1.x at the time). So I installed 9.10 on my flash drive and started configuring everything.

Heres the specs:
FreeNAS 9.10
AMD Athlon X2 5400+
8GB Ram
2 x 2TB HDDs
1 x 3TB HDD (was external, then removed from case and made internal)
HDDs connected via SATA2 in AHCI

Fast forward to a couple weeks ago, and I noticed my external drive was failing SMART checks so I decided I'd take it out of the external shell and make it an internal drive. This fixed the SMART failures. However, now volume disappeared but I decided "thats fine I have a backup and I'll just wipe it clean and restore it". Everything restores fine and the system is stable and I go on with life.

Last night I decided to reboot my system after it had been up since I made the above swap, but the new volume is gone! It should have mounted to /mnt/Hydrogen instead I have an error "The volume Hydrogen (ZFS) state is UNKNOWN"

My USB stick also recently has a DEGRADED status (I've ordered an SSD to replace it). This is the output of zpool status -v:

Code:
freenas# zpool status -v

  pool: Giant

state: ONLINE

  scan: scrub repaired 0 in 3h43m with 0 errors on Sun May 29 05:43:21 2016

config:


NAME                                          STATE     READ WRITE CKSUM

Giant                                         ONLINE       0     0     0

  gptid/08d31604-8d61-11e3-b925-001fd090a73f  ONLINE       0     0     0


errors: No known data errors


  pool: Scratch

state: ONLINE

  scan: scrub repaired 0 in 1h10m with 0 errors on Sun May  1 01:10:19 2016

config:


NAME                                          STATE     READ WRITE CKSUM

Scratch                                       ONLINE       0     0     0

  gptid/4306bc8d-ac3e-11e4-b13b-001fd090a73f  ONLINE       0     0     0


errors: No known data errors


  pool: freenas-boot

state: DEGRADED

status: One or more devices has experienced an error resulting in data

corruption.  Applications may be affected.

action: Restore the file in question if possible.  Otherwise restore the

entire pool from backup.

  see: http://illumos.org/msg/ZFS-8000-8A

  scan: none requested

config:


NAME        STATE     READ WRITE CKSUM

freenas-boot  DEGRADED     0     0     1

  da0p2     DEGRADED     0     0     2  too many errors


errors: Permanent errors have been detected in the following files:


        /var/db/system/update/base-os-9.10-STABLE-201605021851-d55ab9177fa7bbcd849b9f0687646c3d-9.10-STABLE-201605240427-b1814726769db6f75a5797438735378e.tgz


I also ran zpool import, which gave no output. "zpool import Hydrogen" gave the following:
Code:
cannot import 'Hydrogen': no such pool available


The disk is definitely being detected, "smartctl -a /dev/ada0" gives the following:
Code:
smartctl 6.4 2015-06-04 r4109 [FreeBSD 10.3-RELEASE amd64] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-9YN166
Serial Number:    Z1F0F1JE
LU WWN Device Id: 5 000c50 03f918f3e
Firmware Version: CC9D
User Capacity:    3,000,588,754,432 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon May 30 12:12:02 2016 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (  584) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 333) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x3081)    SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   119   099   006    Pre-fail  Always       -       213242304
  3 Spin_Up_Time            0x0003   093   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   099   099   020    Old_age   Always       -       1558
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   073   056   030    Pre-fail  Always       -       12968535153
  9 Power_On_Hours          0x0032   072   072   000    Old_age   Always       -       24948
10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
12 Power_Cycle_Count       0x0032   099   099   020    Old_age   Always       -       1529
183 Runtime_Bad_Block       0x0032   024   024   000    Old_age   Always       -       76
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   097   097   000    Old_age   Always       -       3
190 Airflow_Temperature_Cel 0x0022   062   029   045    Old_age   Always   In_the_past 38 (26 18 38 38 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       819
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       253811
194 Temperature_Celsius     0x0022   038   071   000    Old_age   Always       -       38 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       12119h+04m+50.964s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       61677123469235
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       280798454013571

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%     24948         -
# 2  Extended offline    Completed without error       00%         5         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


I can't import the volume from the gui, there are no volumes in the list to select from, and the disk shows up in the "View Disks" gui.

I'm not very FreeNAS saavy, I know enough to be dangerous, but I'm at a complete loss of why this would happen. I've tried re-seating the sata power/data connections on my HDDs as a last ditch effort. Could the data-corruption on my boot drive have caused this? Its corrupt on a .tgz file, which (if I'm not mistaken), is just the compressed update when downloaded (which gets unpacked) so it shouldn't really affect anything, right?

The HDD in question has ~25,000 hours on it (from the smrtctl command above), that might be the cause? I dunno, I'm lost :(

I didn't have periodic snapshots set up, I've lost my old backup from the original swap (I did an rm -r 2 days ago!!), and I don't have the recycle bin option checked on the relevant CIFS shares. If I can't recover the volume I'm pretty sure I've lost ~2TB of data. Do I have any options left, or am I boned?
 
Last edited:

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
What is the type of the vdev of the pool? RAID-Z1?
 

Erd

Cadet
Joined
May 30, 2016
Messages
5
They are all normal ZFS volumes, no fancy RAID configurations
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
They are all normal ZFS volumes, no fancy RAID configurations
What on Earth is that supposed to mean?
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
Just to warn right now: stripe + one or more drive dead = no data anymore. I really hope it's a RAID-Z1 or a mirror...
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
ST3000DM001
The disk is probably dying. Running a SMART test might confirm that.
Do I have any options left, or am I boned?
If the disk is indeed dying, you could try GNU ddrescue as a last resort.
My USB stick also recently has a DEGRADED status (I've ordered an SSD to replace it)
There's a chance your problems are a result of a failed boot device, so you might be able to recover after doing a clean install to a new device.
 

Erd

Cadet
Joined
May 30, 2016
Messages
5
Thanks for the help guys, I'll look into ddrescue as an option, but I'm not getting my hopes up. I just ordered 4x2TB drives to pair up with my 2 existing drives, and convert over to a RAIDZ2 setup.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
Don't forget to set up a schedule for short and extended SMART tests and make sure your email notifications are working.
 
Status
Not open for further replies.
Top