Can't Access Pool?

jonathanb4fitch · Sep 26, 2015

I've got an old machine running 4gb RAM with a core 2 duo CPU and a 1tb black and 4tb red running striped (yes that was not an intelligent thing to do). It was running fine until I transported it to my new apartment. I am new to FreeNAS and an inexperienced linux user, so I'd like to have your help in determining the issue. Thanks!

Code:

[root@freenas ~]# camcontrol devlist                                           
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus0 target 0 lun 0 (pass0,ada0)       
<WDC WD1002FAEX-00Z3A0 05.01D05>   at scbus1 target 0 lun 0 (pass1,ada1)       
<SanDisk Extreme 0001>             at scbus5 target 0 lun 0 (pass2,da0)        
<Generic USB SD Reader 1.00>       at scbus6 target 0 lun 0 (pass3,da1)        
<Generic USB CF Reader 1.01>       at scbus6 target 0 lun 1 (pass4,da2)        
<Generic USB SM Reader 1.02>       at scbus6 target 0 lun 2 (pass5,da3)        
<Generic USB MS Reader 1.03>       at scbus6 target 0 lun 3 (pass6,da4)

Code:

[root@freenas ~]# gpart show                                                   
=>      34  61282564  da0  GPT  (29G)                                          
        34      1024    1  bios-boot  (512k)                                   
      1058         6       - free -  (3.0k)                                    
      1064  61281528    2  freebsd-zfs  (29G)                                  
  61282592         6       - free -  (3.0k)                                    
                                                                               
=>        34  1953525101  ada1  GPT  (931G)                                    
          34          94        - free -  (47k)                                
         128     4194304     1  freebsd-swap  (2.0G)                           
     4194432  1949330696     2  freebsd-zfs  (929G)                            
  1953525128           7        - free -  (3.5k)

Code:

[root@freenas ~]# zpool status                                                 
  pool: freenas-boot                                                           
state: ONLINE                                                                 
  scan: none requested                                                         
config:                                                                        
                                                                               
        NAME                                          STATE     READ WRITE CKSUM
        freenas-boot                                  ONLINE       0     0     0
          gptid/6977d0ce-565f-11e5-8b8f-0019213efb01  ONLINE       0     0     0
                                                                               
errors: No known data errors

Code:

[root@freenas ~]# zpool import                                                 
   pool: RED1                                                                  
     id: 12290036975553001056                                                  
  state: FAULTED                                                               
status: The pool metadata is corrupted.                                       
action: The pool cannot be imported due to damaged devices or data.           
        The pool may be active on another system, but can be imported using    
        the '-f' flag.                                                         
   see: http://illumos.org/msg/ZFS-8000-72                                     
config:                                                                       
                                                                               
        RED1                                          FAULTED  corrupted data  
          gptid/a1d51311-5cc1-11e5-bf0d-0019213efb01  ONLINE

BigDave · Sep 26, 2015

Something's probably come loose during the trip over, check the data cable to the WD Red drive first.
We'll hope it's simple.

jonathanb4fitch · Sep 26, 2015

BigDave said:
Something's probably come loose during the trip over, check the data cable to the WD Red drive first.
We'll hope it's simple.

Checked and reversed the sata cable, same issue.

jgreco · Sep 26, 2015

Run a conveyance test on the Red. Go to the CLI and type

# smartctl -t conveyance /dev/ada0

Wait the suggested number of minutes and then

# smartctl -a /dev/ada0

and see if it shows as "Completed without error".

jonathanb4fitch · Sep 26, 2015

Run a conveyance test on the Red. Go to the CLI and type

# smartctl -t conveyance /dev/ada0

Wait the suggested number of minutes and then

# smartctl -a /dev/ada0

and see if it shows as "Completed without error".

Code:

smartctl -a /dev/ada0
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p25 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Black
Device Model:     WDC WD1002FAEX-00Z3A0
Serial Number:    WD-WCATRC640413
LU WWN Device Id: 5 0014ee 25e880f43
Firmware Version: 05.01D05
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Sep 26 16:31:15 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (15960) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 185) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3037) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   177   175   021    Pre-fail  Always       -       4108
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       254
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       6797
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       246
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       64
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       189
194 Temperature_Celsius     0x0022   098   092   000    Old_age   Always       -       49
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%      6797         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

jgreco · Sep 27, 2015

Try the other disk. I'm trying to figure out what it was that made me suspect that disk and I'm not seeing it. So do the same thing on ada1.

Guessing: something's wrong with it. The conveyance test isn't a comprehensive test but it is a good thing to run after any physical shock to the system.

DrKK · Sep 27, 2015

Wow. Cyberjock must be pretty busy or something to let this thread (guy using crappy stuff, with not enough RAM, and clearly not reading even the basic guides for ZFS setup) go without a light dusting of BBQ sauce.

Even jgreco too; you can normally be counted on for a dose of habanero. Have you guys both gone to charm/finishing school this summer and I didn't hear about it? I mean, I'm not complaining.

jgreco · Sep 27, 2015

I think that you are confusing me and CJ. I think the guy already realizes that is set up has some issues, he already owned up to that, so the big question is whether or not he can get back his data.

jgreco · Sep 27, 2015

damn voice recognition sucks

DrKK · Sep 27, 2015

I, too, am interested in how the user's situation resolves. What exactly are the consequences for him "forcing" the import as the system suggests?

jgreco · Sep 27, 2015

if there are missing vdevs, possibly the loss of the pool.

DrKK · Sep 27, 2015

Well, his camcontrol devlist looks good. Provided the other smartctl looks good on the other drive....

jgreco · Sep 27, 2015

well, one step at a time. The second test gives us some idea where the problem may be.

jonathanb4fitch · Sep 27, 2015

jgreco said:
Try the other disk. I'm trying to figure out what it was that made me suspect that disk and I'm not seeing it. So do the same thing on ada1.

Guessing: something's wrong with it. The conveyance test isn't a comprehensive test but it is a good thing to run after any physical shock to the system.

Code:

[root@freenas] ~# smartctl -a /dev/ada1
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p25 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E3YA9RDC
LU WWN Device Id: 5 0014ee 20c185a40
Firmware Version: 82.00A82
User Capacity:    4,000,785,948,160 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Sep 27 10:14:22 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (53460) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 534) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   180   179   021    Pre-fail  Always       -       8000
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       27
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       217
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       27
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       14
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       46
194 Temperature_Celsius     0x0022   110   100   000    Old_age   Always       -       42
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Conveyance offline  Completed without error       00%       217         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~#

I'm confused... From the camcontrol command, it looks like the 1tb WD Black is assigned to ada1, and the 4tb Red is assigned to ad0, but the conveyance test shows the opposite. Or am I interpreting this wrong?

BigDave · Sep 27, 2015

jonathanb4fitch said:
I'm confused... From the camcontrol command, it looks like the 1tb WD Black is assigned to ada1, and the 4tb Red is assigned to ad0, but the conveyance test shows the opposite. Or am I interpreting this wrong?

In your first post, gpart show lists da0 as 29GB (your boot drive) & ada1 as 931GB (WD 1TB disk)
If FreeNAS can't detect ada0 (4TB Red disk) it would then "bump" the 1TB Black drive to ada0
and thus it shows the 4TB drive as ada1 during the recent conveyance test.
Someone correct me if I'm wrong here, but it seems you are just going to have to eliminate the possible
cause, one, by one :(

jgreco · Sep 27, 2015

It'd be useful to know the output of camcontrol devlist now

jonathanb4fitch · Sep 27, 2015

jgreco said:
It'd be useful to know the output of camcontrol devlist now

Code:

[root@freenas] ~# camcontrol devlist
<WDC WD1002FAEX-00Z3A0 05.01D05>   at scbus1 target 0 lun 0 (pass0,ada0)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus2 target 0 lun 0 (pass1,ada1)
<SanDisk Extreme 0001>             at scbus5 target 0 lun 0 (pass2,da0)
<Generic USB SD Reader 1.00>       at scbus6 target 0 lun 0 (pass3,da1)
<Generic USB CF Reader 1.01>       at scbus6 target 0 lun 1 (pass4,da2)
<Generic USB SM Reader 1.02>       at scbus6 target 0 lun 2 (pass5,da3)
<Generic USB MS Reader 1.03>       at scbus6 target 0 lun 3 (pass6,da4)
[root@freenas] ~#

jgreco · Sep 27, 2015

and "zpool import RED1" does what? still reports FAULTED?

jonathanb4fitch · Sep 27, 2015

jgreco said:
and "zpool import RED1" does what? still reports FAULTED?

Code:

[root@freenas] ~# zpool import RED1
cannot import 'RED1': I/O error
        Destroy and re-create the pool from
        a backup source.

if I specify RED1 as above then it returns that message. Just running zpool import yields the same as shown in the OP

jgreco · Sep 27, 2015

I/O error. Weird. Is there actually an I/O error reported by the kernel when you do that?

Type "dmesg" and look at the end of it (or watch the console).

Important Announcement for the TrueNAS Community.

Can't Access Pool?

Cadet

FreeNAS Enthusiast

Cadet

Resident Grinch

Cadet

Resident Grinch

FreeNAS Generalissimo

Resident Grinch

Resident Grinch

FreeNAS Generalissimo

Resident Grinch

FreeNAS Generalissimo

Resident Grinch

Cadet

FreeNAS Enthusiast

Resident Grinch

Cadet

Resident Grinch

Cadet

Resident Grinch

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Can't Access Pool?"

Similar threads