Degreaded freenas pool

[Kurt]

I have been spending a considerable time with the Freenas Mini .. But things are mostly working after a boot device fail, except the following issue:

Code:

  pool: main
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0 in 0 days 01:31:09 with 0 errors on Sun May 19 01:31:10 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        main                                            DEGRADED     0     0     0
          mirror-0                                      DEGRADED     0     0     0
            14361325566141919790                        OFFLINE      0     0     0  was /dev/gptid/13cd0d02-d642-11e7-8936-d05099c311bd
            gptid/14f630b6-d642-11e7-8936-d05099c311bd  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/161d45a1-d642-11e7-8936-d05099c311bd  ONLINE       0     0     0
            gptid/1743ac36-d642-11e7-8936-d05099c311bd  ONLINE       0     0     0

Yes, I have tried to online it, was looking at replacing it when I found what I *think* the problem, is some of the device labels have disappeared.

gpart lists the device and all its partitions, , the device has passed the short SMART test I have run..

After running glable status, I find that its missing a few of == gpart devices.

Bottom line, the was /dev/gptid/13cd0d02-d642-11e7-8936-d05099c311bd bit is missing from

Code:

glabel status
                                      Name  Status  Components
gptid/ab294291-d2a2-11e9-905e-d05099c311bd     N/A  ada0p1
gptid/ab2c3d07-d2a2-11e9-905e-d05099c311bd     N/A  ada0p2
gptid/4cef39f6-8d75-11e9-b12f-d05099c311bd     N/A  ada1p1
gptid/14f630b6-d642-11e7-8936-d05099c311bd     N/A  ada2p2
gptid/161d45a1-d642-11e7-8936-d05099c311bd     N/A  ada3p2
gptid/1743ac36-d642-11e7-8936-d05099c311bd     N/A  ada4p2
gptid/14dfe5da-d642-11e7-8936-d05099c311bd     N/A  ada2p1

So, I am pretty sure I just need to re-add the missing labels .. but I am not sure I know the correct command to use so that it will re-join the pool successfully. I suspect restoring all the missing devices labels might be needed, however, I only know what this one devices label should be.

Thanks for any suggestions..

 

[Chris Moore]

One of the first things we need to know is what version of FreeNAS you are using. Please review this guide:
https://www.ixsystems.com/community/threads/forum-guidelines.45124

I have tried to online it, was looking at replacing it when I found what I *think* the problem, is some of the device labels have disappeared.

Did you take some action that would have deleted the partition? It looks like you are missing some other partitions also. Each drive should have a p1 and a p2 with p2 being the data partition but several of these drives only have a p2 partition. If you didn't make the change, why would you think it disappeared on it's own, if there were not some fault with the drive?
The first troubleshooting step to take when a drive goes out like this is to run a SMART long test and look at the extended results of that test.
You should have both short and long tests scheduled through the task scheduler in FreeNAS. Short tests only take a few minutes to complete and can be done daily. Long tests can take several hours (depends on drive capacity) and I do them weekly.
Have you done a LONG test of the drive? Have you looked at the extended SMART output?

Hard Drive Troubleshooting Guide (All Versions of FreeNAS)
https://www.ixsystems.com/community...bleshooting-guide-all-versions-of-freenas.17/

Other Useful Commands
https://www.ixsystems.com/community/threads/useful-commands.30314/#post-195192

 

[Kurt]

I apologize for omitting the version: FreeNAS-11.2-U3

I also had a support ticket open: NWE-711-48168, on a different but precipitating issue (see this thread: https://www.ixsystems.com/community/threads/freenas-mini-boot-from-ssd.76927/#post-535122)

That issue still remains partially unresolved.. I can boot from my new ssd now, but have to manually use F11 to select it... Still waiting for a response to get a new SATADOM from ixsystems ... anyway..

As for your question about SMART tests. I have run the short test, and there is NOTHING to suggest there is a problem with the device. However, I am running the long test now. Maybe freenas will be vindicated, and the (18 month old, lightly used WD RED) drive is failing ..

As to your other observation -- which I did notice, and is what I meant by " I find that (glabel status) its missing a few of == (equivalent) gpart list devices. "

So your observations are spot on :) However, despite labels being missing for 3 of my 4 data drives (ada1( missing p2), ada2(GOOD), ada3(missing p1, but WORKING!), ada4(missing p1, but WORKING), after the boot drive crashed (and was restored from a config only a day or so old), this one (ada1) device remains inaccessible by the FREENAS OS, not the motherboard or the smatctrl testing tools. I suspect this is directly related to the ada1p2 mapping being missing ....

Your response raises the some concern, as I interpret that to mean that NONE of these labels *SHOULD* be missing, and I totally agree!! (BUT THEY ARE) However, I did not even know what glable , gpart, zpool where until sitting down to figure out why this is not working, so this was a problem long before I started down the shell route. Also note, this is all running on "FREENAS HARDWARE, and FRENASS SOFTWARE!!!". After spending a number of hours trying to find a way to get this device back to 100%, all of the problems I am have been trying to resolve, unsuccessfully so far, fell like OS issues not hardware.

My current assessment remains, the missing labels is the most likely problem.. How do I fix the labels ? (If the OS is in charge of the labels, then, well, there is the answer).

 

[Chris Moore]

So your observations are spot on :) However, despite labels being missing for 3 of my 4 data drives (ada1( missing p2), ada2(GOOD), ada3(missing p1, but WORKING!), ada4(missing p1, but WORKING), after the boot drive crashed (and was restored from a config only a day or so old), this one (ada1) device remains inaccessible by the FREENAS OS, not the motherboard or the smatctrl testing tools. I suspect this is directly related to the ada1p2 mapping being missing ....

No. The p1 / p2 indicates partition 1 and partition 2. If the drive is present and accounted for, the partition should be present and accounted for. There is no way for a partition to not be visible if the drive is visible. Did you disconnect any of your drives?
If you added a physical drive (such as the SSD) it could easily have changed the ada# and that happens all the time and is of no consequence. The storage pool is mapped based on the gptid, so if the drive is there, ZFS will find it. If it isn't showing up, either the drive is broken or it is disconnected.

Maybe freenas will be vindicated, and the (18 month old, lightly used WD RED) drive is failing ..

Can easily happen, but it might just as easily be disconnected and the drive you are looking at (ada1) is actually the boot drive. Don't look at ada1 and think that has a direct relationship to a SATA port on the system board. Did you look at the drive troubleshooting information I shared earlier?
You need to pull the SMART status on each of your drives just to make sure that the drive is actually the drive you think it is.
Also, short tests can pass all day every day and the drive can still be bad. The long test is a surface scan of the drive and is the way we actually test drives. It needs to be done on a regular schedule. I test my drives weekly. I manage hundreds of drives at work and it is not uncommon for a drive to fail in the first three to six months. Don't think for a moment that it can't happen, regardless of who makes it or what model it is.

My current assessment remains, the missing labels is the most likely problem..

That simply isn't possible. You need to look at other options because that isn't even an option.

If the OS is in charge of the labels, then, well, there is the answer

The partitions are named (the gptid) when the drive is formatted. It is a randomly generated code that the FreeNAS middle-ware assigns. The only way it would go away or even change is if the drive was reformatted. It is recorded on the drive. You can move a drive (set of drives) from one FreeNAS system to another and those numbers do not change. Your idea is simply NOT how it works.

 

[Kurt]

Chris, I agree, this should NOT be happening.. But it is. So, when you get past that let me know.

 

[Chris Moore]

Chris, I agree, this should NOT be happening.. But it is. So, when you get past that let me know.

Good luck with that.
Post the output of smartctl –a /dev/ada # substituting the # of every drive in your system. You can put the output of each drive inside a separate set of [CODE]code tags like these[/CODE] and the results should look something like this:

Code:

root@Emily-NAS:~ # smartctl -x /dev/da4
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Desktop HDD.15
Device Model:     ST4000DM000-1F2168
Serial Number:    xxxxxxxx
LU WWN Device Id: 5 000c50 09221606a
Firmware Version: CC54
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 17 16:38:32 2019 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  107) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 506) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x1085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   115   099   006    -    99032192
  3 Spin_Up_Time            PO----   091   091   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    102
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   077   060   030    -    54265527
  9 Power_On_Hours          -O--CK   078   078   000    -    19371
10 Spin_Retry_Count        PO--C-   100   100   097    -    0
12 Power_Cycle_Count       -O--CK   100   100   020    -    101
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0 0 0
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   064   052   045    -    36 (Min/Max 35/45)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    69
193 Load_Cycle_Count        -O--CK   083   083   000    -    35751
194 Temperature_Celsius     -O---K   036   048   000    -    36 (0 13 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
240 Head_Flying_Hours       ------   100   253   000    -    16051h+52m+21.178s
241 Total_LBAs_Written      ------   100   253   000    -    53449642764
242 Total_LBAs_Read         ------   100   253   000    -    1417092037891
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O   1223  Current Device Internal Status Data log
0x25       GPL     R/O   1223  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      20  Device vendor specific log
0xa2       GPL     VS    4496  Device vendor specific log
0xa8       GPL,SL  VS     129  Device vendor specific log
0xa9       GPL,SL  VS       1  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xb0       GPL     VS    5176  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL,SL  VS      10  Device vendor specific log
0xc3       GPL,SL  VS       8  Device vendor specific log
0xc4       GPL,SL  VS       5  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19355         -
# 2  Short offline       Completed without error       00%     19331         -
# 3  Short offline       Completed without error       00%     19307         -
# 4  Extended offline    Completed without error       00%     19275         -
# 5  Short offline       Completed without error       00%     19259         -
# 6  Extended offline    Completed without error       00%     19227         -
# 7  Short offline       Completed without error       00%     19211         -
# 8  Short offline       Completed without error       00%     19187         -
# 9  Short offline       Completed without error       00%     19163         -
#10  Short offline       Completed without error       00%     19139         -
#11  Extended offline    Completed without error       00%     19107         -
#12  Short offline       Completed without error       00%     19091         -
#13  Extended offline    Completed without error       00%     19059         -
#14  Short offline       Completed without error       00%     19043         -
#15  Short offline       Completed without error       00%     19019         -
#16  Short offline       Completed without error       00%     18995         -
#17  Short offline       Completed without error       00%     18971         -
#18  Extended offline    Completed without error       00%     18939         -
#19  Short offline       Completed without error       00%     18923         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    36 Celsius
Power Cycle Min/Max Temperature:     35/45 Celsius
Lifetime    Min/Max Temperature:     13/48 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Data Table command not supported

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 2) ==
0x01  0x008  4             101  ---  Lifetime Power-On Resets
0x01  0x010  4           19371  ---  Power-on Hours
0x01  0x018  6     53743102015  ---  Logical Sectors Written
0x01  0x020  6       515004590  ---  Number of Write Commands
0x01  0x028  6    168297044561  ---  Logical Sectors Read
0x01  0x030  6      1112770269  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           19371  ---  Spindle Motor Power-on Hours
0x03  0x010  4           10874  ---  Head Flying Hours
0x03  0x018  4           35751  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            1  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS

You do that, for all your drives, then we might have something to talk about.

 

[Kurt]

Chris, thank your for all your help and suggesting, that was very helpful. You confidence in my diagnosis being wrong helped alot.

The only way it would go away or even change is if the drive was reformatted.

Was spot on!

I now know what happened, though I am still not really sure how it occurred (aside from the obvious anyway :( ).

Root cause: OS was installed on one of the data drives. (Obvious cause, I did that at some point. That said, I was aware of which disks were which, and was trying very hard to NOT ruin the array).

I ended up pulling all the disks in the system except the boot disk and discovered it was not actually boot-able. Reinstalled the OS using UEFI option, still not bootbale, reinstalled the OS again with BIOS option, YEAH .. bootable. Take home point, the freenas mini built with the ASRok C2750D41 motherboard does not work with the Freenas UEFI, at least not using a SATAIII port.

After I determined the root case, I found this helpful post:
https://forums.freebsd.org/threads/fast-and-easy-delete-of-partition-and-filesystem-data.26282/

I deleted the partitions for the data volume, and the system now boots correctly. This left me with trying to bring that disk back. Finally, gave up. Replacing an already added device to the a pool it already belong to was proving difficult .. I finally gave up and purchased a replacement drive.

In the end, I doubt I would have encountered any issues if the UEFI install option actually worked, or was just not an option. I ended up re-installing the OS multiple times, from different ports in an attempt to get things working - so it must have been at some point in that iterative process where the wrong device was selected during the install process. I found another post by someone who suggested unplugging all but the boot device .. I lost track of that post, but that was a great idea.

 

[Chris Moore]

You confidence in my diagnosis being wrong helped a lot.

I do have some experience with these things. Thank you for getting back to us about the root of the problem.