Hey everyone,
My FreeNAS box has been running fine for over a year now without any issues. Recently I began noticing a sector error with one of the drives, so I went to replace it but it seems I have made things worse. I've been trying to work it out myself, however I am worried about my data at this point and am hoping for some help.
I have too much data on the pool to backup unfortunately. All the mission critical stuff is backed up however.
Disk 12129061989276932477 is the one with errors and needs to be physically replaced. However the second disk that is faulted has never shown any errors, it was unfortunately removed instead of the other disk. Both disks are installed and have power. I tried to shutdown the server to replace the disk, but it won't shutdown. So, I have no GUI, only CLI.
Here is some information. If I missed anything important let me know.
My FreeNAS box has been running fine for over a year now without any issues. Recently I began noticing a sector error with one of the drives, so I went to replace it but it seems I have made things worse. I've been trying to work it out myself, however I am worried about my data at this point and am hoping for some help.
I have too much data on the pool to backup unfortunately. All the mission critical stuff is backed up however.
Disk 12129061989276932477 is the one with errors and needs to be physically replaced. However the second disk that is faulted has never shown any errors, it was unfortunately removed instead of the other disk. Both disks are installed and have power. I tried to shutdown the server to replace the disk, but it won't shutdown. So, I have no GUI, only CLI.
Here is some information. If I missed anything important let me know.
Code:
[root@freenas] /var/log# zpool status tank pool: tank state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 4h49m with 0 errors on Wed May 28 14:19:13 2014 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/67f41f2e-b5b4-11e2-80a1-001a4d5a464d ONLINE 0 0 0 gptid/6872deef-b5b4-11e2-80a1-001a4d5a464d ONLINE 0 0 0 gptid/68f30a78-b5b4-11e2-80a1-001a4d5a464d ONLINE 0 0 0 gptid/697049b0-b5b4-11e2-80a1-001a4d5a464d ONLINE 0 0 0 12129061989276932477 OFFLINE 0 0 0 was /dev/da11 gptid/6a6bb924-b5b4-11e2-80a1-001a4d5a464d ONLINE 0 0 0 raidz2-1 DEGRADED 0 0 0 gptid/f03441ae-b615-11e2-82ec-001a4d5a464d ONLINE 0 0 0 gptid/f257c65f-b615-11e2-82ec-001a4d5a464d FAULTED 3 0 0 too many errors gptid/f3e21ad8-b615-11e2-82ec-001a4d5a464d ONLINE 0 0 0 gptid/f4764cd6-b615-11e2-82ec-001a4d5a464d ONLINE 0 0 0 gptid/f609d8ab-b615-11e2-82ec-001a4d5a464d ONLINE 0 0 0 gptid/55d57ec3-b692-11e2-8734-001a4d5a464d ONLINE 0 0 0 cache gptid/574cf05e-b616-11e2-82ec-001a4d5a464d ONLINE 0 0 0
Code:
[root@freenas] /var/log# camcontrol devlist <ATA WDC WD20EARX-00P AB51> at scbus0 target 9 lun 0 (pass0,da0) <ATA WDC WD20EARX-00P AB51> at scbus0 target 10 lun 0 (da1,pass1) <ATA WDC WD20EARX-00P AB51> at scbus0 target 11 lun 0 (pass2,da2) <ATA WDC WD20EARX-00P AB51> at scbus0 target 12 lun 0 (pass3,da3) <ATA WDC WD30EZRX-00D 0A80> at scbus0 target 13 lun 0 (pass4,da4) <ATA WDC WD20EARX-00P AB51> at scbus0 target 14 lun 0 (pass5,da5) <ATA INTEL SSDSC2CT06 300i> at scbus0 target 15 lun 0 (pass6,da6) <ATA WDC WD20EZRX-00D 0A80> at scbus0 target 16 lun 0 (pass7,da7) <ATA WDC WD20EZRX-00D 0A80> at scbus0 target 17 lun 0 (pass8,da8) <ATA WDC WD20EZRX-00D 0A80> at scbus0 target 18 lun 0 (pass9,da9) <ATA WDC WD20EZRX-00D 0A80> at scbus0 target 19 lun 0 (pass10,da10) <ATA WDC WD20EARS-00M AB51> at scbus0 target 21 lun 0 (pass11,da11) <ATA WDC WD20EARS-00M AB51> at scbus0 target 22 lun 0 (pass12,da12) <Lexar JumpDrive 1100> at scbus5 target 0 lun 0 (pass13,da13)
Code:
[root@freenas] /var/log# gpart show => 34 3907029101 da0 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da2 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da3 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 5860533101 da4 GPT (2.7T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 5856338696 2 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5k) => 34 3907029101 da5 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 117231341 da6 GPT (55G) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 113036943 2 freebsd-zfs (53G) => 34 3907029101 da7 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da8 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da9 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da10 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da12 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 63 62652353 da13 MBR (29G) 63 1930257 1 freebsd [active] (942M) 1930320 63 - free - (31k) 1930383 1930257 2 freebsd (942M) 3860640 3024 3 freebsd (1.5M) 3863664 41328 4 freebsd (20M) 3904992 58747424 - free - (28G) => 0 1930257 da13s1 BSD (942M) 0 16 - free - (8.0k) 16 1930241 1 !0 (942M) => 34 3907029101 da11 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k) => 34 3907029101 da1 GPT (1.8T) 34 94 - free - (47k) 128 4194304 1 freebsd-swap (2.0G) 4194432 3902834696 2 freebsd-zfs (1.8T) 3907029128 7 - free - (3.5k)
Code:
[root@freenas] /var/log# glabel status Name Status Components gptid/f03441ae-b615-11e2-82ec-001a4d5a464d N/A da0p2 gptid/f3e21ad8-b615-11e2-82ec-001a4d5a464d N/A da2p2 gptid/f4764cd6-b615-11e2-82ec-001a4d5a464d N/A da3p2 gptid/55d57ec3-b692-11e2-8734-001a4d5a464d N/A da4p2 gptid/f609d8ab-b615-11e2-82ec-001a4d5a464d N/A da5p2 gptid/574cf05e-b616-11e2-82ec-001a4d5a464d N/A da6p2 gptid/6872deef-b5b4-11e2-80a1-001a4d5a464d N/A da7p2 gptid/68f30a78-b5b4-11e2-80a1-001a4d5a464d N/A da8p2 gptid/6a6bb924-b5b4-11e2-80a1-001a4d5a464d N/A da9p2 gptid/67f41f2e-b5b4-11e2-80a1-001a4d5a464d N/A da10p2 gptid/697049b0-b5b4-11e2-80a1-001a4d5a464d N/A da12p2 ufs/FreeNASs3 N/A da13s3 ufs/FreeNASs4 N/A da13s4 ufs/FreeNASs1a N/A da13s1a gptid/92f8d8d0-e6b0-11e3-b647-001a4d5a464d N/A da11p2 gptid/f237dfdf-b615-11e2-82ec-001a4d5a464d N/A da1p1 gptid/f257c65f-b615-11e2-82ec-001a4d5a464d N/A da1p2