snowmirage
Dabbler
- Joined
- Sep 20, 2011
- Messages
- 29
This afternoon I was alerted to a degradation on my volume after rebuilding my server this past weekend.
I'm running FreeNAS 9.2.1.5-RELEASEx64 (80x1d35)
Intel Celeron G530 2.4GHZ CPU ( its what I had .... :( )
24GB RAM
My volume consists of
2TB x 7 in RAIDZ2
+
500GB x 6 in RAIDZ2
Started digging around found the following after searching the forums for a refresher on some of the commands.
It appears da6 has the faults from the GUI I found the serial number and was smart enough to note the location of each drive by serial number when I built the server this weekend. I had taken a 7th 2TB drive from my desktop that was working fine and rarely used to increase the array size. So happens this is the drive that appears to have failed...
Before shutting down the server and pulling the drive I did some searching and tried to get some info from smartctl.
Hmm no such luck... when I rebuilt the FreeNAS server I used the beloved IBM M1015 which I flashed to IT mode (all the 2TB disks are on this controller, the others are directly on the ASUS motherboard). Had a hunch I might need to specify the type (-d option) in smartctl. After some google fu it appeared others using the card with SATA drives had it working with "smartctl -d sat"
No luck there either...
Checked the serial of the drive it was just about to expire under warranty, following the mfg guide for RMAing a drive I ran seagates recommended drive test. Which passed.... ran it twice even...
Could someone help me understand FreeNAS kicking back a FAULTED disk that isn't DEFECTIVE and replaceable according to the mfg? I understand hard drives for the most part, they don't usually just "fail" its my understanding that an unreasonable amount of the disk becomes unusable / unreadable / unwritable. Is FreeNAs / ZFS just being super sensitive in this case? Is there anything that can be done to test the disk in Freenas? I haven't had to look this close at a hard drive since windows XP days. I seem to recall utilities that would "repair" the drive, if I remember correctly they would search find the bad sectors mark them "DO NOT USE" or something to that effect?
Any assistance would be greatly appreciated!
I'm running FreeNAS 9.2.1.5-RELEASEx64 (80x1d35)
Intel Celeron G530 2.4GHZ CPU ( its what I had .... :( )
24GB RAM
My volume consists of
2TB x 7 in RAIDZ2
+
500GB x 6 in RAIDZ2
Started digging around found the following after searching the forums for a refresher on some of the commands.
Code:
[root@freenas8] ~# zpool status -v
pool: BetoriaCloud
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
BetoriaCloud DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/d34eb1ac-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/d45cdd91-95d8-11e4-9954-08606e799c45 FAULTED 7 51 0 too many errors
gptid/d5784fe6-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/d75ecda8-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/d94132db-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/da6201f9-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/daf3db54-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/db69757e-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/dbce23db-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/dc2ffede-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/dc8887a5-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/dce7465d-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
gptid/de4e0c23-95d8-11e4-9954-08606e799c45 ONLINE 0 0 0
errors: No known data errors
Code:
[root@freenas8] ~# gpart list Geom name: da8 modified: false state: OK fwheads: 255 fwsectors: 63 last: 15124991 first: 63 entries: 4 scheme: MBR Providers: 1. Name: da8s1 Mediasize: 988291584 (942M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 32256 Mode: r1w0e1 attrib: active rawtype: 165 length: 988291584 offset: 32256 type: freebsd index: 1 end: 1930319 start: 63 2. Name: da8s2 Mediasize: 988291584 (942M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 988356096 Mode: r0w0e0 rawtype: 165 length: 988291584 offset: 988356096 type: freebsd index: 2 end: 3860639 start: 1930383 3. Name: da8s3 Mediasize: 1548288 (1.5M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 1976647680 Mode: r0w0e0 rawtype: 165 length: 1548288 offset: 1976647680 type: freebsd index: 3 end: 3863663 start: 3860640 4. Name: da8s4 Mediasize: 21159936 (20M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 1978195968 Mode: r1w1e2 rawtype: 165 length: 21159936 offset: 1978195968 type: freebsd index: 4 end: 3904991 start: 3863664 Consumers: 1. Name: da8 Mediasize: 7743995904 (7.2G) Sectorsize: 512 Mode: r2w1e4 Geom name: da8s1 modified: false state: OK fwheads: 255 fwsectors: 63 last: 1930256 first: 0 entries: 8 scheme: BSD Providers: 1. Name: da8s1a Mediasize: 988283392 (942M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 40448 Mode: r1w0e1 rawtype: 0 length: 988283392 offset: 8192 type: !0 index: 1 end: 1930256 start: 16 Consumers: 1. Name: da8s1 Mediasize: 988291584 (942M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 32256 Mode: r1w0e1 Geom name: da5 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da5p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: d2f614fe-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da5p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: d34eb1ac-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da5 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r2w2e5 Geom name: da6 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da6p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: d43dd3c1-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da6p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r0w0e0 rawuuid: d45cdd91-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da6 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 Geom name: da0 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da0p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: d56a20e1-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da0p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: d5784fe6-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da0 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r2w2e5 Geom name: da1 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da1p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: d6db7091-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da1p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: d75ecda8-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da1 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r2w2e5 Geom name: da2 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da2p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: d8bbf775-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da2p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: d94132db-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da2 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r2w2e5 Geom name: da3 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da3p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: da4548e0-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da3p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: da6201f9-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da3 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r2w2e5 Geom name: da4 modified: false state: OK fwheads: 255 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da4p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e1 rawuuid: dadb1db3-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da4p2 Mediasize: 1998251364352 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e2 rawuuid: daf3db54-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 1998251364352 offset: 2147549184 type: freebsd-zfs index: 2 end: 3907029127 start: 4194432 Consumers: 1. Name: da4 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r2w2e5 Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 65536 Mode: r1w1e1 rawuuid: db5bd394-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: ada0p2 Mediasize: 497960295936 (463G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2147549184 Mode: r1w1e2 rawuuid: db69757e-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 497960295936 offset: 2147549184 type: freebsd-zfs index: 2 end: 976773134 start: 4194432 Consumers: 1. Name: ada0 Mediasize: 500107862016 (465G) Sectorsize: 512 Mode: r2w2e5 Geom name: ada1 modified: false state: OK fwheads: 16 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada1p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 65536 Mode: r1w1e1 rawuuid: dbc0d449-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: ada1p2 Mediasize: 497960295936 (463G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2147549184 Mode: r1w1e2 rawuuid: dbce23db-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 497960295936 offset: 2147549184 type: freebsd-zfs index: 2 end: 976773134 start: 4194432 Consumers: 1. Name: ada1 Mediasize: 500107862016 (465G) Sectorsize: 512 Mode: r2w2e5 Geom name: ada2 modified: false state: OK fwheads: 16 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada2p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 65536 Mode: r1w1e1 rawuuid: dc238e64-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: ada2p2 Mediasize: 497960295936 (463G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2147549184 Mode: r1w1e2 rawuuid: dc2ffede-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 497960295936 offset: 2147549184 type: freebsd-zfs index: 2 end: 976773134 start: 4194432 Consumers: 1. Name: ada2 Mediasize: 500107862016 (465G) Sectorsize: 512 Mode: r2w2e5 Geom name: ada3 modified: false state: OK fwheads: 16 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada3p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 65536 Mode: r1w1e1 rawuuid: dc7ee555-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: ada3p2 Mediasize: 497960295936 (463G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2147549184 Mode: r1w1e2 rawuuid: dc8887a5-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 497960295936 offset: 2147549184 type: freebsd-zfs index: 2 end: 976773134 start: 4194432 Consumers: 1. Name: ada3 Mediasize: 500107862016 (465G) Sectorsize: 512 Mode: r2w2e5 Geom name: ada4 modified: false state: OK fwheads: 16 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada4p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 65536 Mode: r1w1e1 rawuuid: dcd5cac3-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: ada4p2 Mediasize: 497960295936 (463G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2147549184 Mode: r1w1e2 rawuuid: dce7465d-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 497960295936 offset: 2147549184 type: freebsd-zfs index: 2 end: 976773134 start: 4194432 Consumers: 1. Name: ada4 Mediasize: 500107862016 (465G) Sectorsize: 512 Mode: r2w2e5 Geom name: da7 modified: false state: OK fwheads: 255 fwsectors: 63 last: 976773134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: da7p1 Mediasize: 2147483648 (2.0G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 65536 Mode: r1w1e1 rawuuid: ddae409f-95d8-11e4-9954-08606e799c45 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 2147483648 offset: 65536 type: freebsd-swap index: 1 end: 4194431 start: 128 2. Name: da7p2 Mediasize: 497960295936 (463G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 2147549184 Mode: r1w1e2 rawuuid: de4e0c23-95d8-11e4-9954-08606e799c45 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: (null) length: 497960295936 offset: 2147549184 type: freebsd-zfs index: 2 end: 976773134 start: 4194432 Consumers: 1. Name: da7 Mediasize: 500107862016 (465G) Sectorsize: 512 Mode: r2w2e5 [root@freenas8] ~#
It appears da6 has the faults from the GUI I found the serial number and was smart enough to note the location of each drive by serial number when I built the server this weekend. I had taken a 7th 2TB drive from my desktop that was working fine and rarely used to increase the array size. So happens this is the drive that appears to have failed...
Before shutting down the server and pulling the drive I did some searching and tried to get some info from smartctl.
Code:
[root@freenas8] ~# smartctl -a /dev/da6 smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p4 amd64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org Smartctl open device: /dev/da6 failed: INQUIRY failed [root@freenas8] ~#
Hmm no such luck... when I rebuilt the FreeNAS server I used the beloved IBM M1015 which I flashed to IT mode (all the 2TB disks are on this controller, the others are directly on the ASUS motherboard). Had a hunch I might need to specify the type (-d option) in smartctl. After some google fu it appeared others using the card with SATA drives had it working with "smartctl -d sat"
Code:
[root@freenas8] ~# smartctl -a -d sat /dev/da6 smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p4 amd64] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org Read Device Identity failed: scsi error medium or hardware error (serious) A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. [root@freenas8] ~#
No luck there either...
Checked the serial of the drive it was just about to expire under warranty, following the mfg guide for RMAing a drive I ran seagates recommended drive test. Which passed.... ran it twice even...
Could someone help me understand FreeNAS kicking back a FAULTED disk that isn't DEFECTIVE and replaceable according to the mfg? I understand hard drives for the most part, they don't usually just "fail" its my understanding that an unreasonable amount of the disk becomes unusable / unreadable / unwritable. Is FreeNAs / ZFS just being super sensitive in this case? Is there anything that can be done to test the disk in Freenas? I haven't had to look this close at a hard drive since windows XP days. I seem to recall utilities that would "repair" the drive, if I remember correctly they would search find the bad sectors mark them "DO NOT USE" or something to that effect?
Any assistance would be greatly appreciated!
