Disk removed from zpool. Drive or controller?

Pierremenard · Oct 25, 2013

Hello, I have raidz1 zpool on my freenas 9.1.1 release.

Since yesterday one of the disk of the zpool go "removed".

Can someone help me to understand which is faulty, the disk or the controller?

This is the output of zpool

Code:

  pool: Funes
state: DEGRADED
status: One or more devices has been removed by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
  scan: resilvered 2.84M in 0h1m with 0 errors on Fri Oct 25 13:11:58 2013
config:
 
    NAME                                            STATE    READ WRITE CKSUM
    Funes                                          DEGRADED    0    0    0
      raidz1-0                                      DEGRADED    0    0    0
        7403170072441323061                        REMOVED      0    0    0  was /dev/gptid/90f1a311-fce1-11e2-9d79-00138f609fb1
        gptid/cc0ad478-b0e3-11e1-afff-00138f609fb1  ONLINE      0    0    0
        ada2p2                                      ONLINE      0    0    0
        gptid/46587c8f-18b7-11e3-abb2-00138f609fb1  ONLINE      0    0    0
 
errors: No known data errors

This is the output of gpart list

Code:

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 461dd016-18b7-11e3-abb2-00138f609fb1
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada1p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
  rawuuid: 46587c8f-18b7-11e3-abb2-00138f609fb1
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada1
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r2w2e5
 
Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: cc7533f7-b0e3-11e1-afff-00138f609fb1
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada2p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e1
  rawuuid: cc7dfa45-b0e3-11e1-afff-00138f609fb1
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada2
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Mode: r2w2e4
 
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: cbecd89e-b0e3-11e1-afff-00138f609fb1
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada3p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
  rawuuid: cc0ad478-b0e3-11e1-afff-00138f609fb1
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada3
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r2w2e5
 
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 31266815
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 32256
  Mode: r1w0e1
  attrib: active
  rawtype: 165
  length: 988291584
  offset: 32256
  type: freebsd
  index: 1
  end: 1930319
  start: 63
2. Name: da0s2
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 988356096
  Mode: r0w0e0
  rawtype: 165
  length: 988291584
  offset: 988356096
  type: freebsd
  index: 2
  end: 3860639
  start: 1930383
3. Name: da0s3
  Mediasize: 1548288 (1.5M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1976647680
  Mode: r0w0e0
  rawtype: 165
  length: 1548288
  offset: 1976647680
  type: freebsd
  index: 3
  end: 3863663
  start: 3860640
4. Name: da0s4
  Mediasize: 21159936 (20M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1978195968
  Mode: r1w1e2
  rawtype: 165
  length: 21159936
  offset: 1978195968
  type: freebsd
  index: 4
  end: 3904991
  start: 3863664
Consumers:
1. Name: da0
  Mediasize: 16008609792 (14G)
  Sectorsize: 512
  Mode: r2w1e4
 
Geom name: da0s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s1a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r1w0e1
  rawtype: 0
  length: 988283392
  offset: 8192
  type: !0
  index: 1
  end: 1930256
  start: 16
Consumers:
1. Name: da0s1
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 32256
  Mode: r1w0e1
 
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 90e470dd-fce1-11e2-9d79-00138f609fb1
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada0p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 90f1a311-fce1-11e2-9d79-00138f609fb1
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada0
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0

This is the content of dev directory (from where I see that the drive is still available)

Code:

./
../
acpi
ad10@
ad10p1@
ad10p1.eli@
ad10p2@
ad4@
ad4p1@
ad4p2@
ad6@
ad6p1@
ad6p1.eli@
ad6p2@
ad8@
ad8p1@
ad8p1.eli@
ad8p2@
ada0
ada0p1
ada0p2
ada1
ada1p1
ada1p1.eli
ada1p2
ada2
ada2p1
ada2p1.eli
ada2p2
ada3
ada3p1
ada3p1.eli
ada3p2
atkbd0
audit
bpf
bpf0@
console
consolectl
crypto
ctty
cuaU0
cuaU0.init
cuaU0.lock
cuau0
cuau0.init
cuau0.lock
da0
da0s1
da0s1a
da0s2
da0s3
da0s4
devctl
devstat
dtrace/
dumpdev@
fd/
fido
geom.ctl
ggctl
gptid/
io
kbd0@
kbd1@
kbdmux0
klog
kmem
log@
lpt0
lpt0.ctl
md0
md1
md2
mdctl
mem
nfslock
null
pass0
pass1
pass2
pass3
pass4
pci
ptmx
pts/
random
snp
stderr@
stdin@
stdout@
sysmouse
ttyU0
ttyU0.init
ttyU0.lock
ttyu0
ttyu0.init
ttyu0.lock
ttyv0
ttyv1
ttyv2
ttyv3
ttyv4
ttyv5
ttyv6
ttyv7
ttyv8
ttyv9
ttyva
ttyvb
ttyvc
ttyvd
ttyve
ttyvf
ufs/
ufssuspend
ugen0.1@
ugen0.2@
ugen1.1@
ugen1.2@
urandom@
usb/
usbctl
xpt0
zero
zfs

and lastly this is the output of dmesg

Code:

ada2: Previously was known as ad8
ada3 at ata5 bus 0 scbus3 target 0 lun 0
ada3: <ST1000DM003-9YN162 CC4B> ATA-8 SATA 3.x device
ada3: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
SMP: AP CPU #1 Launched!
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus1
Root mount waiting for: usbus1
Root mount waiting for: usbus1
uhub1: 8 ports with 8 removable, self powered
Root mount waiting for: usbus1
Root mount waiting for: usbus1
ugen1.2: <SanDisk> at usbus1
umass0: <SanDisk Cruzer, class 0/0, rev 2.00/1.02, addr 2> on usbus1
da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
da0: <SanDisk Cruzer 1.10> Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 15267MB (31266816 512 byte sectors: 255H 63S/T 1946C)
da0: quirks=0x2<NO_6_BYTE>
Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
ugen0.2: <Prolific Technology Inc.> at usbus0
uplcom0: <Prolific Technology Inc. USB-Serial Controller, class 0/0, rev 1.10/3.00, addr 2> on usbus0
(aprobe0:ata2:0:0:0): SETFEATURES SET TRANSFER MODE. ACB: ef 03 00 00 00 40 00 00 00 00 45 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_RAID5: Module loaded, version 1.1.20110927.40 (rev 00ce00e5abb4)
(aprobe0:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
(ada0:ata2:0:0:0): READ_DMA. ACB: c8 00 a0 02 40 40 00 00 00 00 e0 00
(ada0:ata2:0:0:0): CAM status: ATA Status Error
(ada0:ata2:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada0:ata2:0:0:0): RES: 51 40 a0 02 40 00 00 00 00 cf 00
(ada0:ata2:0:0:0): Retrying command
----cut same error multiple time----
(ada0:ata2:0:0:0): RES: 51 40 b8 02 40 00 00 00 00 bf 00
(ada0:ata2:0:0:0): Retrying command
(ada0:ata2:0:0:0): READ_DMA. ACB: c8 00 a0 02 40 40 00 00 00 00 e0 00
(ada0:ata2:0:0:0): CAM status: ATA Status Error
(ada0:ata2:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada0:ata2:0:0:0): RES: 51 40 b8 02 40 00 00 00 00 bf 00
(ada0:ata2:0:0:0): Retrying command
(ada0:ata2:0:0:0): READ_DMA. ACB: c8 00 a0 02 40 40 00 00 00 00 e0 00
(ada0:ata2:0:0:0): CAM status: ATA Status Error
(ada0:ata2:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada0:ata2:0:0:0): RES: 51 40 b8 02 40 00 00 00 00 bf 00
(ada0:ata2:0:0:0): Error 5, Retries exhausted
Solaris: WARNING: Disk, '/dev/gptid/90f1a311-fce1-11e2-9d79-00138f609fb1', has a block alignment that is larger than the pool's alignment
 
bridge0: Ethernet address: 02:81:f4:ec:68:00
bridge0: link state changed to UP
nfe0: promiscuous mode enabled
epair0a: Ethernet address: 02:71:9e:00:07:0a
epair0b: Ethernet address: 02:71:9e:00:08:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: promiscuous mode enabled
epair1a: Ethernet address: 02:b8:09:00:08:0a
epair1b: Ethernet address: 02:b8:09:00:09:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: promiscuous mode enabled
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
(aprobe0:ata2:0:0:0): SETFEATURES SET TRANSFER MODE. ACB: ef 03 00 00 00 40 00 00 00 00 45 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
(aprobe0:ata2:0:0:0): SETFEATURES SET TRANSFER MODE. ACB: ef 03 00 00 00 40 00 00 00 00 45 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
(aprobe0:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
(aprobe0:ata2:0:0:0): SETFEATURES SET TRANSFER MODE. ACB: ef 03 00 00 00 40 00 00 00 00 45 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
(ada0:ata2:0:0:0): lost device
(ada0:ata2:0:0:0): removing device entry
ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD10EARS-00Y5B1 80.00A80> ATA-8 SATA 2.x device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
arp: 192.168.5.99 moved from 00:13:8f:60:9f:b1 to 02:71:9e:00:07:0a on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
(ada0:ata2:0:0:0): lost device
(ada0:ata2:0:0:0): removing device entry
ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD10EARS-00Y5B1 80.00A80> ATA-8 SATA 2.x device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes)
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 00:13:8f:60:9f:b1 to 02:71:9e:00:07:0a on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 00:13:8f:60:9f:b1 to 02:71:9e:00:07:0a on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 00:13:8f:60:9f:b1 to 02:71:9e:00:07:0a on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
(aprobe0:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
(aprobe0:ata2:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ata2:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ata2:0:0:0): Error 5, Retry was blocked
arp: 192.168.5.99 moved from 00:13:8f:60:9f:b1 to 02:71:9e:00:07:0a on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b
arp: 192.168.5.99 moved from 00:13:8f:60:9f:b1 to 02:71:9e:00:07:0a on epair0b
arp: 192.168.5.99 moved from 02:71:9e:00:07:0a to 00:13:8f:60:9f:b1 on epair0b

Thank you very much in advance

F.

cyberjock · Oct 25, 2013

ada0 looks like its having errors based on your dmesg output. But the controller can be the cause as it is in the path between your computer and the disk. More than likely its the disk though.

And since you have gpt ids and adaXpX mixed together someone hasn't followed the FreeNAS manual when doing disk replacements either....

And, you have less than 4GB of RAM where the recommendation is 8GB minimum with ZFS.

Pierremenard · Oct 26, 2013

4 gigs of RAM: sometimes you can't allocate to your project the money they deserve. For more RAM I should dump the mobo & processor. But in the end the system is quite stable anyhow.

For the gpt ids, that is strange. I only replaced 1 disk following manual and at the beginning they were correctly ada0p2 ada1p2 ada2p2 ada3p2. Than my controller broke up, I exported the pool and when I imported with the new controller the gpt ids came up instead of adaxpx. But I don't think it's a major problem, except usability when I'll have to replace disks, I think.

So your suggestion is to buy a new disk? Do you need other report?

Thank you very much

F.

cyberjock · Oct 26, 2013

Pierremenard said:
4 gigs of RAM: sometimes you can't allocate to your project the money they deserve. For more RAM I should dump the mobo & processor. But in the end the system is quite stable anyhow.

And if your pool suddenly disappears into the black abyss(which has happened to people) don't expect much sympathy. The reason why we made it 8GB is because it has been a problem for people. It's your own risk and your own peril. I'll warn you that the cost of data recovery is 5-digits.. to start. So suddenly having that 8GB of RAM is cheap. I don't get why people take Microsoft at their word when they say 2GB of RAM minimum, but then sit here and argue with me that "we don't get the money".

Pierremenard said:
So your suggestion is to buy a new disk? Do you need other report?

It's probably the disk, I'd replace the disk unless you have reason to replace the controller. If you think its the controller then replace the controller too. There's stickies around here with advice on what controller to get and such, so you can do your own research and shopping in that regard.

Pierremenard · Oct 31, 2013

[For who cares]

It was the controller.

cyberjock · Oct 31, 2013

Pierremenard said:
[For who cares]

It was the controller.

Better than being the disk(s). :)

Glad you got it worked out.

Pierremenard · Oct 31, 2013

Of course I noticed it after changing the disk and 15% of resilvering of the new one. Then the controller broke up definetively.

Important Announcement for the TrueNAS Community.

Disk removed from zpool. Drive or controller?

Pierremenard

Dabbler

cyberjock

Inactive Account

Pierremenard

Dabbler

cyberjock

Inactive Account

Pierremenard

Dabbler

cyberjock

Inactive Account

Pierremenard

Dabbler

Similar threads