Cant offline disk.

[Kayman]

Hi.

Over the last 2 months or so I've been going through the process of expanding my pool by replacing all the drives. I've gone through more than half the drives now without issue.

I don't have hot swap or spare bays in the case so I offline a drive shutdown remove and replace with new then power up and resilver. I'm replacing the drives with a mix of 2 & 3 TB WD Reds and Greens. At first I was just replacing disks at random but later decided I would like to keep all the bigger 3 Tb drives in the same Vdev so I could expand the pool again easily later on.

Now I'm trying to offline one of the 3TB disks that has recently been resilvered into the pool so I can replace it with a 2 Tb and then later add it back in again to the other vdev with the other 3 TB drives.

Now here's what happens.

Code:

Apr 14 19:09:21 freenas notifier: geli: No such device: /dev/da5p1.
Apr 14 19:09:21 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Disk offline failed: "cannot offline gptid/c871fbe2-d716-11e4-90be-0cc47a00428b: no such device in pool, "]
Apr 14 19:09:21 freenas GEOM_ELI: Device da5p1.eli destroyed.
Apr 14 19:09:21 freenas GEOM_ELI: Detached da5p1.eli on last close.

Even after multiple attempts I get the same message. The pool remains healthy and I even did a scrub to confirm everything is healthy.

I'm very reluctant to just pull the plug on the disk because my backup isn't complete yet. I'm using the retired disks to back everything up to. Maybe not the best idea but still better then nothing. Can anyone shed some light as to why this is happening and what I can do. The pool is almost full and I really need the extra capacity. I've got all the disks just waiting.

System specs are as follows.

FreeNAS 9.3
Xeon E3-1220v2
Supermicro X9SCM-F
32 Gb ECC Ram
Dual Lsi 9211-8i

Pool Configuration.

Code:

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Main 27.2T 21.4T 5.82T - 2% 78% 1.00x ONLINE /mnt
freenas-boot 7.44G 943M 6.52G - - 12% 1.00x ONLINE -

[root@freenas ~]# zpool status
pool: Main
state: ONLINE
scan: scrub repaired 0 in 10h12m with 0 errors on Tue Apr 14 18:44:18 2015
config:

NAME STATE READ WRITE CKSUM
Main ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b71dbc93-3f06-11e4-b93c-00237dfbe0f0 ONLINE 0 0 0
gptid/b79b8b8b-3f06-11e4-b93c-00237dfbe0f0 ONLINE 0 0 0
gptid/b875b15e-3f06-11e4-b93c-00237dfbe0f0 ONLINE 0 0 0
gptid/219366ec-4543-11e4-93fc-00237dfbe0f0 ONLINE 0 0 0
gptid/b95234f5-3f06-11e4-b93c-00237dfbe0f0 ONLINE 0 0 0
gptid/c871fbe2-d716-11e4-90be-0cc47a00428b ONLINE 0 0 0
gptid/ba703018-3f06-11e4-b93c-00237dfbe0f0 ONLINE 0 0 0
gptid/f09f246a-4936-11e4-86e8-00237dfbe0f0 ONLINE 0 0 0
gptid/57474628-d2c9-11e4-b74d-0cc47a00428b ONLINE 0 0 0
gptid/16a216da-ceee-11e4-8764-0cc47a00428b ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/c9acc021-d152-11e4-b719-0cc47a00428b ONLINE 0 0 0
gptid/ce3bc243-80c4-11e4-8289-0cc47a00428b ONLINE 0 0 0
gptid/4f22dafc-d934-11e4-92a2-0cc47a00428b ONLINE 0 0 0
gptid/a4de6b6c-3f07-11e4-b93c-00237dfbe0f0 ONLINE 0 0 0
gptid/50f8fb19-d6ad-11e4-a2b4-0cc47a00428b ONLINE 0 0 0
gptid/99e8bdb9-dd20-11e4-b830-0cc47a00428b ONLINE 0 0 0
gptid/6fac4876-be3b-11e4-945d-0cc47a00428b ONLINE 0 0 0
gptid/70b609bf-ddd9-11e4-b4f5-0cc47a00428b ONLINE 0 0 0
gptid/15b3c501-d39f-11e4-9aee-0cc47a00428b ONLINE 0 0 0
gptid/cf505d11-d458-11e4-a3c6-0cc47a00428b ONLINE 0 0 0

errors: No known data errors

 

[danb35]

The GPTid mentioned in the middleware error doesn't seem to appear in the output of zpool status, so maybe the GUI has gotten confused about what's in the pool and what isn't. Exporting and then re-importing the pool from the GUI should straighten that out. Of course, don't check the box for "mark disks as new" when you export.

 

[Kayman]

I just detached the pool then re imported it and the same thing happens.

Code:

Apr 14 21:55:36 freenas notifier: geli: No such device: /dev/da5p1.
Apr 14 21:55:36 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Disk offline failed: "cannot offline gptid/c871fbe2-d716-11e4-90be-0cc47a00428b: no such device in pool, "]
Apr 14 21:55:37 freenas GEOM_ELI: Device da5p1.eli destroyed.
Apr 14 21:55:37 freenas GEOM_ELI: Detached da5p1.eli on last close.

Here's some extra info.

Code:

[root@freenas ~]# camcontrol devlist
<ATA WDC WD15EARS-00M AB51> at scbus0 target 0 lun 0 (pass0,da0)
<ATA WDC WD15EARS-00Z 0A80> at scbus0 target 3 lun 0 (pass1,da1)
<ATA WDC WD15EADS-00S 5G04> at scbus0 target 9 lun 0 (pass2,da2)
<ATA WDC WD20EARX-00P AB51> at scbus0 target 15 lun 0 (pass3,da3)
<ATA WDC WD20EARS-00M AB51> at scbus0 target 16 lun 0 (pass4,da4)
<ATA WDC WD30EFRX-68A 0A80> at scbus0 target 17 lun 0 (pass5,da5)
<ATA WDC WD30EFRX-68A 0A80> at scbus0 target 18 lun 0 (pass6,da6)
<ATA WDC WD30EZRX-00D 0A80> at scbus0 target 19 lun 0 (pass7,da7)
<ATA WDC WD15EADS-00R 0A01> at scbus1 target 8 lun 0 (pass8,da8)
<ATA WDC WD20EARX-00P AB51> at scbus1 target 20 lun 0 (pass9,da9)
<ATA WDC WD20EARS-00M AB51> at scbus1 target 21 lun 0 (pass10,da10)
<ATA WDC WD20EADS-00S 0A01> at scbus1 target 22 lun 0 (pass11,da11)
<ATA WDC WD20EFRX-68A 0A80> at scbus1 target 23 lun 0 (pass12,da12)
<ATA WDC WD20EARS-00S 0A80> at scbus1 target 24 lun 0 (pass13,da13)
<ATA WDC WD30EFRX-68A 0A80> at scbus1 target 25 lun 0 (pass14,da14)
<ATA WDC WD30EZRX-00D 0A80> at scbus1 target 26 lun 0 (pass15,da15)
<WDC WD15EARS-00J2GB0 80.00A80> at scbus2 target 0 lun 0 (pass16,ada0)
<WDC WD20EARX-00PASB0 51.0AB51> at scbus3 target 0 lun 0 (pass17,ada1)
<WDC WD20EARX-00PASB0 51.0AB51> at scbus4 target 0 lun 0 (pass18,ada2)
<WDC WD15EADS-00P8B0 01.00A01> at scbus5 target 0 lun 0 (pass19,ada3)
<SanDisk Cruzer Blade 1.26> at scbus9 target 0 lun 0 (da16,pass20)
[root@freenas ~]# glabel status
Name Status Components
gptid/b71dbc93-3f06-11e4-b93c-00237dfbe0f0 N/A da0p2
gptid/b875b15e-3f06-11e4-b93c-00237dfbe0f0 N/A da1p2
gptid/b95234f5-3f06-11e4-b93c-00237dfbe0f0 N/A da2p2
gptid/f09f246a-4936-11e4-86e8-00237dfbe0f0 N/A da3p2
gptid/cf505d11-d458-11e4-a3c6-0cc47a00428b N/A da4p2
gptid/c871fbe2-d716-11e4-90be-0cc47a00428b N/A da5p2
gptid/4f22dafc-d934-11e4-92a2-0cc47a00428b N/A da6p2
gptid/99e8bdb9-dd20-11e4-b830-0cc47a00428b N/A da7p2
gptid/b79b8b8b-3f06-11e4-b93c-00237dfbe0f0 N/A da8p2
gptid/219366ec-4543-11e4-93fc-00237dfbe0f0 N/A da9p2
gptid/ce3bc243-80c4-11e4-8289-0cc47a00428b N/A da10p2
gptid/6fac4876-be3b-11e4-945d-0cc47a00428b N/A da11p2
gptid/57474628-d2c9-11e4-b74d-0cc47a00428b N/A da12p2
gptid/15b3c501-d39f-11e4-9aee-0cc47a00428b N/A da13p2
gptid/50f8fb19-d6ad-11e4-a2b4-0cc47a00428b N/A da14p2
gptid/70b609bf-ddd9-11e4-b4f5-0cc47a00428b N/A da15p2
gptid/a4de6b6c-3f07-11e4-b93c-00237dfbe0f0 N/A ada0p2
gptid/16a216da-ceee-11e4-8764-0cc47a00428b N/A ada1p2
gptid/c9acc021-d152-11e4-b719-0cc47a00428b N/A ada2p2
gptid/ba703018-3f06-11e4-b93c-00237dfbe0f0 N/A ada3p2
gptid/3d541e3b-82a1-11e4-82de-0cc47a00428b N/A da16p1
gptid/3d5e7e5a-82a1-11e4-82de-0cc47a00428b N/A da16p2
gptid/c85c7bca-d716-11e4-90be-0cc47a00428b N/A da5p1
[root@freenas ~]#

 

[danb35]

Looking more closely at your zpool status, I just overlooked the line that was the gptid in question. So, yes, it is in the pool, but the middleware is still erroring out. Strange.

 

[SweetAndLow]

Please post the debug file that is downloadable from the GUI.

 

[Kayman]

Please post the debug file that is downloadable from the GUI.

 

[cyberjock]

I'd submit a bug ticket, include your system specs and the error you are getting along with the debug file.