Disk Replacement

Adam Randazzo · Sep 21, 2012

Hi All -

I have a FreeNAS-8.2.0-RELEASE-p1-x64 (r11950) system that recently had a drive go bad in a RaidZ1 configuration.

Once I determined the bad drive, I offlined it using the GUI. This evening, I replaced the drive in the physical location of the bad drive. I rebooted the machine, and attempted to replace the drive with the GUI. When I did that, it gave me an error, and the following printout in

Code:

Sep 21 22:45:51 freenas notifier: swapoff: /dev/ada4p1: Invalid argument
Sep 21 22:45:52 freenas notifier: 1+0 records in
Sep 21 22:45:52 freenas notifier: 1+0 records out
Sep 21 22:45:52 freenas notifier: 1048576 bytes transferred in 0.011320 secs (92629455 bytes/sec)
Sep 21 22:45:52 freenas notifier: dd: /dev/ada4: short write on character device
Sep 21 22:45:52 freenas notifier: dd: /dev/ada4: end of device
Sep 21 22:45:52 freenas notifier: 5+0 records in
Sep 21 22:45:52 freenas notifier: 4+1 records out
Sep 21 22:45:52 freenas notifier: 4939776 bytes transferred in 0.110677 secs (44632361 bytes/sec)
Sep 21 22:45:53 freenas notifier: cannot online ada4p2: no such device in pool
Sep 21 22:45:53 freenas notifier: swapon: /dev/ada4p1: device already in use

Here is what my zpool status is showing:

Code:

NAME                                            STATE     READ WRITE CKSUM
	zfs_pool                                        DEGRADED     0     0     0
	  raidz1                                        DEGRADED     0     0     0
	    ada1                                        ONLINE       0     0     0
	    ada3                                        ONLINE       0     0     0
	    ada4                                        OFFLINE      0     0     0
	  raidz1                                        ONLINE       0     0     0
	    gptid/7bff2d82-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0
	    gptid/7c5565ab-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0
	    gptid/7cade619-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0

I have attempted to replace the drive via command line as well, but none of these worked:

[root@freenas] ~# zpool replace zfs_pool ada4
cannot replace ada4 with ada4: no such pool or dataset

[root@freenas] ~# zpool replace zfs_pool /dev/ada4
cannot replace /dev/ada4 with /dev/ada4: no such pool or dataset

[root@freenas] ~# zpool replace zfs_pool /dev/ada4 /dev/ada4
cannot replace /dev/ada4 with /dev/ada4: no such pool or dataset

Any help on where I can go from here would be greatly appreciated as I am stuck on where to go from here.

-Adam

paleoN · Sep 21, 2012

paleoN said:
From a SSH session as root paste the output of:
Code:
zpool status -v camcontrol devlist gpart show glabel status
Throw the output inside of some [code][/code] tags as it will preserve the formatting and keep my eyes from crossing.

Is the new drive the same size or larger than the failed one?

Adam Randazzo · Sep 21, 2012

Sorry -- I missed your quoted instructions above.. here they are..

zpool status -v

Code:

pool: zfs_pool
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
 scrub: scrub in progress for 0h45m, 17.28% done, 3h38m to go
config:

	NAME                                            STATE     READ WRITE CKSUM
	zfs_pool                                        DEGRADED     0     0     0
	  raidz1                                        DEGRADED     0     0     0
	    ada1                                        ONLINE       0     0     0
	    ada3                                        ONLINE       0     0     0
	    ada4                                        OFFLINE      0     0     0
	  raidz1                                        ONLINE       0     0     0
	    gptid/7bff2d82-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0
	    gptid/7c5565ab-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0
	    gptid/7cade619-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0

camcontrol devlist

Code:

<TS2GCF133 20100804>               at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD10EARS-00Y5B1 80.00A80>     at scbus2 target 0 lun 0 (pass1,ada1)
<ST9500420ASG 0008APM2>            at scbus3 target 0 lun 0 (pass2,ada2)
<WDC WD10EARS-00Y5B1 80.00A80>     at scbus4 target 0 lun 0 (pass3,ada3)
<WDC WD10EARX-00N0YB0 51.0AB51>    at scbus5 target 0 lun 0 (pass4,ada4)
<WDC WD3200AAKS-00L9A0 01.03E01>   at scbus6 target 0 lun 0 (pass5,ada5)
<WDC WD3200AAKS-00L9A0 01.03E01>   at scbus8 target 0 lun 0 (pass6,ada6)
<WDC WD3200AAKS-00L9A0 01.03E01>   at scbus9 target 0 lun 0 (pass7,ada7)

gpart show

Code:

=>     63  3915009  ada0  MBR  (1.9G)
       63  1930257     1  freebsd  (943M)
  1930320       63        - free -  (32K)
  1930383  1930257     2  freebsd  [active]  (943M)
  3860640     3024     3  freebsd  (1.5M)
  3863664    41328     4  freebsd  (20M)
  3904992    10080        - free -  (4.9M)

=>      0  1930257  ada0s1  BSD  (943M)
        0       16          - free -  (8.0K)
       16  1930241       1  !0  (943M)

=>      0  1930257  ada0s2  BSD  (943M)
        0       16          - free -  (8.0K)
       16  1930241       1  !0  (943M)

=>       34  976773101  ada2  GPT  (466G)
         34         94        - free -  (47K)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  972578703     2  freebsd-zfs  (464G)

=>       34  625142381  ada5  GPT  (298G)
         34         94        - free -  (47K)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  620947983     2  freebsd-zfs  (296G)

=>       34  625142381  ada6  GPT  (298G)
         34         94        - free -  (47K)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  620947983     2  freebsd-zfs  (296G)

=>       34  625142381  ada7  GPT  (298G)
         34         94        - free -  (47K)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  620947983     2  freebsd-zfs  (296G)

=>        34  1953525101  ada4  GPT  (932G)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  1949330703     2  freebsd-zfs  (930G)

glabel status

Code:

                                      Name  Status  Components
                             ufs/FreeNASs3     N/A  ada0s3
                             ufs/FreeNASs4     N/A  ada0s4
                    ufsid/4e1dfc44cf1049f1     N/A  ada0s1a
                            ufs/FreeNASs1a     N/A  ada0s1a
                            ufs/FreeNASs2a     N/A  ada0s2a
gptid/cc8a56c4-0111-11e2-8454-6805ca0bf42d     N/A  ada2p2
gptid/7bff2d82-0111-11e2-8454-6805ca0bf42d     N/A  ada5p2
gptid/7c5565ab-0111-11e2-8454-6805ca0bf42d     N/A  ada6p2
gptid/7cade619-0111-11e2-8454-6805ca0bf42d     N/A  ada7p2
gptid/ad461729-0465-11e2-8893-6805ca0bf42d     N/A  ada4p2

Adam Randazzo · Sep 21, 2012

One other point of history to add. The 3 WD10EARS drives were originally part of a FreeNAS 7. I upgraded to 8 a while back, and updated my zfs pool version. I added the WD3200AAKS drives at a later date.

What I don't understand is why gpart and glabel don't show ada1 and ada3, which are the other members of my zfs_pool.

paleoN · Sep 21, 2012

First, stop the scrub:

Code:

zpool scrub -s zfs_pool

OK, I take it the first raidz1 vdev was created in an older version of FreeNAS? That vdev appears to be using whole disks.

Wipe ada4, in the GUI under View Disks
Code:
```
zpool online zfs_pool ada4
```
Code:
```
zpool replace zfs_pool ada4
```

Don't forget to stop the scrub first.

Adam Randazzo · Sep 21, 2012

Ok.. did a stop on the scrub.. here is the output from the other commands:

Ok.. I could not wipe ada4 through the GUI. I received this error:

Code:

Umount the following mount points before proceeding:
zfs_pool

# zpool online zfs_pool ada4

Code:

warning: device 'ada4' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present

# zpool replace zfs_pool ada4

Code:

cannot replace ada4 with ada4: no such pool or dataset

paleoN · Sep 21, 2012

Export the pool first and then wipe ada4. Import the pool and post:

Code:

zpool status -v

Adam Randazzo · Sep 21, 2012

I can't export the pool because of the plugin jail. I did run a "service stop jail"..

Code:

service jail stop
Stopping jails: jail192.168.2.230 (192.168.2.230) deleted

but I am getting the following:

Code:

#zpool export zfs_pool
cannot unmount '/mnt/zfs_pool/jail': Device busy

paleoN · Sep 21, 2012

Try shutting down the jail from the GUI. I don't know that is the correct CLI command to use to stop the jail. If the GUI doesn't work then reboot and then shut it down and export.

Adam Randazzo · Sep 21, 2012

K rebooting did it.. only thing left is my timemachine, which is on another physical disk..

Code:

[root@freenas] ~# zpool status -v
  pool: timemachines
 state: ONLINE
 scrub: none requested
config:

	NAME                                          STATE     READ WRITE CKSUM
	timemachines                                  ONLINE       0     0     0
	  gptid/cc8a56c4-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0

errors: No known data errors

paleoN · Sep 21, 2012

Wipe ada4, in the GUI under View Disks
import the pool via GUI auto-import
Code:
```
zpool online zfs_pool ada4
```
Code:
```
zpool replace zfs_pool ada4
```

Adam Randazzo · Sep 21, 2012

Auto Import Error

Code:

You already have a volume with same name

paleoN · Sep 21, 2012

Did ada4 get wiped? If it did then:

Code:
```
zpool import zfs_pool
```
Code:
```
zpool online zfs_pool ada4
```
Code:
```
zpool replace zfs_pool ada4
```

Otherwise you have to deal with wiping the disk first.

Adam Randazzo · Sep 21, 2012

Ok.. since the auto import didn't work, I was able to run a zpool import via command line..

that seemed to work, except it didn't mount my disks?!

Then, I was able to do a replace.. and it is now reslivering

Code:

pool: zfs_pool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 0.14% done, 7h5m to go
config:

	NAME                                            STATE     READ WRITE CKSUM
	zfs_pool                                        DEGRADED     0     0     0
	  raidz1                                        DEGRADED     0     0     0
	    ada1                                        ONLINE       0     0     0
	    ada3                                        ONLINE       0     0     0
	    replacing                                   DEGRADED     0     0     0
	      ada4/old                                  OFFLINE      0     0     0
	      ada4                                      ONLINE       0     0     0  1.19G resilvered
	  raidz1                                        ONLINE       0     0     0
	    gptid/7bff2d82-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0
	    gptid/7c5565ab-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0
	    gptid/7cade619-0111-11e2-8454-6805ca0bf42d  ONLINE       0     0     0

What am I going to have to do to get the pool remounted?

Adam Randazzo · Sep 21, 2012

Ok.. I think the only issue left is that when I did zfs_pool import, I was in / when I should have been in /mnt.

Can I export it, and then reimport it safely while it's reslivering?

paleoN · Sep 21, 2012

The above zpool import tries to mount at '/' instead of '/mnt'. I would save your FreeNAS config now if you don't have a recent version.

Once the resilver completes I would try a reboot. If that doesn't work then try:

Code:

zpool export zfs_pool

Followed with a reboot and try to auto import again.

paleoN · Sep 21, 2012

Adam Randazzo said:
Can I export it, and then reimport it safely while it's reslivering?

No. Wait, then play with where it's mounted.

Adam Randazzo · Sep 21, 2012

Yah.. that seems like the smart idea at this point. Thank you so much for the help.. it is much appreciated.

If you like beer, shoot me your paypal address, and I'd like to buy you a six-pack ;)

Adam Randazzo · Sep 22, 2012

As of this morning, the re-sliver finished just fine. I was able to export my pool, reimport it to the correct path using the instructions here: http://docs.oracle.com/cd/E19082-01/817-2271/gbcgl/index.html

Running a scrub, and everything seems good!!

My takeaway from this is that the Replace function should probably do some detection on whether the pool was created on an older freenas. Assuming it should be partitioned with fs + swap was a major PITA to deal with.

Important Announcement for the TrueNAS Community.

Disk Replacement

Dabbler

Wizard

Dabbler

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Dabbler

Wizard

Wizard

Dabbler

Dabbler

Similar threads