Replaced drive shows Unavail, unable to offline/detach

[davidmoore]

I have an 8 slot NAS tower that is full. I connected a 9th drive to the motherboard and performed a "Replace" on one of my existing 3TB drives and replaced it with a new 8TB drive. After the re-silver process finished I powered off the server and removed the 3TB drive and installed the 8TB in its place, so I'm back to 8 drives total. When I booted back up, my zpool shows a degraded state because the 3TB drive is unavailable. I've attempted to offline the drive and to detach it to no avail. What am I missing?


root@Mimir:~ # zpool status -v
pool: MIMIR
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 0 days 11:11:00 with 1 errors on Sun Nov 18 13:11:02 2018
config:

NAME STATE READ WRITE CKSUM
MIMIR DEGRADED 0 0 2
raidz1-0 ONLINE 0 0 0
gptid/2d60f534-5ace-11e7-a52b-d05099c120dd ONLINE 0 0 0
gptid/2e4d66ba-5ace-11e7-a52b-d05099c120dd ONLINE 0 0 0
gptid/2f3613f9-5ace-11e7-a52b-d05099c120dd ONLINE 0 0 0
gptid/301ee7d4-5ace-11e7-a52b-d05099c120dd ONLINE 0 0 0
raidz1-1 DEGRADED 0 0 4
replacing-0 DEGRADED 0 0 0
9334307204256484179 UNAVAIL 0 0 0 was /dev/gptid/ed97c1ef-63ee-11e7-8e83-d05099c120dd
gptid/3383412f-decb-11e8-9bfc-d05099c3a2c2 ONLINE 0 0 0
gptid/ef124f72-63ee-11e7-8e83-d05099c120dd ONLINE 0 0 0
gptid/f075eb88-63ee-11e7-8e83-d05099c120dd ONLINE 0 0 0
gptid/0415a6fd-06d5-11e8-93bf-d05099c120dd ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

MIMIR:<0x3bdfba>

 

[davidmoore]

Output when attempting to detach or offline.

root@Mimir:~ # zpool detach MIMIR 9334307204256484179
cannot detach 9334307204256484179: no valid replicas

root@Mimir:~ # zpool offline MIMIR 9334307204256484179
cannot offline 9334307204256484179: no valid replicas

 

[davidmoore]

Since I have 4x 5tb drives, 2x 3tb, and 2x 8tb, I decided to go a different route with my storage. I have purchased 2x 8tb external drives. I'm going to backup my data onto those and start over. Instead of having multiple Z1 pools, I'm going to create mirrored pairs. With my current drives, that will give me 21TB of space. Each pair will have that redundancy and then I'll stripe across all the pairs. This will allow for different size drives and give me better IOPS. This isn't really a resolution to my problem, but it's a better route than what I was doing.