Unable to detach missing disk

[micdud]

Hi,

My system was powered down and disk was replaced with new one. After boot using gui 9.10 at that time i think i did replace on missing disk and resilvering process has started. once it was done i can still see old drive as unavail. i tried to detach using gui and command line. any help is gladly appreciated.

its currently resilvering due to reboot - but it always finishes no problem


root@freenas:~ # zpool status -v
pool: BigBerta
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jul 14 05:41:29 2017
2.97T scanned out of 6.54T at 259M/s, 4h1m to go
1.32G resilvered, 45.37% done
config:

NAME STATE READ WRITE CKSUM
BigBerta DEGRADED 0 0 20
raidz1-0 DEGRADED 0 0 40
gptid/9c934a43-66a7-11e7-9b8e-000c2955a8d0 ONLINE 0 0 0 (resilvering)
replacing-1 DEGRADED 0 0 0
14039406194134957747 UNAVAIL 0 0 0 was /dev/gptid/173105e5-c951-11e5-9392-001f295ffc6d
gptid/220162e8-6771-11e7-a747-000c2955a8d0 ONLINE 0 0 0 (resilvering)
gptid/18274962-c951-11e5-9392-001f295ffc6d ONLINE 19 0 0 (resilvering)


root@freenas:~ # zpool detach BigBerta 14039406194134957747
cannot detach 14039406194134957747: no valid replicas


root@freenas:~ # zpool detach BigBerta 173105e5-c951-11e5-9392-001f295ffc6d
cannot detach 173105e5-c951-11e5-9392-001f295ffc6d: no such device in pool



root@freenas:~ # zfs upgrade -v
The following filesystem versions are supported:

VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS filesystem version
2 Enhanced directory entries
3 Case insensitive and filesystem user identifier (FUID)
4 userquota, groupquota properties
5 System attributes

For more information on a particular version, including supported releases,
see the ZFS Administration Guide.



This system supports ZFS pool feature flags.

The following features are supported:

FEAT DESCRIPTION
-------------------------------------------------------------
async_destroy (read-only compatible)
Destroy filesystems asynchronously.
empty_bpobj (read-only compatible)
Snapshots use less space.
lz4_compress
LZ4 compression algorithm support.
multi_vdev_crash_dump
Crash dumps to multiple vdev pools.
spacemap_histogram (read-only compatible)
Spacemaps maintain space histograms.
enabled_txg (read-only compatible)
Record txg at which a feature is enabled
hole_birth
Retain hole birth txg for more precise zfs send
extensible_dataset
Enhanced dataset functionality, used by other features.
embedded_data
Blocks which compress very well use even less space.
bookmarks (read-only compatible)
"zfs bookmark" command
filesystem_limits (read-only compatible)
Filesystem and snapshot limits.
large_blocks
Support for blocks larger than 128KB.
sha512
SHA-512/256 hash algorithm.
skein
Skein hash algorithm.

The following legacy versions are also supported:

VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit
15 user/group space accounting
16 stmf property support
17 Triple-parity RAID-Z
18 Snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties
23 Slim ZIL
24 System attributes
25 Improved scrub stats
26 Improved snapshot deletion performance
27 Improved snapshot creation performance
28 Multiple vdev replacements

For more information on a particular version, including supported releases,
see the ZFS Administration Guide.



 

[BigDave]

My system was powered down and disk was replaced with new one. After boot using gui 9.10 at that time i think i did replace on missing disk and resilvering process has started. once it was done i can still see old drive as unavail. i tried to detach using gui and command line. any help is gladly appreciated.

Backup your data now, you are using RAIDz1 and your data is at risk during a resilver.
Did you start the original drive replacement on version 9.10.x and have now updated to 11.0?
Please give current version of FreeNAS.
Please post entire current output (withintags) of # zpool status
List your drive's brand name, model number and capacity.
List your motherboard, cpu, RAM
Someone will help you, but more information is needed. Thank you!

 

[micdud]

alright here is full disclosure. Idea was to replace 3T disk with 8T.
System was updated to corral - initial release no other updates of corral were applied. after all that fiasco with corral i switched in gui version but never had chance to reboot. to put change in place.
with reboot for first drive replacement system was downgraded to now I'm not sure... i may have accidentally put it on 9.3 instead 9.10. first drive replacement went OK.
second drive was replaced (freenas version was not changed - 9.3 i think) - and that when im stock now. it resilvers and wont allow me to detach old drive. I do have a copy of my data. im simply trying to avoid 12h of restoring.
after 2 days of 'googling' i finally changed os to 11 hoping this may solve it... it did not.

Host is running following:
Hardware:
RAM 32G
CPU 1x E5-2667 v3 3.2Ghz
Motherboard Asus x99-m ws
Originally: 3x 3T
LSI 9207-4i4e


Now:
8T IronWolf SAN ST8000VN0022-2EL112
8T IronWolf SAN ST8000VN0022-2EL112
3T WDC WD30EZRX-00MMMB0

System:
Host is running esxi 6.5 with freenas assigned 24G of ram and LSI 9207 is in pass-through mode
freenas 11



root@freenas:~ # zpool status
pool: BigBerta
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jul 14 05:41:29 2017
3.93T scanned out of 6.54T at 247M/s, 3h4m to go
3.98G resilvered, 60.06% done
config:

NAME STATE READ WRITE CKSUM
BigBerta DEGRADED 0 0 21
raidz1-0 DEGRADED 0 0 42
gptid/9c934a43-66a7-11e7-9b8e-000c2955a8d0 ONLINE 0 0 0 (resilvering)
replacing-1 DEGRADED 0 0 0
14039406194134957747 UNAVAIL 0 0 0 was /dev/gptid/173105e5-c951-11e5-9392-001f295ffc6d
gptid/220162e8-6771-11e7-a747-000c2955a8d0 ONLINE 0 0 0 (resilvering)
gptid/18274962-c951-11e5-9392-001f295ffc6d ONLINE 20 0 0 (resilvering)

errors: 10 data errors, use '-v' for a list

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Wed Jul 12 03:45:23 2017
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors
root@freenas:~ #



 

[micdud]

Datastore for ESXi - 1T SSD Samsung Evo PRO attahced to MB controller not LSI


root@freenas:~ # sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2308_1(D1)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------

0 SAS2308_1(D1) 20.00.04.00 14.01.30.16 07.39.00.00 00:03:00:00

Finished Processing Commands Successfully.
Exiting SAS2Flash.



 

[BigDave]

gptid/18274962-c951-11e5-9392-001f295ffc6d ONLINE 20 0 0 (resilvering)

You have this drive currently listing 20 read errors during a resilver.

Since your data is safely backed up, I would (for the sake of stability) start over. :)

The use of RAIDz1 with the large capacity drives is a documented no no. :p

First you should test the hard drives with badblocks before adding them to the array.
You may ignore this, if this has been done. ;)

Reflash firmware to version 20.00.07.00 for use with FreeNAS 11.0

 

[micdud]

You have this drive currently listing 20 read errors during a resilver.

Since your data is safely backed up, I would (for the sake of stability) start over. :)

The use of RAIDz1 with the large capacity drives is a documented no no. :p

First you should test the hard drives with badblocks before adding them to the array.
You may ignore this, if this has been done. ;)

Reflash firmware to version 20.00.07.00 for use with FreeNAS 11.0

errors are on last 3T drive which i will replace.
what RAIDz is recommended for larger drives? and how many drives minimum required for that mode?

 

[micdud]

My biggest fear is to use FreeNas/zfs again - it seems like basic functionality like busted drive replacement is a 50/50 challenge. not to mention one of the releases back some time ago broke scrubing (cron path issue?) or corral...
I'm a home user - data absolutely needed is being backed up to another storage device + cloud. all im looking is to have stable large storage that in case drive dies will buy me some time to get replacement instead crashing immediately. added bonus is ability to play with technology.

 

[BigDave]

https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

 

[BigDave]

all im looking is to have stable large storage that in case drive dies will buy me some time to get replacement instead crashing immediately.

RAIDz2 allows two drive failures before data loss happens, but the minimum number of drives is 4 drives.
Five drives would be better and Six drives the sweet spot for RAIDz2

 

[danb35]

its currently resilvering due to reboot

Your array normally resilvers on reboot? This is not OK.

 

[micdud]

Your array normally resilvers on reboot? This is not OK.

its resilvers on reboot when state is degraded from what i understand

 

[rs225]

Were scrubs running on a reasonable schedule with email notice to you?

It looks like what has happened is that you already had a failing drive, and then you swapped out another drive to replace from 3TB to 8TB.

Post zpool status -v BigBerta to get an idea of the corruption. You can redact file names if you want.
Also post smartctl -a /dev/da4 or whatever device names you see in camcontrol devlist

 

[micdud]

https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

thank you - i read it all :)

 

[micdud]

Were scrubs running on a reasonable schedule with email notice to you?

It looks like what has happened is that you already had a failing drive, and then you swapped out another drive to replace from 3TB to 8TB.

Post zpool status -v BigBerta to get an idea of the corruption. You can redact file names if you want.
Also post smartctl -a /dev/da4 or whatever device names you see in camcontrol devlist

Yes - i noticed this too - scrubs were not running - last one was months ago. i did loose some files not much not important but its still doesnt explain why i cant detach disk that was already replaced and it is gone- last 3T disk is the one with bad sectors but i cant proceed til disk #2 detaches.

Code:

root@freenas:~ # zpool status -v
  pool: BigBerta
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
   continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Jul 14 05:41:29 2017
  5.71T scanned out of 6.54T at 221M/s, 1h5m to go
  4.54G resilvered, 87.28% done
config:

   NAME  STATE  READ WRITE CKSUM
   BigBerta  DEGRADED  0  0  28
	raidz1-0  DEGRADED  0  0  56
	gptid/9c934a43-66a7-11e7-9b8e-000c2955a8d0  ONLINE  0  0  0  (resilvering)
	replacing-1  DEGRADED  0  0  0
	14039406194134957747  UNAVAIL  0  0  0  was /dev/gptid/173105e5-c951-11e5-9392-001f295ffc6d
	gptid/220162e8-6771-11e7-a747-000c2955a8d0  ONLINE  0  0  0  (resilvering)
	gptid/18274962-c951-11e5-9392-001f295ffc6d  ONLINE  26  0  0  (resilvering)

errors: Permanent errors have been detected in the following files:

  /mnt/BigBerta/backup/backup_XPS.img.gz
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED
  /mnt/BigBerta/Video/REDACTED

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Wed Jul 12 03:45:23 2017
config:

   NAME  STATE  READ WRITE CKSUM
   freenas-boot  ONLINE  0  0  0
	da0p2  ONLINE  0  0  0

errors: No known data errors

Code:

root@freenas:~ # smartctl -a /dev/da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:  ST8000VN0022-2EL112
Serial Number:  XXXX
LU WWN Device Id: 5 000c50 0a359f383
Firmware Version: SC61
User Capacity:  8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  7200 rpm
Form Factor:  3.5 inches
Device is:  Not in smartctl database [for details use: -P showall]
ATA Version is:  ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Jul 14 13:15:59 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82)   Offline data collection activity
		   was completed without error.
		   Auto Offline Data Collection: Enabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  567) seconds.
Offline data collection
capabilities:		 (0x7b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  1) minutes.
Extended self-test routine
recommended polling time:	 ( 756) minutes.
Conveyance self-test routine
recommended polling time:	 (  2) minutes.
SCT capabilities:	 (0x50bd)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000f  081  064  044  Pre-fail  Always  -  140392424
  3 Spin_Up_Time  0x0003  088  088  000  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  020  Old_age  Always  -  6
  5 Reallocated_Sector_Ct  0x0033  100  100  010  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000f  071  060  045  Pre-fail  Always  -  13796154
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  65 (220 221 0)
10 Spin_Retry_Count  0x0013  100  100  097  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  020  Old_age  Always  -  6
184 End-to-End_Error  0x0032  100  100  099  Old_age  Always  -  0
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  100  000  Old_age  Always  -  0
189 High_Fly_Writes  0x003a  100  100  000  Old_age  Always  -  0
190 Airflow_Temperature_Cel 0x0022  059  038  040  Old_age  Always  In_the_past 41 (Min/Max 35/42 #120)
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  359
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  3
193 Load_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  98
194 Temperature_Celsius  0x0022  041  062  000  Old_age  Always  -  41 (0 25 0 0 0)
195 Hardware_ECC_Recovered  0x001a  004  001  000  Old_age  Always  -  140392424
197 Current_Pending_Sector  0x0012  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0010  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x003e  200  200  000  Old_age  Always  -  0
240 Head_Flying_Hours  0x0000  100  253  000  Old_age  Offline  -  62 (226 185 0)
241 Total_LBAs_Written  0x0000  100  253  000  Old_age  Offline  -  4750546796
242 Total_LBAs_Read  0x0000  100  253  000  Old_age  Offline  -  13026667174

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

root@freenas:~ # smartctl -a /dev/da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:  ST8000VN0022-2EL112
Serial Number:  XXXXX
LU WWN Device Id: 5 000c50 0a2e58263
Firmware Version: SC61
User Capacity:  8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Rotation Rate:  7200 rpm
Form Factor:  3.5 inches
Device is:  Not in smartctl database [for details use: -P showall]
ATA Version is:  ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Jul 14 13:17:11 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82)   Offline data collection activity
		   was completed without error.
		   Auto Offline Data Collection: Enabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (  567) seconds.
Offline data collection
capabilities:		 (0x7b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  1) minutes.
Extended self-test routine
recommended polling time:	 ( 743) minutes.
Conveyance self-test routine
recommended polling time:	 (  2) minutes.
SCT capabilities:	 (0x50bd)   SCT Status supported.
		   SCT Error Recovery Control supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x000f  074  065  044  Pre-fail  Always  -  28090152
  3 Spin_Up_Time  0x0003  094  094  000  Pre-fail  Always  -  0
  4 Start_Stop_Count  0x0032  100  100  020  Old_age  Always  -  3
  5 Reallocated_Sector_Ct  0x0033  100  100  010  Pre-fail  Always  -  0
  7 Seek_Error_Rate  0x000f  069  060  045  Pre-fail  Always  -  8370952
  9 Power_On_Hours  0x0032  100  100  000  Old_age  Always  -  39 (193 242 0)
10 Spin_Retry_Count  0x0013  100  100  097  Pre-fail  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  020  Old_age  Always  -  3
184 End-to-End_Error  0x0032  100  100  099  Old_age  Always  -  0
187 Reported_Uncorrect  0x0032  100  100  000  Old_age  Always  -  0
188 Command_Timeout  0x0032  100  100  000  Old_age  Always  -  0
189 High_Fly_Writes  0x003a  100  100  000  Old_age  Always  -  0
190 Airflow_Temperature_Cel 0x0022  059  037  040  Old_age  Always  In_the_past 41 (Min/Max 35/43 #164)
191 G-Sense_Error_Rate  0x0032  100  100  000  Old_age  Always  -  258
192 Power-Off_Retract_Count 0x0032  100  100  000  Old_age  Always  -  2
193 Load_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  5
194 Temperature_Celsius  0x0022  041  063  000  Old_age  Always  -  41 (0 25 0 0 0)
195 Hardware_ECC_Recovered  0x001a  004  001  000  Old_age  Always  -  28090152
197 Current_Pending_Sector  0x0012  100  100  000  Old_age  Always  -  0
198 Offline_Uncorrectable  0x0010  100  100  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x003e  200  200  000  Old_age  Always  -  0
240 Head_Flying_Hours  0x0000  100  253  000  Old_age  Offline  -  39 (76 99 0)
241 Total_LBAs_Written  0x0000  100  253  000  Old_age  Offline  -  4721081532
242 Total_LBAs_Read  0x0000  100  253  000  Old_age  Offline  -  9539488139

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

root@freenas:~ # smartctl -a /dev/da3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 11.0-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:  Western Digital Green
Device Model:  WDC WD30EZRX-00MMMB0
Serial Number:  XXXXX
LU WWN Device Id: 5 0014ee 2b17723a5
Firmware Version: 80.00A80
User Capacity:  3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:  512 bytes logical, 4096 bytes physical
Device is:  In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:  Fri Jul 14 13:17:55 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x85)   Offline data collection activity
		   was aborted by an interrupting command from host.
		   Auto Offline Data Collection: Enabled.
Self-test execution status:  (  0)   The previous self-test routine completed
		   without error or no self-test has ever
		   been run.
Total time to complete Offline
data collection:	  (52380) seconds.
Offline data collection
capabilities:		 (0x7b) SMART execute Offline immediate.
		   Auto Offline data collection on/off support.
		   Suspend Offline collection upon new
		   command.
		   Offline surface scan supported.
		   Self-test supported.
		   Conveyance Self-test supported.
		   Selective Self-test supported.
SMART capabilities:  (0x0003)   Saves SMART data before entering
		   power-saving mode.
		   Supports SMART auto save timer.
Error logging capability:  (0x01)   Error logging supported.
		   General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (  2) minutes.
Extended self-test routine
recommended polling time:	 ( 503) minutes.
Conveyance self-test routine
recommended polling time:	 (  5) minutes.
SCT capabilities:	 (0x3035)   SCT Status supported.
		   SCT Feature Control supported.
		   SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG  VALUE WORST THRESH TYPE  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate  0x002f  200  200  051  Pre-fail  Always  -  0
  3 Spin_Up_Time  0x0027  140  140  021  Pre-fail  Always  -  9966
  4 Start_Stop_Count  0x0032  100  100  000  Old_age  Always  -  675
  5 Reallocated_Sector_Ct  0x0033  200  200  140  Pre-fail  Always  -  2
  7 Seek_Error_Rate  0x002e  100  253  000  Old_age  Always  -  0
  9 Power_On_Hours  0x0032  060  060  000  Old_age  Always  -  29744
10 Spin_Retry_Count  0x0032  100  100  000  Old_age  Always  -  0
11 Calibration_Retry_Count 0x0032  100  100  000  Old_age  Always  -  0
12 Power_Cycle_Count  0x0032  100  100  000  Old_age  Always  -  420
192 Power-Off_Retract_Count 0x0032  200  200  000  Old_age  Always  -  413
193 Load_Cycle_Count  0x0032  194  194  000  Old_age  Always  -  20761
194 Temperature_Celsius  0x0022  111  101  000  Old_age  Always  -  41
196 Reallocated_Event_Count 0x0032  198  198  000  Old_age  Always  -  2
197 Current_Pending_Sector  0x0032  200  200  000  Old_age  Always  -  33
198 Offline_Uncorrectable  0x0030  200  200  000  Old_age  Offline  -  0
199 UDMA_CRC_Error_Count  0x0032  200  200  000  Old_age  Always  -  0
200 Multi_Zone_Error_Rate  0x0008  200  200  000  Old_age  Offline  -  0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description  Status  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline  Completed: read failure  90%  29694  3964038408

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
  1  0  0  Not_testing
  2  0  0  Not_testing
  3  0  0  Not_testing
  4  0  0  Not_testing
  5  0  0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[rs225]

It looks like the remaining 3TB WD is the one with problems.

It won't let you remove the drive it is currently trying to replace until the resilver is finished, and it is correct to do so. Until the resilver is finished, there is no replacement. This may also leave open the possibility of you reconnecting that disk (which I don't recommend), and repairing those damaged files with the good data from the removed disk.

 

[micdud]

this pool resilvers every time i reboot to try something different - it takes 8h... i waited at least 4 times to resilver finish and still no luck detaching. I agree with everyones comments about my negligence to maintain this zpool properly. at this point im just looking for way out of this other than use backups to restore.
i guess i know there is no way out of this - even if i manage to fix this pool i bought 4 new drives 'nas rated' and i will need to recreate my zpool as raidz2.

i dont like situation when there is a technical problem and i cant explain it.

 

[Stux]

Have you still got the other 2 3TB drives?

If so, put them back in, I guess as a 6TB stripe (no redundancy)

Then rsync your files to the 2x3TB pool.

Then make a NEW pool with your 8TB drives (at a minimum a mirror). Then replicate the 2x3TB pool to the new pool.

Hopefully neither of the 3TB drives fails in the mean time.

And if they do, you have a backup.

RaidZ1 with 8TB drives is a categorically bad idea. The chances of what has just happened to you happening to you are almost certain during a disk replacement operation.

And what has just happened, ie a drive failing as you're resilvering another... is why RaidZ1 is a bad idea these days.