One or more devices has been removed by the administrator during scrub?

Yangorang

Cadet
Joined
Sep 17, 2015
Messages
7
On FreeNAS 11.3. Lately on the last two weekly scrub days one of my drives (the same one both times) has been getting removed from the pool with the error message "state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state."

The drive has been showing some S.M.A.R.T. errors during long tests so I have a replacement ordered but if I reboot the server the pool simply shows as healthy again, albeit with 13 checksum errors on that drive. Is the drive still usable or should I just replace it now? Is FreeNAS removing the drive because of the checksum errors or is it drive somehow shutting down during the scrub process?

########## SMART status report for ada5 drive (Western Digital Red: WD-WCC4E4DCXYA1) ##########
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p14 amd64] (local build)

SMART overall-health self-assessment test result: PASSED

ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 201 177 021 Pre-fail Always - 6941
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 96
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 196 195 000 Old_age Always - 303
9 Power_On_Hours 0x0032 045 045 000 Old_age Always - 40380
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 96
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 30
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 444
194 Temperature_Celsius 0x0022 117 106 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1

No Errors Logged

Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
Extended offline Completed: read failure 80% 40375 1562099680
Short offline Completed without error 00% 40335 -
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Code:
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1


and


Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

Extended offline    Completed: read failure       80%     40375         1562099680


Both of these indicate there is indeed an error on that disk.

Rebooting or using zpool clear just resets the error counts, so evenrything looks good until that block (1562099680) is tested/read again.

Where there is one bad block, more will follow, so I would replace it if it were mine.
 

Yangorang

Cadet
Joined
Sep 17, 2015
Messages
7
Is the bad block what's causing FreeNAS to kick the disk from the pool? The disk hasn't shut down itself down or anything like that.
I can still use the drive for non-critical storage I guess maybe for my Xbox One as secondary storage or something....
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Is the bad block what's causing FreeNAS to kick the disk from the pool?
probably... A failed read which exceeds retry limits will cause a disk to get booted.

Have a look in dmesg and you'll probably see a bunch of CAM retry messages.
 

Yangorang

Cadet
Joined
Sep 17, 2015
Messages
7
Yup...just like you said...
Replaced the drive and resilvered without any further issues.

(ada5:ahcich5:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada5:ahcich5:0:0:0): CAM status: Command timeout
(ada5:ahcich5:0:0:0): Retrying command
ahcich5: Timeout on slot 23 port 0
ahcich5: is 00000000 cs 00800000 ss 00000000 rs 00800000 tfd c0 serr 00000000 cmd 0004d717
(ada5:ahcich5:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada5:ahcich5:0:0:0): CAM status: Command timeout
(ada5:ahcich5:0:0:0): Retrying command
ahcich5: AHCI reset: device not ready after 31000ms (tfd = 00000080)
(aprobe0:ahcich5:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
(aprobe0:ahcich5:0:0:0): CAM status: Unconditionally Re-queue Request
(aprobe0:ahcich5:0:0:0): Error 5, Retry was blocked
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada5: <WDC WD40EFRX-68WT0N0 82.00A82> s/n WD-WCC4E4DCXYA1 detached
GEOM_MIRROR: Device swap0: provider ada5p1 disconnected.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702

Syptec

Dabbler
Joined
Aug 3, 2018
Messages
42
Replacement of the disk with a good one.
Wished that were truthful, but it is not. There are issue with the freeBSD MPS/MPR driver set for LSI controllers. We purchased new controllers and 8 different drives. Nytro, QVO, EVO, WD BLUE, WD GOLD, HGST ENTERPRISE and all give almost identical errors (SSD), regardless of chassis and HBA. the older 9211 (9200) appear to resolve this, the 12G LSI controllers(many)l appear to present this issue.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Which firmware are you using on your 12G LSI controller?

 
Top