Can use some help with kicked drive

[MisterE2002]

Received a degraded pool mail. Verified using command line

Code:

  pool: tank
state: DEGRADED
status: One or more devices has been removed by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
  scan: scrub repaired 0 in 0 days 13:35:13 with 0 errors on Mon May 18 13:35:15 2020
config:

    NAME                                            STATE     READ WRITE CKSUM
    tank                                            DEGRADED     0     0     0
      raidz2-0                                      DEGRADED     0     0     0
        gptid/8ad6d742-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/8e663a45-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/91df67e8-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/9600e8ca-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/9983ada6-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/9da62577-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/a15101a1-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/a4e57169-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        gptid/a9221554-735d-11e9-b886-00074305cc8f  ONLINE       0     0     0
        11480871015459646449                        REMOVED      0     0     0  was /dev/gptid/acd52da1-735d-11e9-b886-00074305cc8f

Checked the logs on disk

Code:

Jun 18 05:06:46 nas2 ada9 at ahcich13 bus 0 scbus13 target 0 lun 0
Jun 18 05:06:46 nas2 ada9: <WDC WD100EFAX-68LHPN0 83.H0A83> s/n JEJ15JBN detached
Jun 18 05:06:46 nas2 ZFS: vdev state changed, pool_guid=11356577163475570859 vdev_guid=11480871015459646449
Jun 18 05:06:46 nas2 ZFS: vdev is removed, pool_guid=11356577163475570859 vdev_guid=11480871015459646449
Jun 18 05:06:46 nas2 GEOM_MIRROR: Device swap0: provider ada9p1 disconnected.
Jun 18 05:06:46 nas2 (ada9:ahcich13:0:0:0): Periph destroyed
Jun 18 05:06:48 nas2 ZFS: vdev state changed, pool_guid=11356577163475570859 vdev_guid=11480871015459646449
Jun 18 05:06:52 nas2 GEOM_ELI: Device mirror/swap0.eli destroyed.
Jun 18 05:06:53 nas2 savecore: /dev/ada8p1: Operation not permitted
Jun 18 05:06:53 nas2 GEOM_MIRROR: Device swap0: provider destroyed.
Jun 18 05:06:53 nas2 GEOM_MIRROR: Device swap0 destroyed.
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): CAM status: ATA Status Error
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): RES: 51 04 00 00 00 00 00 00 00 00 00
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): Retrying command
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 00 00
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): CAM status: ATA Status Error
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): RES: 51 04 00 00 00 00 00 00 00 00 00
Jun 18 05:07:04 nas2 (aprobe0:ahcich13:0:0:0): Error 5, Retries exhausted

So, it kicked ada9 with serial JEJ15JBN
My first obvious step would be to do a SMART test.

But i think the drive is *really* kicked.
Can not see the drive using " geom disk list" or "camcontrol devlist".
What are the best steps to proceed?
* reboot system?
* hope the drive is visible again
* If visible, smart check
* re-add drive or warranty replacement

or:
* shutdown system
* remove drive. Attach to Linux
* SMART test in Linux.
* Keep system off until replacement disk

 

[Samuel Tai]

The pool is RAIDZ2, so it can tolerate the loss of one drive. You should shut down the system and pull the dead drive. Restart the system, and when a replacement drive arrives, go through the disk replacement process from the Guide, Section 10.5.1.

You can try futzing with the kicked drive in another system, but it's not really worth the effort.