(11.2-u5) I have a pool, ssd2x6x480GBz1 (btw its da0 to da11 that make up this pool) , which is 12x 480g HGST enterprise HUSMR1650ASS204 disks, connected via a BPN-SAS-216el1 expander / 2u diskshelf to my main FN 4u box (2u is connected via a lsi 9207-8e HBA (at FW= 20). It is using a intel p900 optane as its zil/slog
All has been running great for about a week or 2 now. (pool is mainly used as a NFS share to my vmware cluster of 3x hosts running latest version of 6.5 esxi w Vcenter.
I wake up this morning and all hell has broken loose on the vmware cluster, as the NFS share for ssd2x6x480GBz1 is gone.
on FREENAS, this pool is listed as ONLINE but none of the datastores are shown, pool status shows a few read and write errors across all the disks of the pool. (all other pools of other disks are fine but are all via different controllers too)
Im almost 100% sure this was cause by all of the disks starting their (scheduled) smart-test -long (i had just set this up to run once a month following a guide here on the forums, which i agree with). - so currently my best guess as to this issue is: Due to all the ssd drives doing their smart long test (which 56min later, they all passed btw)
Those smart-long tests increased latency and caused timeouts for IO requests (or either the expander or HBA started timing out IO req's) which intern caused FN to start logging errors, and eventually take down the pool (as a safeguard maybe?). (after a reboot (shutdown hung, so power off / power up) , the disks showed up fine, at which point i did a scrub, 0 errors)
here are the logs i was able to grab (and why i think this was the cause).
(ofcourse my splunk syslog server was running on this NFS ssd vol so it died! lesson learned there)
its a long log, so here is pastebin of all, and ill try to pull out some of relevant here below:
https://pastebin.com/GR863TTA
note: in nearly 10 months of stress testing this exact hardware (and FN), i had never once seen any CDB errors from these SSDs / expander / hba - however this is the first time ive run smart-long tests on these specific drives (if that is the cause). looking for any info or input from you all, as this was pretty shocking (and seems like could have been alot worse).
smart data on all 12 ssds looks about perfect
some raw data i pulled:
All has been running great for about a week or 2 now. (pool is mainly used as a NFS share to my vmware cluster of 3x hosts running latest version of 6.5 esxi w Vcenter.
I wake up this morning and all hell has broken loose on the vmware cluster, as the NFS share for ssd2x6x480GBz1 is gone.
on FREENAS, this pool is listed as ONLINE but none of the datastores are shown, pool status shows a few read and write errors across all the disks of the pool. (all other pools of other disks are fine but are all via different controllers too)
Im almost 100% sure this was cause by all of the disks starting their (scheduled) smart-test -long (i had just set this up to run once a month following a guide here on the forums, which i agree with). - so currently my best guess as to this issue is: Due to all the ssd drives doing their smart long test (which 56min later, they all passed btw)
Those smart-long tests increased latency and caused timeouts for IO requests (or either the expander or HBA started timing out IO req's) which intern caused FN to start logging errors, and eventually take down the pool (as a safeguard maybe?). (after a reboot (shutdown hung, so power off / power up) , the disks showed up fine, at which point i did a scrub, 0 errors)
here are the logs i was able to grab (and why i think this was the cause).
(ofcourse my splunk syslog server was running on this NFS ssd vol so it died! lesson learned there)
its a long log, so here is pastebin of all, and ill try to pull out some of relevant here below:
https://pastebin.com/GR863TTA
note: in nearly 10 months of stress testing this exact hardware (and FN), i had never once seen any CDB errors from these SSDs / expander / hba - however this is the first time ive run smart-long tests on these specific drives (if that is the cause). looking for any info or input from you all, as this was pretty shocking (and seems like could have been alot worse).
smart data on all 12 ssds looks about perfect
some raw data i pulled:
Code:
during the issue: pool: ssd2x6x480GBz1 state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://illumos.org/msg/ZFS-8000-JQ scan: none requested config: NAME STATE READ WRITE CKSUM ssd2x6x480GBz1 ONLINE 16 8 0 raidz1-0 ONLINE 15 4 0 gptid/1eae5c41-c634-11e9-975e-00259084f1c8 ONLINE 0 0 0 gptid/1f09b571-c634-11e9-975e-00259084f1c8 ONLINE 2 0 0 gptid/1f7dea92-c634-11e9-975e-00259084f1c8 ONLINE 8 120 0 gptid/1fde322b-c634-11e9-975e-00259084f1c8 ONLINE 2 0 0 gptid/203ae8b8-c634-11e9-975e-00259084f1c8 ONLINE 3 2 0 gptid/20ae8720-c634-11e9-975e-00259084f1c8 ONLINE 0 2 0 raidz1-1 ONLINE 10 4 0 gptid/212383df-c634-11e9-975e-00259084f1c8 ONLINE 3 6 0 gptid/218371b5-c634-11e9-975e-00259084f1c8 ONLINE 3 6 0 gptid/21e22950-c634-11e9-975e-00259084f1c8 ONLINE 3 6 0 gptid/22567ad2-c634-11e9-975e-00259084f1c8 ONLINE 0 0 0 gptid/22b87683-c634-11e9-975e-00259084f1c8 ONLINE 2 0 0 gptid/231a7df4-c634-11e9-975e-00259084f1c8 ONLINE 1 4 0 logs gptid/f166ac78-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <metadata>:<0x0> ssd2x6x480GBz1/VMwareNFSssdZ1:<0xa21> ssd2x6x480GBz1/VMwareNFSssdZ1:<0x576> ssd2x6x480GBz1/VMwareNFSssdZ1:<0x3cc> ssd2x6x480GBz1/VMwareNFSssdZ1:<0x2d6> all the SSDs smart looks like this: # smartctl -a /dev/da2 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: HGST Product: HSCP0480S5xnNMRI Revision: 3P02 Compliance: SPC-4 User Capacity: 480,999,997,440 bytes [480 GB] Logical block size: 512 bytes Physical block size: 4096 bytes LU is resource provisioned, LBPRZ=1 Rotation Rate: Solid State Device Form Factor: 2.5 inches Logical Unit id: 0x5000cca04e0b9898 Serial number: 0QV6BPRA Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Mon Sep 2 15:34:09 2019 CDT SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Enabled === START OF READ SMART DATA SECTION === SMART Health Status: OK Percentage used endurance indicator: 1% Current Drive Temperature: 33 C Drive Trip Temperature: 70 C Manufactured in week 04 of year 2015 Specified cycle count over device lifetime: 0 Accumulated start-stop cycles: 0 Specified load-unload count over device lifetime: 0 Accumulated load-unload cycles: 0 Vendor (Seagate) cache information Blocks sent to initiator = 56558735039922948 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 0 0 0 0 0 80589.840 0 write: 0 0 0 0 0 73225.161 0 verify: 0 0 0 0 0 56863.026 0 Non-medium error count: 0 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background long Completed - 24828 - [- - -] Long (extended) Self Test duration: 3468 seconds [57.8 minutes] and items i pulled from logs during issue: Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da11, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da12 [SAT], starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da2, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da21 [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 69 Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da21 [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 32 to 31 Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da21 [SAT], starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da3, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da34, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da35, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da36, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da4, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da5, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da6, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da7, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da8, starting scheduled Long Self-Test. Sep 2 06:21:37 freenas smartd[84638]: Device: /dev/da9, starting scheduled Long Self-Test. Sep 2 06:36:59 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 34 7b 50 90 00 00 38 00 length 28672 SMID 757 Aborting command 0xfffffe0001527190 Sep 2 06:36:59 freenas mps0: Sending reset from mpssas_send_abort for target ID 63 Sep 2 06:37:02 freenas (da0:mps0:0:59:0): WRITE(10). CDB: 2a 00 29 87 96 40 00 00 08 00 length 4096 SMID 404 Aborting command 0xfffffe000150a240 Sep 2 06:37:02 freenas (da4:mps0:0:65:0): WRITE(10). CDB: 2a 00 20 72 c8 98 00 00 38 00 length 28672 SMID 667 Aborting command 0xfffffe000151fb70 Sep 2 06:37:02 freenas mps0: Sending reset from mpssas_send_abort for target ID 59 Sep 2 06:37:02 freenas mps0: Sending reset from mpssas_send_abort for target ID 65 Sep 2 06:37:03 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 34 7b 4f f0 00 00 38 00 length 28672 SMID 400 Aborting command 0xfffffe0001509d00 Sep 2 06:37:03 freenas (da2:mps0:0:63:0): CAM status: CAM subsystem is busy Sep 2 06:37:03 freenas (da2:mps0:0:63:0): CAM status: Command timeout Sep 2 06:37:03 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 34 7b 50 90 00 00 38 00 Sep 2 06:37:03 freenas (da2:mps0:0:63:0): Retrying command Sep 2 06:37:03 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 63 Sep 2 06:37:03 freenas mps0: Sending reset from mpssas_send_abort for target ID 63 Sep 2 06:37:04 freenas (da2:mps0:0:63:0): CAM status: CAM subsystem is busy Sep 2 06:37:04 freenas (da2:mps0:0:63:0): Error 5, Retries exhausted Sep 2 06:37:04 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00 Sep 2 06:37:04 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 34 7b 50 90 00 00 38 00 Sep 2 06:37:04 freenas (da2:mps0:0:63:0): Retrying command Sep 2 06:37:04 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 63 Sep 2 06:37:05 freenas (da2:mps0:0:63:0): CAM status: CAM subsystem is busy Sep 2 06:37:05 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00 Sep 2 06:37:05 freenas (da2:mps0:0:63:0): Retrying command Sep 2 06:37:05 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 63 Sep 2 06:37:06 freenas (da0:mps0:0:59:0): CAM status: Command timeout Sep 2 06:37:06 freenas (da0:mps0:0:59:0): Retrying command Sep 2 06:37:06 freenas Unfreezing devq for target ID 59 Sep 2 06:37:07 freenas (da2:mps0:0:63:0): READ(10). CDB: 28 00 34 7b 50 58 00 00 38 00 length 28672 SMID 1084 Aborting command 0xfffffe0001541ec0 Sep 2 06:37:13 freenas zfsd: Vdev State = ONLINE Sep 2 06:37:13 freenas zfsd: CaseFile(12609516878478959410,8386571431618165161,id1,enc@n5003048000c0fa7d/type@0/slot@8/elmdesc@Slot_08/p2) Sep 2 06:37:13 freenas zfsd: CaseFile(12609516878478959410,8386571431618165161) closed - State ONLINE Sep 2 06:37:13 freenas zfsd: Creating new CaseFile: Sep 2 06:37:14 freenas (da2:mps0:0:63:0): CAM status: CAM subsystem is busy Sep 2 06:37:14 freenas (da2:mps0:0:63:0): Error 5, Retries exhausted Sep 2 06:38:32 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 65 Sep 2 06:38:33 freenas (da4:mps0:0:65:0): CAM status: CAM subsystem is busy Sep 2 06:38:33 freenas (da4:mps0:0:65:0): Error 5, Retries exhausted Sep 2 06:38:33 freenas (da4:mps0:0:65:0): READ(10). CDB: 28 00 00 40 02 90 00 00 10 00 Sep 2 06:38:33 freenas (da4:mps0:0:65:0): Retrying command Sep 2 06:38:33 freenas (da4:mps0:0:65:0): WRITE(10). CDB: 2a 00 09 ed 7f 80 00 00 10 00 Sep 2 06:38:33 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 65 Sep 2 06:40:34 freenas mps0: Unfreezing devq for target ID 68 Sep 2 06:40:42 freenas zfsd: Vdev State = ONLINE Sep 2 06:40:42 freenas zfsd: CaseFile(12609516878478959410,13190870688141053705,id1,enc@n5003048000c0fa7d/type@0/slot@d/elmdesc@Slot_13/p2) Sep 2 06:40:42 freenas zfsd: Creating new CaseFile: Sep 2 06:40:42 freenas zfsd: Evaluating existing case file Sep 2 06:40:42 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=14853523331502576641 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567424442 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=11368 zio_err=5 zio_level=0 zio_object=726 zio_objset=79 zio_offset=498760335360 zio_size=112640 Sep 2 06:40:42 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=14853523331502576641 parent_guid=16827335826208397105 parent_type=raidz pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567424442 type=ereport.fs.zfs.io vdev_guid=13190870688141053705 vdev_path=/dev/gptid/20ae8720-c634-11e9-975e-00259084f1c8 vdev_type=disk zio_blkid=11368 zio_err=5 zio_level=0 zio_object=726 zio_objset=79 zio_offset=83130916864 zio_size=20480 Sep 2 06:40:42 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=14853523331502579713 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567424442 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=28281 zio_err=5 zio_level=0 zio_object=726 zio_objset=79 zio_offset=498760695808 zio_size=26112 Sep 2 06:40:42 freenas zfsd: ZfsEvent::Process: Event for unknown vdev 16827335826208397105 queued Sep 2 06:40:43 freenas (da4:mps0:0:65:0): WRITE(10). CDB: 2a 00 0a 17 57 58 00 00 38 00 length 28672 SMID 1025 Aborting command 0xfffffe000153d150 Sep 2 06:40:43 freenas mps0: Sending reset from mpssas_send_abort for target ID 65 Sep 2 06:40:44 freenas (da9:mps0:0:77:0): UNMAP. CDB: 42 00 00 00 00 00 00 00 18 00 length 24 SMID 993 Aborting command 0xfffffe000153a750 Sep 2 06:40:44 freenas mps0: Sending reset from mpssas_send_abort for target ID 77 Sep 2 06:40:47 freenas (da4:mps0:0:65:0): READ(10). CDB: 28 00 34 7b 51 98 00 00 38 00 length 28672 SMID 303 Aborting command 0xfffffe0001501db0 Sep 2 06:40:47 freenas (pass0:mps0:0:59:0): LOG SENSE. CDB: 4d 00 43 00 00 00 00 00 58 00 length 88 SMID 358 Aborting command 0xfffffe00015065e0 Sep 2 06:40:47 freenas (da4:mps0:0:65:0): CAM status: CAM subsystem is busy Sep 2 06:40:47 freenas (da4:mps0:0:65:0): CAM status: Command timeout Sep 2 06:40:47 freenas (da4:mps0:0:65:0): Retrying command Sep 2 06:40:47 freenas (da4:mps0:0:65:0): WRITE(10). CDB: 2a 00 0a 17 57 58 00 00 38 00 Sep 2 06:40:47 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 65 Sep 2 06:40:47 freenas mps0: Sending reset from mpssas_send_abort for target ID 59 Sep 2 06:40:47 freenas mps0: Sending reset from mpssas_send_abort for target ID 65 Sep 2 06:40:48 freenas (da4:mps0:0:65:0): CAM status: CAM subsystem is busy Sep 2 06:40:48 freenas (da4:mps0:0:65:0): Error 5, Retries exhausted Sep 2 06:41:01 freenas (da4:mps0:0:65:0): Retrying command Sep 2 06:41:01 freenas (da4:mps0:0:65:0): WRITE(10). CDB: 2a 00 0a 17 57 c8 00 00 30 00 Sep 2 06:41:01 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 65 Sep 2 06:41:01 freenas zfsd: Vdev State = ONLINE Sep 2 06:41:01 freenas zfsd: CaseFile(12609516878478959410,7983501312401782520,id1,enc@n5003048000c0fa7d/type@0/slot@a/elmdesc@Slot_10/p2) Sep 2 06:41:01 freenas zfsd: CaseFile(12609516878478959410,7983501312401782520) closed - State ONLINE Sep 2 06:41:01 freenas zfsd: Creating new CaseFile: Sep 2 06:41:01 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=14873616139029581825 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567424461 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=666701 zio_err=6 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=2691976478720 zio_size=131072 Sep 2 06:41:01 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=14873616139029584897 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567424461 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=39370 zio_err=6 zio_level=0 zio_object=2593 zio_objset=79 zio_offset=987861475328 zio_size=23552 Sep 2 06:41:01 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=14873616139029593089 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567424461 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=666702 zio_err=6 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=2691976642560 zio_size=131072 Sep 2 06:41:01 freenas zfsd: ZfsEvent::Process: Event for unknown vdev 16827335826208397105 queued Sep 2 06:41:02 freenas (da4:mps0:0:65:0): CAM status: CAM subsystem is busy Sep 2 06:41:02 freenas (da4:mps0:0:65:0): Retrying command Sep 2 06:41:02 freenas (da4:mps0:0:65:0): WRITE(10). CDB: 2a 00 0a 17 57 c8 00 00 30 00 Sep 2 06:41:02 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 6 Sep 2 06:49:55 freenas (da10:mps0:0:78:0): READ(10). CDB: 28 00 37 fe e8 90 00 00 10 00 Sep 2 06:49:55 freenas (da10:mps0:0:78:0): Retrying command Sep 2 06:49:55 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 78 Sep 2 06:49:56 freenas (da10:mps0:0:78:0): CAM status: CAM subsystem is busy Sep 2 06:49:56 freenas (da10:mps0:0:78:0): CAM status: Command timeout Sep 2 06:49:56 freenas (da10:mps0:0:78:0): READ(10). CDB: 28 00 37 fe e8 90 00 00 10 00 Sep 2 06:49:56 freenas (da10:mps0:0:78:0): Retrying command Sep 2 06:49:56 freenas (da10:mps0:0:78:0): UNMAP. CDB: 42 00 00 00 00 00 00 00 18 00 Sep 2 06:49:56 freenas mps0: mpssas_action_scsiio: Freezing devq for target ID 78 Sep 2 06:49:56 freenas mps0: Unfreezing devq for target ID 78 Sep 2 06:50:07 freenas zfsd: Evaluating existing case file Sep 2 06:50:07 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15446347261536113665 parent_guid=16827335826208397105 parent_type=raidz pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425007 type=ereport.fs.zfs.io vdev_guid=1758322195652712981 vdev_path=/dev/gptid/1fde322b-c634-11e9-975e-00259084f1c8 vdev_type=disk zio_blkid=668231 zio_err=5 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=115123744768 zio_size=24576 Sep 2 06:50:28 freenas zfsd: Evaluating existing case file Sep 2 06:50:28 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15446347261536113665 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425028 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=668231 zio_err=5 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=690717302784 zio_size=131072 Sep 2 06:50:28 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15446347261536113665 parent_guid=16827335826208397105 parent_type=raidz pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425028 type=ereport.fs.zfs.io vdev_guid=5936719101646054698 vdev_path=/dev/gptid/1f09b571-c634-11e9-975e-00259084f1c8 vdev_type=disk zio_blkid=668231 zio_err=5 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=115123744768 zio_size=28672 Sep 2 06:50:28 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15467621818753354753 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425027 type=ereport.fs.zfs.io vdev_guid=16827335826208397105 vdev_type=raidz zio_blkid=668230 zio_err=6 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=690717138944 zio_size=131072 Sep 2 06:50:28 freenas zfsd: ZfsEvent::Process: Event for unknown vdev 16827335826208397105 queued Sep 2 06:50:40 freenas zfsd: Vdev State = ONLINE Sep 2 06:50:40 freenas zfsd: CaseFile(12609516878478959410,13579473535141282300,id1,enc@n5003048000c0fa7d/type@0/slot@15/elmdesc@Slot_21/p2) Sep 2 06:50:40 freenas zfsd: Creating new CaseFile: Sep 2 06:50:40 freenas zfsd: Evaluating existing case file Sep 2 06:50:40 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15480865334799371265 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425040 type=ereport.fs.zfs.io vdev_guid=3203759444463917272 vdev_type=raidz zio_blkid=668233 zio_err=5 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=2537086459904 zio_size=131072 Sep 2 06:50:40 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15480865334799371265 parent_guid=3203759444463917272 parent_type=raidz pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425040 type=ereport.fs.zfs.io vdev_guid=13579473535141282300 vdev_path=/dev/gptid/22b87683-c634-11e9-975e-00259084f1c8 vdev_type=disk zio_blkid=668233 zio_err=5 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=422851936256 zio_size=28672 Sep 2 06:50:40 freenas zfsd: ZFS: Notify class=ereport.fs.zfs.io ena=15480865334799373313 parent_guid=12609516878478959410 parent_type=root pool=ssd2x6x480GBz1 pool_context=0 pool_failmode=continue pool_guid=12609516878478959410 subsystem=ZFS timestamp=1567425040 type=ereport.fs.zfs.io vdev_guid=3203759444463917272 vdev_type=raidz zio_blkid=668232 zio_err=5 zio_level=0 zio_object=1398 zio_objset=79 zio_offset=2537085968384 zio_size=131072 Sep 2 06:50:40 freenas zfsd: ZfsEvent::Process: Event for unknown vdev 3203759444463917272 queued Sep 2 06:50:56 freenas smartd[84638]: Device: /dev/da27 [SAT], self-test in progress, 90% remaining
Last edited: