Losing Connection Sporadically to iSCSI

[SwisherSweet]

I am wondering how I troubleshoot lost iSCSI connections. Yesterday as I was using my computer, I saw 4 iSCSI connections disappear on different machines at the same time, then a few minutes later, they reconnected. The server is highly underutilized.

It is possible that I lost shares as well, but I didn't think to check that.

Is there a place I can look to check for any errors or problems that need to be resolved?

A little more setup information:

• Mac Pro, 2 x 6 Core 3.46Hz Xeons, 64gb ECC RAM
• FreeNAS-9.10.2-U3 (e1497f269)
• 7 x 3TB Toshiba drives in "primary" data pool in raidz2
• 5 x 2TB Seagate drives in "backup1" backup pool (externally attached) in raidz1
• NETGEAR GS-308 Switch

Please let me know if I'm missing any information that might help.

Thank you.

 

[SweetAndLow]

FreeNAS version? /var/log/messages

Sent from my Nexus 5X using Tapatalk

 

[SwisherSweet]

Freenas version? /var/log/messages

Sent from my Nexus 5X using Tapatalk

FreeNAS-9.10.2-U3 (e1497f269)


Here's what I found that appeared to be around the time of the connection dropping:

Jun 18 16:03:53 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 13 tasks
Jun 18 16:03:58 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:03:58 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:05:16 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:05:16 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:05:44 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:05:44 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:06:12 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:06:12 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:06:40 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:06:40 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:07:08 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:07:08 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:07:36 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:07:36 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:08:01 Megatron autosnap.py: [tools.autosnap:66] Popen()ing: /sbin/zfs snapshot -r "primary/bonecrusher-backup@auto-20170618.1608-90d"
Jun 18 16:08:01 Megatron autosnap.py: [tools.autosnap:66] Popen()ing: /sbin/zfs snapshot -r "primary/bonecrusher-data@auto-20170618.1608-90d"
Jun 18 16:08:04 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:08:04 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:08:32 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:08:32 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:09:00 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:09:00 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:10:18 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:10:18 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:10:52 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:10:52 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:11:20 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:11:20 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:11:48 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:11:48 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:12:16 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:12:16 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:12:44 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:12:44 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:13:12 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:13:12 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:13:42 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:13:42 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 698 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 678 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 01 a4 00 00 02 28 00 00
Jun 18 16:15:11 Megatron (0:3:0/0): Tag: 0x4bb53100, type 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 620 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 595 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 592 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 567 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 564 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 539 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 536 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 511 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 509 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 484 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 481 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 456 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 453 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
:
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 698 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 678 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 01 a4 00 00 02 28 00 00
Jun 18 16:15:11 Megatron (0:3:0/0): Tag: 0x4bb53100, type 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 620 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 595 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 592 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 567 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 564 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 539 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 536 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 511 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 509 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 484 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 481 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 456 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 453 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
:
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 698 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 678 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 01 a4 00 00 02 28 00 00
Jun 18 16:15:11 Megatron (0:3:0/0): Tag: 0x4bb53100, type 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 620 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 595 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 592 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 567 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 564 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 539 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 536 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 511 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 509 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 484 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 481 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 456 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 453 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 428 seconds
:
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:10 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:14:38 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): no ping reply (NOP-Out) after 5 seconds; dropping connection
Jun 18 16:15:06 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): waiting for CTL to terminate 1 tasks
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 698 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 678 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 01 a4 00 00 02 28 00 00
Jun 18 16:15:11 Megatron (0:3:0/0): Tag: 0x4bb53100, type 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 673 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 620 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 595 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 592 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 567 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 564 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 539 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 536 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 511 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 509 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 484 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 481 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 456 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron (0:3:0/0): ctl_process_done: 453 seconds
Jun 18 16:15:11 Megatron (4294967295:3:0/0): Task I/O type: 4, Tag: 0x0000, Tag Type: 1
Jun 18 16:15:11 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:11 Megatron (4294967295:3:0/0): ctl_process_done: 428 seconds
Jun 18 16:15:11 Megatron (0:3:0/0): Task I/O type: 5, Tag: 0x0000, Tag Type: 1
:

 

[SwisherSweet]

I could only post small piece of the relevant log entries due to message size restrictions on the forum. Please let me know if you need to see more.

It kinda does the same stuff over and over, then the last bit looks like this:

Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:14 Megatron (0:3:0/0): Tag: 0x57b53100, type 0
Jun 18 16:15:14 Megatron (0:3:0/0): ctl_process_done: 235 seconds
Jun 18 16:15:14 Megatron (0:3:0/0): INQUIRY. CDB: 12 00 00 00 24 00
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:14 Megatron (0:3:0/0): Tag: 0x58b53100, type 0
Jun 18 16:15:14 Megatron (0:3:0/0): ctl_process_done: 207 seconds
Jun 18 16:15:14 Megatron (0:3:0/0): INQUIRY. CDB: 12 00 00 00 24 00
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:14 Megatron (0:3:0/0): Tag: 0x59b53100, type 0
Jun 18 16:15:14 Megatron (0:3:0/0): ctl_process_done: 179 seconds
Jun 18 16:15:14 Megatron (0:3:0/0): INQUIRY. CDB: 12 00 00 00 24 00
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:14 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:14 Megatron (0:3:0/0): Tag: 0x5ab53100, type 0
Jun 18 16:15:15 Megatron (0:3:0/0): ctl_process_done: 151 seconds
Jun 18 16:15:15 Megatron (0:3:0/0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:15 Megatron (0:3:0/0): Tag: 0x5bb53100, type 0
Jun 18 16:15:15 Megatron (0:3:0/0): ctl_process_done: 123 seconds
Jun 18 16:15:15 Megatron (0:3:0/0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:15 Megatron (0:3:0/0): Tag: 0x5cb53100, type 0
Jun 18 16:15:15 Megatron (0:3:0/0): ctl_process_done: 93 seconds
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): connection error; dropping connection
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:15 Megatron WARNING: 192.168.1.158 (iqn.1991-05.com.microsoft:bonecrusher): tasks terminated
Jun 18 16:15:15 Megatron (0:3:0/0): EXTENDED COPY. CDB: 83 11 00 00 00 00 00 00 01 a4 00 00 02 28 00 00
Jun 18 16:15:15 Megatron (0:3:0/0): Tag: 0x3fb53100, type 1
Jun 18 16:15:15 Megatron (0:3:0/0): ctl_process_done: 763 seconds

 

[jgreco]

RAIDZ and iSCSI aren't going to be good partners. You have the throughput of a single disk device, and if you flood it with writes, the system can definitely start to choke.

https://forums.freenas.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/

https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

see esp. "Tangent: Dangers of overly large ZFS write cache."

What are you doing with this, anyways? Microsoft doesn't really have any good clustered filesystems, so if you're running something like NTFS on top of a shared iSCSI block device, this is fraught with hazard and peril, and is expected to not work.

 

[SwisherSweet]

RAIDZ and iSCSI aren't going to be good partners. You have the throughput of a single disk device, and if you flood it with writes, the system can definitely start to choke.

https://forums.freenas.org/index.ph...d-why-we-use-mirrors-for-block-storage.44068/

https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

see esp. "Tangent: Dangers of overly large ZFS write cache."

What are you doing with this, anyways? Microsoft doesn't really have any good clustered filesystems, so if you're running something like NTFS on top of a shared iSCSI block device, this is fraught with hazard and peril, and is expected to not work.

Wow, I wish I knew iSCSI doesn't work well with RAIDZ before I moved everything to the server. I have several (extremely low volume) uses:

1. Windows Backup Server, where I backup about 8 GB per day to an iSCSI Target formatted NTFS. It happens around midnight, where nothing else is happening.
2. Another Windows backup to a separate iSCSI Target formatted NTFS to backup a Windows development machine. Perhaps a few gb per 1 hour.
3. Mac with iSCSI target formatted HFS+, storing about 400GB of mostly static data (Photo Libraries).

Basically, not much is happening.

Are you suggesting that one should not use block storage on RAIDZ if you are going to use Windows NTFS?

Is there anyway to make this work without destroying and recreating my FreeNAS server with mirrors?

Thank you.

 

[SwisherSweet]

see esp. "Tangent: Dangers of overly large ZFS write cache."

If I didn't wanna blow-up my FreeNAS server and create mirrors just for my iSCSI targets, could I possibly add a SLOG to help address the concerns you pointed out in your referenced posts? Do you believe my disconnects are due to too many writes?

 

[jgreco]

The block storage vs RAIDZ thing is fairly well-known, though obviously it keeps tripping people up. It's not *unusable*, just way suboptimal.

As long as your NTFS are separate volumes on separate targets, you're fine. We run into people trying to use NTFS on a single target simultaneously on multiple machines. This Does Not Work And Can Never Work. There is no reason you cannot use NTFS on top of an iSCSI target for a single host.

SLOG won't really help the problem that is probably hitting you, which I'm guessing is probably write bottlenecking, which is usually the thing that stops iSCSI cold. What happens is that a sudden burst of writes to a fragmented pool results in a large transaction group being queued to commit, and then not finishing within the five seconds that are available to it. Disk seeking sucks.

You can try SLOG and it may even seem to help, but if it helps, my guess would be it is only because it is slowing down the rate at which writes happen to the pool. You're better off looking at more rapid transaction group flushing (1 second instead of 5 seconds) to help the system not get into a situation where it is stunned by a sudden burst of traffic. This will slow down overall performance somewhat, but in return for that tradeoff, you usually get a much more consistent response time for iSCSI.

In general, for iSCSI, you gotta throw lots of resources at the system. Then it'll fly.

 

[SwisherSweet]

Thanks jgreco. Actually, this appears to be a bug logged for this issue, which I added to:

https://bugs.freenas.org/issues/19892

It doesn't have "bug" status yet, but at least it's being looked at.

I've forced my iSCSI targets to sync=always. I will monitor and see if helps. So far, the problem appears to confined to one Windows 8.1 workstation.

As expected, forcing the zvol/iSCSI to sync=always slowed down the write speed a lot:

Before:

After:

I am curious about sync writes. If you don't have a SLOG (which I obviously currently do not), then writes are still no protected from power failures, since it still is writing to the ZIL via a standard non-power-protected SATA spinning disk. If the plug is pulled, then you have the same problem as if you were using a non-power-protected SSD as your SLOG, correct?

If this is the case, the one could conclude:

1. Async writes are not protected against crashes and pulled plugs. FreeNAS sends ack of data still in RAM when plug is pulled.
2. Sync writes without a slog are not protected against crashes and pulled plugs. FreeNAS sends ack of data still in SATA disk buffer, but not written when plug is pulled.
3. Only sync writes with power protected SLOG are safe from crashes and pulled plugs.

Is this accurate?

Thanks again!

 

[jgreco]

Thanks jgreco. Actually, this appears to be a bug logged for this issue, which I added to:

https://bugs.freenas.org/issues/19892

It doesn't have "bug" status yet, but at least it's being looked at.

Basically I think this is "not a bug" and is summed up by what I've said above and mav's comment in #27 in the issue report.

ZFS isn't real good at block storage unless you're willing to do it the ZFS way, which involves mirrors and low pool occupancy rates and high levels of fragmentation and all the necessary steps to mitigate fragmentation.

If you do NOT play the game by ZFS's rules, then when you suddenly write large blobs of data, ZFS will end up trying to flush a massive transaction group to disk, I/O to the pool will suspend pending completion, and your iSCSI sessions will drop when that doesn't happen quickly. Fragmentation, high pool occupancy, and RAIDZ are the major contributing factors to that. You appear to have a trifecta. (I'm sure you don't want to hear that, and I'm sorry).

I am curious about sync writes. If you don't have a SLOG (which I obviously currently do not), then writes are still no protected from power failures, since it still is writing to the ZIL via a standard non-power-protected SATA spinning disk. If the plug is pulled, then you have the same problem as if you were using a non-power-protected SSD as your SLOG, correct?

Incorrect. Mostly. If you disable the disk's write cache, which you should, then when ZFS commits a block to the in-pool ZIL, you are guaranteed that upon completion that the system has committed the block to stable storage, which is the POSIX mandate. The actual commit of the transaction group to the pool happens a bit later. The ZIL is there to be able to recreate the "in-flight" transaction group. Once the ZIL has your data in the intent log, that's "good enough" even though the system hasn't actually written the data to the pool yet. If the system crashes AFTER the ZIL commit but BEFORE the transaction group flush, then what happens is when the system reboots and the pool is imported, the ZIL is read from disk (or SLOG) and the system makes very sure that the stuff in the ZIL is actually committed to the pool.

So for sync writes without a SLOG, because each block you're writing to disk involves a ZIL block write request to a disk that may already be busy committing a transaction group to the pool, and ZFS will *block* until that ZIL write request is actually committed to the platters, so under good conditions this is merely sucky-slow, but under bad conditions it is like sending your blocks to disk by Western Union telegram. This is why people like SLOG.

But people enable their disk's write cache. And they do other dumb stuff, like put on a random SSD without power-loss protection as a SLOG. If you don't care about data corruption in a power loss or system crash, just DO IT RIGHT, disable sync entirely, turn on disk write caching, and let it fly as fast as it can go. If you DO care about data corruption in a power loss or system crash, then you really have to dot all your i's and cross all your t's. Going halfway is like driving around in your car without a seatbelt thinking the airbags will save you.

So if you've enabled the write cache on your HDD and you're using the in-pool ZIL, then, yes, that's corruptible, but that's also a bad configuration.

I'm kind of going the extra step here with you because your discussion of the issue seems to show that you're trying hard to understand, and I appreciate that. ;-) Most of the information about ZFS is hard to discover on your own.

 

[SwisherSweet]

I'm kind of going the extra step here with you because your discussion of the issue seems to show that you're trying hard to understand, and I appreciate that. ;-) Most of the information about ZFS is hard to discover on your own.

And your help is immensely appreciated!!! I was hesitant to use FreeNAS because it's open source and built on FreeBSD, which is completely foreign to me. I decided to go for it cause folks like you are generous with your time and knowledge and are willing to help others out. So thank you!

I feel like I've been drinking from the firehose. FreeNAS is definitely not for the faint of heart. :)

I will need to go back and reread your replies and let it all sync, ahem I mean sink in... LOL

 

[jgreco]

You wouldn't believe the number of people who used to arrive here with their little 4GB E350 systems and three disk drives in a RAIDZ and wonder why these things weren't awesome iSCSI servers (etc). And who then seemed to think that we were conspiring against them to make their thing-that-they-knew-was-awesome not work.

If you come to me and say that you're dismayed that ZFS wants tremendous resources to do what seems like a simple task, hey, I'll buy you a beer and tell you stories that'll curl your toes. I don't know that anyone actually likes it, but from a computer science perspective it all makes a certain kind of sense. To those who will listen to the hard-won words I have to say, I'm happy to share insights. Professionally, one of my roles is telling people the things they need to hear rather than the things they want to hear.

It is poor comfort to you, I'm sure, but you're far from the first disappointed person who wanted to do this sort of thing. On the upside, you're a hell of a lot closer than most of them, because you've at least got hardware that's ballpark-capable of it.

If you were to take that 7-drive RAIDZ2 of 3TB drives, and create three two-way 3TB mirror vdevs, that'd get you a 9TB pool, of which you could use around 4-5TB of. Gives you one warm spare drive to deal with drive failures. Not quite "RAIDZ2" but not bad.

So here's the thing. If you did that - or, heck, if you started out with a fresh RAIDZ2 pool - what happens is that while the pool is new and unfragmented, everything seems fairly fast and maybe even fast enough that you think all the ZFS bad performance stuff you hear about is B.S. But then as time goes on, the pool performance slowly degrades. It eventually reaches a more-or-less steady state level of performance. You can read more about that here. Look at the picture and also read the followups under the linked post.

I'm sure that finding out that 21TB of raw space might only give you 4-5TB of actual good usable space is probably not what you want to hear, because nobody wants to hear it. But that's the way it works out. If you have questions about any of this that aren't sufficiently explained in the posts I've linked to, I am usually happy to go on and on and on about it.