[SOLVED] Deleting Files results in ctl_datamove: tag ... on ... aborted

[Pointeo13]

EDIT: FreeNAS-9.10-STABLE has solved the Issue!

I thought I had this issue solved awhile ago but I guess not (I don't delete files that often) but I can replicate this issue each time I copy a directory that has 48.6GB of data, delete the files using the console (which does immediately) and then I begin to receive these ctl_datamove aborted messages on the console. I have replaced all my SAS cables, and tried different LSI M1015 cards with the supported firmware version. I have also tried to copy the files from one pool to another pool, so i copy the files from my archive pool (Raidz_2_SATA) that's all SATA drives to my vmware pool (Raidz_VMware_SAS) that's all SAS drives and I run into the same issue.

These pool's live on different Supermicro chassis hardware . I currently have three LSI M1015 raid card's installed, one card for l2arc and slog, second card is for archive storage (Raidz_2_SATA) and the third raid card is for vmware storage (Raidz_VMware_SAS.)

I have also removed l2arc and slog to make sure that wasn't the issue. The current services that are running are cifs and iSCSI, but I always replicate the issue by doing the delete through the console or through CIFS. Not only do I receive the messages below, but my CPU maxes out, I lose graph information during that time and my cifs shares and ssh connections appear to be frozen.

Server: IBM x3850 X5
CPU: 4x Intel Xeon E7-4830 2.13Ghz
Memory: 1048528MB
Raid Card - 3x LSI M1015 (IT Mode)
FreeNAS-9.3-STABLE-201512121950

Code:

Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1991-05.com.microsoft:desktop-g6fn6t0): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1991-05.com.microsoft:exc01.domain): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): waiting for CTL to terminate 8 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): waiting for CTL to terminate 6 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): waiting for CTL to terminate 7 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): waiting for CTL to terminate 8 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78beb on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bec on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bed on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bee on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bef on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x5f343d on (9:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x5f343e on (9:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788f8 on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788f9 on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fa on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5605 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5606 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5607 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5608 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5609 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fb on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fc on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fd on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fe on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788ff on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bf0 on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a560a on (4:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb78bf1 on (6:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb78bf2 on (6:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x8a560b on (4:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef720 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5f343f on (9:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef723 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb79178 on (7:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef8ab on (1:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb79179 on (7:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): tasks terminated
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb7a2c1 on (2:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): tasks terminated
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): tasks terminated
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef722 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): tasks terminated
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb7a2c9 on (2:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef721 on (5:5:0) aborted
<freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1991-05.com.microsoft:desktop-g6fn6t0): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:20 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1991-05.com.microsoft:exc01.domain): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): waiting for CTL to terminate 8 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): waiting for CTL to terminate 6 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): waiting for CTL to terminate 7 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): waiting for CTL to terminate 8 tasks
Dec 23 14:19:31 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): no ping reply (NOP-Out) after 5 seconds; dropping connection
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78beb on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bec on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bed on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bee on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bef on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x5f343d on (9:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x5f343e on (9:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788f8 on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788f9 on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fa on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5605 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5606 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5607 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5608 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a5609 on (4:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fb on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fc on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fd on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788fe on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb788ff on (8:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0xb78bf0 on (6:5:0) aborted
Dec 23 14:19:31 <freenas_server_name> ctl_datamove: tag 0x8a560a on (4:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb78bf1 on (6:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb78bf2 on (6:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x8a560b on (4:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef720 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5f343f on (9:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef723 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb79178 on (7:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef8ab on (1:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb79179 on (7:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx04): tasks terminated
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb7a2c1 on (2:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): tasks terminated
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx02): tasks terminated
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef722 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> WARNING: <ip_address> (iqn.1998-01.com.vmware:esx03): tasks terminated
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb7a2c9 on (2:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef721 on (5:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef8ba on (1:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0xb7a2d3 on (2:5:0) aborted
Dec 23 14:19:32 <freenas_server_name> ctl_datamove: tag 0x5ef8ba on (1:5:0) aborted

 

[dlavigne]

Were you able to make any headway with this?

 

[Pointeo13]

I didnt make any headway with the issue, to prevent the problem right now I ordered another six 5TB drives to avoid deleting anything. This week for the heck of it I am going to shutdown all my vm's, turn off cifs and iscsi to see if I run into the issue.

 

[Pointeo13]

Ok finished my testing, seems that stopping the cifs and iscsi services didn't have much impact on the issue, my ssh connection would freeze up each time I ran the command rm -r through the console on a directory that's 48.6GB. The only thing i can say is that when cifs/iscsi was disabled, the ssh console recovered a bit faster but thats pretty much it.

First Box (Green) = iscsi/cifs service has been running for 16 days and ran the rm -r on the 48.6GB directory
Second Box (Blue) = iscsi/cifs service has been stopped and ran the rm -r on the 48.6GB directory
Third/Firth Boxes (Brown) = cp -r the 48.6GB directory so I could delete them again
Fourth Box (Orange) = wanted to run the test again with iscsi/cifs service stopped and ran the rm -r on the 48.6GB directory
Sixth Box (Purple) = iscsi/cifs running again and ran the rm -r on the 48.6GB directory

Does anyone have any other ideas I can try? My next step might be to reinstall freenas, and re-setup the freenas configuration from scratch and then import my pools.

 

[Pointeo13]

I finally have time this week to try to swap out the USB and install a fresh version of FreeNas to see if that resolves the issue. I once again almost hit 80% on my (Raidz_2_SATA) storage, so I shutdown all my vm's that live on (Raidz_VMware_SAS) and went ahead and deleted 10TB of Data through the console, which was a painful process. During that time all my iscsi connections, cif's share, ssh connection, webpage would freeze up for a minute or two and then start working again for about 30 seconds and then begin to freeze up again a minute or two, this went on for maybe an hour or so to delete all 10TB?

 

[toadman]

While it's probably not your issue, why is the L2ARC (98G) 4x *smaller* than the ARC (435G avg)? You want it the other way around right?

 

[Pointeo13]

I normally have over 1TB of memory in my system, had to remove a module for testing, but that ARC your referring to is memory Freenas is using for read cache, I want it to use as much as possible.

EDIT: If I still am not able to figure it out after this week, does anyone know if you can pay a one time fee for a single support ticket?

 

[toadman]

Yep, I'm aware of what the zfs arc is and why you want the most memory you can stuff into the system. I was just curious why there is an L2ARC in the system that's got a hit rate of 0.8%. You'd probably be better off just taking the L2ARC out completely unless you want to make it 4x or more the size of the ARC. But I'm not sure that leaving it in is doing any actual harm performance wise that you can measure, but I could be wrong. (EDIT: I think you are probably unnecessarily magnifying writes, as evacuations from the ARC are going both to the pool and the L2ARC. And since the hit rate is so low on the L2ARC, you're not gaining anything and just suffering the write amplification.)

There are a few bug tickets that talk about the output you are getting. One of them may be applicable to your case. (or maybe one was from you when you originally saw the issue?)

I wish I could be of more help.

 

[Pointeo13]

Couple things to see if this makes sense, L2ARC is being used for the (Raidz_VMware_SAS) only which has all my VM's. Where I am doing the mass delete is on (Raidz_2_SATA) all my large archive files, which is whole different physical storage device on it's own LSI card. If I do a mass delete within a guest VM that's using the (Raidz_VMware_SAS) I have no issues, but if I go into the console and do a huge delete on (Raidz_2_SATA) OR (Raidz_VMware_SAS), that's when my freenas box starts maxing out it's CPU and freezing up.

It's almost like if I do the delete through a console, cif's or ssh it kills the box, but if I do a mass delete through a guest OS that's using a datastore over iscsi it's fine.

From what I understand you only start considering L2ARC when you have more then 64GB of RAM
https://forums.freenas.org/index.php?threads/at-what-point-does-l2arc-make-sense.17373/

Also the system has only been up for two day's so the L2ARC is still growing. I have also checked out the bug tickets, odd thing is that the ctl_datamove error more of a symptom because of the box freezing up and not from a bad switch configuration or because of the Intel 10Gbps Nic's. Here is the thing, I can have all the services stopped by shutting down cifs, iscsi, and network connections, log onto the console and do a rm -rf /mnt/Raidz_2_SATA/storage/trash and the freenas server will begin to freeze up as it deletes.

In the past I removed L2ARC and SLOG from the picture and still had the issue.

Update: I went ahead and took the same files that I have been testing with, 48.6GB worth of files and tried to delete the folder on the VMware Guest OS and the root of the vmware datastore, I didn't have any issues. So when the deletes are done using iscsi it's fine. This method works for both (Raidz_VMware_SAS) and (Raidz_2_SATA)

 

[Pointeo13]

Here is something interesting, I took a file that's 1.79GB and then delete the file with no issues. Next I took this file again and cloned it twice, which is a total of 3.6GB and no issues, I keep doing this and here is what i found. These deletes are all done on the console itself, not using ssh or cifs

1.79GB (1 file) = delete with no issues/errors
3.60GB (2 files) = delete with no issues/errors
5.40GB (3 files) = delete with no issues/errors
7.20GB (4 files) = delete with no issues/errors
9.00GB (5 files) = delete with no issues/errors
10.8GB (6 files) = I'll maybe see one error of "no ping reply after 5 seconds etc."
12.6GB (7 files) = get few more errors "no ping reply after 5 seconds etc." but didn't see any "ctl_datamove errors" yet
14.4GB (8 files) = now even more errors "no ping reply after 5 seconds etc." but no "ctl_datamove" errors yet
16.2GB (9 files) = same amount of error of the 8 files but no "ctl_datamove" errors yet
18.0GB (10 files) = more errors then the 9 files but no "ctl_datamove" errors yet
19.8GB (11 files) = no change just more errors but still no "ctl_datamove" errors yet

I did test with 12 and 13 files but really no change just more errors like the 11 files test

25.2GB (14 files) = lots of "no ping reply after 5 seconds," and finally now getting a few "ctl_datamove" to the point where the freenas is becoming unresponsive. From their it just gets worse and worse the more files I delete.

I went through this test 3-4 times to make sure the results are consistent.

 

[Pointeo13]

OMG, upgrading to FreeNAS-9.10-STABLE has fixed the issue! Before this I changed out from a IBM to HP, I tried replacing all LSI controllers, I tried replacing storage device with new SAS backplane. I went from USB 3.0 back to 2.0 with fresh install. Six months of not being able to really delete anything above 20GB and now I'M FREE TO DO SO!

 

[Dudleydogg]

Well here's a Twist I never had that error Till I updated to 9.10 so I get the ctl errors. So opened bug report to see what is issue.

 

[Brentnall]

Well here's a Twist I never had that error Till I updated to 9.10 so I get the ctl errors. So opened bug report to see what is issue.

I'm also seeing such errors since the upgrade to 9.10. They always seem to happen (along with freenas hanging for short periods, as someone already described higher up this thread) when writing lots of data to iscsi LUNs.

I have a few VM's in a zvol sitting on top of a samsung SSD. This is attached as a datastore to ESXi via iscsi. I've noticed these error when suspending VM's and when copying vmdk's to the datastore.

As this is probably the wrong place to continue on this 9.10 issue, do you think you could point me towards the bug listing you opened? Unfortunately you cant seem to search by username on the freenas bug tracker site (either that or I'm blind!)

 

[Dudleydogg]

They reviewed my Logs and saw one of my switch ports was auto negotiating from 100 to 1000, and when it went to 100 the LAGG would disable that member so as that process happened I got the Errors.
I since hard set my Switch ports to be 1000f (Dell POE Switch) and this subsided those errors. They also suggested I use MPIO so I am considering that if no performance is lost in the network.

Summary all ports in your LAGG must be set to 1000 and not Auto negotiate.

Here is the link to the solution and the suggestion for MPIO this thread may be private sorry if the link fails.
Bug #14932: ctl_datamove: tag aborted