Deleting files causes 100% CPU Usage

[Pointeo13]

I'll give out spec's if that helps, but was wondering if anyone has ever experience an issue where if you delete a bunch of files, 20GB or more, your CPU maxes out and causes your cifs/vm's/box to freeze up for 20 seconds or so. This happens when I delete files on cif share or even if I directly ssh on the freenas server and do rm -r command, it's like the freenas box locks up and during that time i lose graph/reporting details and I can't access the gui and my ssh session is locked up.

Server: IBM x3850 X5
CPU: 4x Intel Xeon E7-4830 2.13Ghz
Memory: 1048528MB
Raid Card - 2x LSI M1015 (IT Mode)

FreeNAS-9.3-STABLE-201511020249

NAME: Raidz_2_SATA
SIZE: 97.5T
ALLOC: 68.2T
FREE: 29.3T
FRAG: 17%
CAP: 69%
DEDUP: 1.00x
HEALTH: ONLINE

 

[HoneyBadger]

That's definitely unusual. Async destroy was added quite some time ago.

Start up some logging through an SSH session and then do a delete, see if it continues logging or even being interactive. It shouldn't be locking up any machine, let alone your Skynet-in-a-box there.

 

[Pointeo13]

I went into the /var/log/messages, it looks like it continued to log their during the issue, does this mean anything, being spammed over and over.

Nov 12 15:10:00 ServerName ctld[8507]: child process 27378 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27377 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27376 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27375 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27374 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27373 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27369 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27368 terminated with signal 13
Nov 12 15:10:00 ServerName ctld[8507]: child process 27364 terminated with signal 13
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cd4 on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cd5 on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cd6 on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cd7 on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cd8 on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cd9 on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cda on (2:4:0) aborted
Nov 12 15:10:02 ServerName ctl_datamove: tag 0xfc7cdb on (2:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0xfc7cdd on (2:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0xfc7cde on (2:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0xfc7cdf on (2:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae07 on (3:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae08 on (3:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae09 on (3:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae0a on (3:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae0b on (3:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae0c on (3:4:0) aborted
Nov 12 15:10:03 ServerName ctl_datamove: tag 0x191ae0d on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae0e on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae0f on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae10 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae11 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae12 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae13 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae14 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae15 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0x191ae16 on (3:4:0) aborted
Nov 12 15:10:04 ServerName ctl_datamove: tag 0xfc7ce3 on (2:4:0) aborted
Nov 12 15:10:04 ServerName ctld[8507]: child process 27388 terminated with signal 13
Nov 12 15:10:04 ServerName ctld[8507]: child process 27384 terminated with signal 13
Nov 12 15:10:04 ServerName ctld[8507]: child process 27383 terminated with signal 13
Nov 12 15:10:04 ServerName ctld[8507]: child process 27371 terminated with signal 13
Nov 12 15:10:04 ServerName ctld[8507]: child process 27366 terminated with signal 13

 

[Robert Trevellyan]

ctl_datamove: tag 0x191ae15 on (3:4:0) aborted

Googling those messages indicates that you probably have something misconfigured between ESXI and iSCSI. Maybe fixing that will fix the lockups too.

 

[tvsjr]

32 cores and 1TB RAM? Wow.

 

[Pointeo13]

Googling those messages indicates that you probably have something misconfigured between ESXI and iSCSI. Maybe fixing that will fix the lockups too.

I checked my network equipment and iscsi setup and couldn't find any misconfigurations, but I was able to fix the issuing by doing ifconfig ixo -tso, if anyone is wondering, I'm using four Intel X540-T2

 

[solarisguy]

@Pointeo13, how did you figure it out that it was TCP Segmentation Offloading not working properly?

 

[Pointeo13]

@Pointeo13, how did you figure it out that it was TCP Segmentation Offloading not working properly?

Posts like these https://bugs.freenas.org/issues/3201 at first I was googling the wrong terms, so I ended up doing searches for ctl_datamove: tag

The next thing I'm going to try is re-enable tso and try mtu 1500 to see if I get the same results.

 

[solarisguy]

Posts like these https://bugs.freenas.org/issues/3201 at first I was googling the wrong terms, so I ended up doing searches for ctl_datamove: tag

The next thing I'm going to try is re-enable tso and try mtu 1500 to see if I get the same results.

@Pointeo13, thank you! Reading https://bugs.freenas.org/issues/4560 made me think that 9.3 would not experience the issue.