Slow GUI, Server freezes when deleting lots of data

[cuvy]

Hi there,

I'm experiencing very odd issues with one of our FreeNAS instance. First, the hardware:

Code:

Machine class: amd64
System architecture: amd64
Machine model: Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
Number of active CPUs: 8
Number of CPUs online: 8
Current CPU frequency: 3.52 Ghz
Physical Memory: 31.94 GiB

• HBA - 2x SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
• Network: Intel I350 Gigabit Network Connection
• Motherboard: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F
• HDD: 96x -- 2.5TB, 3TB, 4TB Seagate and WD

The HDD are connected using 2x 4U Supermicro Chassis and the server itself is a 2U Supermicro server. Both chassis are connected to the server using a SAS cable.

My problem is, the FreeNAS gui is slow. It takes a while to respond, after I login it, it feels slow to load the page. If it was just that, I wouldn't care. Now I've been doing some cleaning and I'm deleting about 10TB of data and while it is deleting, the server just gets unresponsive. We just can't access the server and the delete command starts fast then slows down very quickly, almost to a halt. A CTRL+C to cancel the delete takes minutes to stop. It just seems that it is hanging somewhere.

The server logs are reporting no problem, I'm doing regular scrubs (twice a month) I have SMART checks for all HDD 4 times a month. Last time I resilvered a volume it was very fast, it took 24 hrs for 120TB where another server we have took 3 days for a 140TB.

While I was deleting the 10TB data, the CPU is almost 100% idle, there is a lot of memory available. I also monitored the zpool IO using zpool iostat -v 3 and the max operations read+write was less than 100 per HDD and the bandwidth read+write never exceeded 2MB per HDD.

When it gets VERY slow, the operation and bandwidth IO of the pool is almost idle.

I'm looking for tips and tricks in order to debug this issue.

Any help will be very appreciated.

Thanks!

 

[cuvy]

The server just rebooted itself just now. Happened a few days ago as well...

 

[dlavigne]

Sounds like a hardware issue needs tracking down...

 

[cyberjock]

Two things:

1. If you are deleting lots of files from the CLI, you can expect the server to be almost unresponsive. The I/O you mentioned and the bandwidth are totally consistent with someone deleting lots of files. Gotta round up lots of small metadata fragments and perform the delete operation. This is a high-io and low-bandwidth operation. The CLI deletions are particular hard on the server because they take up a larger slice of the zpool's IO than Samba and others. Mostly because when using Samba each operation that a user wants to perform has latency (network latency, processing of the packets, etc.) where the CLI deletions have fractions of a millisecond, so they totally overpower the relatively few Samba requests that are coming in. So from that angle, sounds like nothing is wrong and this is totally expected.

2. Now the reboot is a different issue entirely, and should *definitely* be corrected. Could be a software or hardware issue. Based on the size of the server it's anyone's guess because large-scale servers like yours have different needs than the smaller servers that 99.99% of the forums "get" to have at home. :P

96 drives that are 2TB+ is *way* too much disk space for 32GB of RAM, so you are most certainly starving your system's ARC. I wouldn't even try less than 128GB of RAM on a zpool of that size. Also a zpool with a starved ARC will perform poorly, particularly when doing things like file deletions and such.

 

[cuvy]

This is very helpful, I'll upgrade the memory and see if it changes anything.