Mass deletes rendering my server useless

[fullspeed]

I've had this happen a bunch of times, basically one of our DBAs will initiate a powershell script which searches for and deletes a bunch of files.. in this case 75TB approx and this is what happens:

Inbound and outbound transfers get kicked right in the teeth.. inbound completely cut off

Everything grinds to a halt and some backups that were running failed. I

 

[cyberjock]

What version of FreeNAS are you running?

And if you do "zpool status" does it say anything about not all feature flags are enabled? Or to ask the same question another way, do you need to upgrade your zpool?

 

[fullspeed]

FreeNAS-9.3-STABLE-201512121950

I can't do a zpool status, currently I'm getting hours worth of this.. I can pull disk tmw when I get in but im concerned.

Why would a non OS disk cause the OS not to load? I saw an earlier thread where someone had this problem after some searching

 

[Johnny Fartpants]

When did you start getting these errors? You know I kept asking the other day about if you had any issues with your 45 bay JBODs in particular replacing disks in the back and you said no everything is fine well this is EXACTLY the issue I've seen on a number of the same Supermicro 45 bay JBODs only when replacing disks in the back and it transpired to be an internal cabling issue within the JBOD itself. See this thread when I give a more detailed explanation.

https://forums.freenas.org/index.php?threads/server-unavailable-all-of-the-sudden.35194/

 

[fullspeed]

When did you start getting these errors? You know I kept asking the other day about if you had any issues with your 45 bay JBODs in particular replacing disks in the back and you said no everything is fine well this is EXACTLY the issue I've seen on a number of the same Supermicro 45 bay JBODs only when replacing disks in the back and it transpired to be an internal cabling issue within the JBOD itself. See this thread when I give a more detailed explanation.

https://forums.freenas.org/index.php?threads/server-unavailable-all-of-the-sudden.35194/

Alright so I think i got it fixed. It was obvious that a disk was bad but I was really struggling to find it, this is partially because we kept throwing in disks so fast with no labels and I rely 100% on the lights blinking to tell me whats going on and what drives to replace.

It may sound silly but it's never let me down, including today. I was really pulling my hair out trying to figure out what disk it was and Freenas would NOT boot regardless of what I did.

So this server has four arrays back and front full of drives, but I see a VERY light blinking coming from one disk. It was hard to see because the sun was right on it.

I yanked that drive and everything unspooled, now ZFS is doing its magic. If someone wants to drop some knowledge on exactly what its doing that would be helpful. I am assuming integrity checks on the snapshots?