Upload Speed Drops to 0 Randomly

[dontknowwhatimdoing]

So, I have a Dell Poweredge R720XD running truenas with 2 Raidz2 Vdevs with 8 drives per vdev. I setup an SMB share and mapped it to my windows computer, but when I try to upload my files, it will randomly drop to 0 bytes/second for a minute or two every few minutes, then return to 120Mb/second. The Drives I am using are Dell (seagate) enterprise SAS drives, and I am almost certain they are not SMR (ST91000640SS). I tried disabling hardware offloading per another thread that seemed to have a similar issue but there was no change in results.

 

[c77dk]

Please tell some more details about the setup - both hardware (Especially the HBA will be a place to look) and the datasets.
At the moment there's too many variables to pinpoint issues.

One thing I need to ask at once is: are you using deduplication?

 

[dontknowwhatimdoing]

Please tell some more details about the setup - both hardware (Especially the HBA will be a place to look) and the datasets.
At the moment there's too many variables to pinpoint issues.

One thing I need to ask at once is: are you using deduplication?

so, I have one dataset that encompasses the whole pool, I am not sure what else you want to know about that. The Hardware is 24gb ECC ram and 2 intel Xeon E5 2670s. The HBA is a Dell PERC H710 mini that's been flashed to IT mode with boot image. I have a boot pool running on 2 250gb crucial SATA SSDs on the rear backplane and 16 ST91000640SS 1TB seagate Enterprise SAS drives.

I have absolutely no Idea what deduplication is or if I am using it. if its a default it is probably in use, if it is something I would have had to intentionally turn on, I do not think I did that.

Edit: I am using a Dell 165T0 Broadcom 57800s NIC that is currently connected to my mikrotik switch over SPF+ fiber channels at 10Gb/s but am trying to copy files from my PC that's only 1gbps copper.

Update: looking at the dataset settings, ZFS deduplication is set to inherit(off)

 

[c77dk]

hmmm, on the surface it looks ok - though I don't have any experience with the broadcoms - someone else might know more.

Have you tried looking at the disk activity during transfer? With gstat you can see how busy they are.

 

[dontknowwhatimdoing]

hmmm, on the surface it looks ok - though I don't have any experience with the broadcoms - someone else might know more.

Have you tried looking at the disk activity during transfer? With gstat you can see how busy they are.

so, using gstat this is the drives mid transfer

they will occasionally bounce up to something more like this

but never actually show any level of activity until the transfer jumps up to 110 mbps or so, which is the theoretical cap of my connection on the upload.

 

[dontknowwhatimdoing]

This is gstat when the transfer jumps up, not sure if it's helpful, sorry, you caught me in the middle of one of the luls and I wasn't sure when it would go back to full transfer speed. it seems to be getting worse.

 

[dontknowwhatimdoing]

after quite a bit of troubleshooting, I started thinking this wasn't the truenas server and started looking to windows. As it would turn out, on my PC I am copying files from a small RAID-5 array that has an L2 cache in front of it using Primocache software. when I looked at the array activity during the transfer, even when the transfer was at 0, the drives were at 100% but the response times of the drive was 2-3 seconds. turned off primocache and those dropped to 2-3 milliseconds and the transfer started not having those issues.

 

[jgreco]

Well, good to see a resolution. I was going to circle around and be critical of the Broadcom, but they're not always at fault.