Inconsistent Performance

[kahlid74]

Good afternoon. I'm hoping peeps have an idea of where I can look to drill deeper into the inconsistent performance I am getting.

My Setup ->
- FreeNAS-11.2-U4.1
- CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz (40 cores)
- Memory: 64 GiB
- 10Gbe NIC
- 10Gbe Ubiquiti Switch
- Twelve 10TB disks
-- 24 Port SAS Expander
-- Single MiniSAS cord (2400Gbps speed)
-- Single Pool
-- ZFS2 (Raid 6) for TWO vDEVs
--- Each vDev has 6 disks.

File Server
- Windows 2016 box
- Dual Xeon Procs
- 32 GB mem
- Intel 10Gbe nics on Windows Server 2016 box doing transfers

Here's the shake down. I am transferring from a QNAP 8 DISK SAN to this Freenas box. I have the 12 Disks connected through a single MINISAS connector on the backplane. I also have a Synology 8 disk SAN as well. I am transferring 4k content, minimum file size 20GB, maximum 89GB. So no small file throttle up/down.

Here's how things look:
- 8 bay QNAP to local SSD -> 189 MBs sustained. this is what I expect for these SSD throughput.
- 8 bay QNAP to 8 bay SYNOLOGY -> 443 MBs sustained. This is what I expect for these types of SANs. Their processors maxs at this full throughput
- 8 bay QNAP to 12 bay FREENAS -> 80-330MBs fluctuating. I would expect ~300 MBs (2400Gbps MINISAS / 8 == 300MBs of max throughput)

I'm really struggling with why this is fluctuating. All other disks get sustained and maintain it but the brand new Freenas, with monster settings can't keep 300MB to save it's life. Where do I start looking? It gets 300MB for seconds and then kind of slows down, which brings me back to where I was years ago when I tried FreeNAS and was dissapointed in it's performance. What could be causing it to ramp up a down, when the files are very large and the other disks and arrays just sit at a consistent throughput? What could I be missing?

 

[drros]

Is network performance ok? What is results of iperf between FN and Windows box?

 

[kahlid74]

Is network performance ok? What is results of iperf between FN and Windows box?

Thanks for the reply! Network performance should be good to go, because transfers between the QNAP and the Synology are through the windows box. So full rate on the QNAP is in the 400-500 MBps range due to the processor in the frame, and then 400-500 in the Synology as well because of the CPU in that one too. The Windows box only have one 10Gbe link in the SAN network, so it's in the 800-900 MBps range when transferring between the QNAP and Synology. I can see this on the NIC and on the switch.

My apologies, but can you dive a bit deeper on iPerf please? I have an idea of what you are asking, but I'm not sure if what I'm thinking of is what you're asking about.

 

[kahlid74]

Additionally, I started a transfer of 38TB about 2 days ago. This transfer should have taken ~1.5 days at max speed, 300MBs. It's over 2 days and now it's doing a 0.0004MBps then a 300-400 MBps and then back to a 0.0004MBps. What the heck is going on.

 

[kahlid74]

FreeNAS is no longer updating the NIC bandwidth on the main page, and the copy has ground to a halt. Something is a foot.

 

[kahlid74]

Here's the graph before I canceled the job. There should be no spikes, it should just be flatline ~300MBs. The spikes lead me to believe FreeNAS can't handle the incoming load and is doing something to compensate.

 

[kahlid74]

I turned off compression, hoping that would be it, but it was not :(. The problem has persisted.

 

[kahlid74]

Is network performance ok? What is results of iperf between FN and Windows box?

Per the iPerf below, network is all gravy. So this is straight up FreeNAS doing something abnormal I take it.

 

[kahlid74]

Additionally, with DD, it hits ~800+ MBs.

 

[xness]

Have you ever solved this issue?