Weird performance improvement (and loss again)

[Rand]

Hey guys,

can somebody explain this behaviour?

I am still running write performance tests (on 9.2.1.Rc2 atm) via cifs.

So usually the speed is around 60/65MB/s but (i believe since RC2 since i didnt see that before) it shoots up to 95+MB/s for a few minutes after a few minutes of copying...
I have no clue whats causing this - any advice on how to track this down?

I am running on an E31230v3,16GB Ram, Intel Nic on FreeNas, Realtek on Client (i know;)), pool is an 8 drive (4x2mirror) of sata drives on 2 LSI2008 HBAs (4ea), dd write perf is >500mb/s

I used NAS performance tester (which might not be the best tool but it allows me to run a large amount of random data for several runs).

The improved speed works for a couple of minutes, then falls back to original values only to return later...

Any ideas??

Thanks,



Nas Tester

NAS performance tester 1.5 http://www.808.dk/?nastester
Running warmup...
Running a 1000MB file write on \\nas2\test2 40 times...
Iteration 1: 63,09 MB/sec
... All around 65MB +-3MB
Iteration 20: 62,48 MB/sec
Iteration 21: 63,59 MB/sec
Iteration 22: 64,54 MB/sec
Iteration 23: 64,83 MB/sec
Iteration 24: 64,03 MB/sec
Iteration 25: 98,41 MB/sec
Iteration 26: 97,19 MB/sec
Iteration 27: 98,67 MB/sec
Iteration 28: 98,72 MB/sec
Iteration 29: 96,67 MB/sec
Iteration 30: 92,38 MB/sec
Iteration 31: 96,16 MB/sec
Iteration 32: 98,13 MB/sec
Iteration 33: 97,08 MB/sec
Iteration 34: 95,61 MB/sec
Iteration 35: 95,96 MB/sec
Iteration 36: 97,27 MB/sec
Iteration 37: 92,66 MB/sec
Iteration 38: 94,79 MB/sec
Iteration 39: 97,00 MB/sec
Iteration 40: 87,86 MB/sec
------------------------------







Windows Lan Performance:


FreeNas Disk (1 of the eight, all identical; peaks show increased speed)


Similar behavior on network



Memory is in constant use

Server Load 1


Load 2

 

[cyberjock]

Well, for starters.. you've got so many variables there's no way we're going to give you good feedback. You've got source and destination NIC speeds, LAN speeds, pool performance, etc.

How about you test between your server and desktop with iperf. Do the standard 10 second test and post the results.

 

[Rand]

Hey cj,
thanks for looking.

Just for general information, I have filed a bug/help report as well for this #4145

While i agree that iperf is a good starting point i dont think it will help here:) (But if you really think it matters i will run it :))

I have tested this from 3 different clients (Desktop, Laptop, Windows server next to FreeNas Box) and the behaviour is identical, across Nics as well (realtek on the desktop, broadcom on the others), will try to add intel nic to desktop soon, tried with directly connected laptop as well;

The issue here is not that the components cannot perform as expected but that they only do it sometimes and othertimes they dont and i am not able to find the reason why its working as desired or not.


I've run a 5x64GB write Transfer test (random data), NasPerformanceTester, yesterday from the desktop withan average transfer rate of >100MB, so the capability is there, even for very large transfers.
On the other hand i copy a 4gb test file and its only using 65mb/s (repeatedly9).

Once i copied another 64gb file and it dropped of speed after half (32gb) - thats double the ram on the server...

What can cause this?
If this wold happe after 15gB (potentially max arc) then i could understand it, but so i cant see a reason; all components *can* deliver but they just do it sometimes not always.

Edit: Found the screenshot of the 32GB dropoff - this is a 64GB random data transfer

 

[DrKK]

Why do you say "iperf is not a good starting point"?

 

[Rand]

Lol,
i didnt say that.
I said that I dont think its needed here b/c the network has shown to be of acceptable performance ?;)

I mean its ok for me to transfer 300GB at 100MB/s from the desktop... or do you still think the network could be the issue?
If you do i will run iperf, no problem:)

 

[cyberjock]

It has shown to be of acceptable performance, but it has not necessarily been shown to be of acceptable performance reliably. In theory, they are the same. In practice, they often vary by a long shot.

And to be honest, I didn't post originally because nastester has proven to be very unreliable, not accurate, and not a good estimate of performance. I can't tell you how many people have used it, claimed to get poor numbers, yet when I do a teamviewer session and moved data to their server at 100MB/sec consistently. We deliberately discuss iperf and not nastester for many reasons in the manual. :)

 

[Rand]

Fair enough.
@NasPT - i am aware of the issues with it but it is a simple way of generating distinct noncacheable data in large quantities;)


... Hmm maybe you guys *are* up to something...
All are best out of 3
Desktop -> FreeNas 544 Mbit - Laptop with Broadcom nic is similarly bad ... attached it to same switch as FreeNas, same result...
Desktop -> W2k8Server 711 Mbit
W2k8Server-> FreeNas -> 890 Mbit

FreeNas -> Desktop 807 Mbit
FreeNas -> W2k8Server 903 Mbit
W2k8Server-> Desktop 709 MBit

I ran iperf right at the start and didnt remember it being so bad... must have been the other laptop with intel nic :/

I will try the Intel Nic in the Desktop after i am back from a business trip; hopefully that does increase things.
Then i guess I'll have haul the Server up the stairs to crosscheck whether its the Desktop Switch (Cisco SLM2008), or the ethernet cables or both.

Still dont understand what kills performance sometimes... and at such consistency to 2/3's of max...

I will report back when i have new info - if you have ideas what else to test or how to further track down potential network issues i am all ears.

 

[cyberjock]

Fair enough.
@NasPT - i am aware of the issues with it but it is a simple way of generating distinct noncacheable data in large quantities;)

I'm sorry, but there are other methods that work just as well(read: better IMO) for generating non-cacheable data in large quantities. That's not unique to net performance tester at all. And the net performance tester does too much testing all at once for it to be diagnostically significant in isolating a problem. It *may* be able to prove there *is* a problem. But that's diagnostically different from diagnosing the problem. You can defend your application all you want. I've been around these parts long enough to know that nobody has ever diagnosed a problem.

To be completely honest(and brutal), I don't really spend much time trying to diagnose problem that appear on iperf unless both ends of the network are Intel nics. Usually Broadcom works just as well, but are often harder to find or more expensive. It's really just not worth my time to spend hours trying to help diagnose a problem that probably 90% of the time is fixed with $25 NICs. I've advocated for them in my guide, they're mentioned in the manual, they're mentioned all over the forums. If you want to spend days/weeks troubleshooting a problem created because you wanted to save a few bucks go for it. I won't spend my time on it. ;)

 

[Rand]

Ah i do hear you;)
I have already added an Intel PT adapter to the desktop but didnt get around testing it last night.

 

[ixdwhite]

Ah didn't know you were updating this thread as well as the bug (#4145).

The traffic graphs are _very_ interesting since it looks extremely abnormal. I see the toothyness on my 4-disk RAID10 test rig -- that is the dropouts I've been explaining -- but not a 30% delta between transaction groups. That looks a lot like some sort of throttling or change in the workload generation method, especially with it just coming and going at very distinct boundaries. Nothing in FreeNAS commands that type of throughput change.

Are you running a firewall or using a switch that other things on it or has a traffic management policy in place? Is there background tasks running on the test client, or a screensaver? Or a scrub, periodic snapshot, or replication on the FreeNAS side? Are you getting ZFS or disk errors on one of the disks that might cause it to get kicked out of the array?

We need to eliminate possibilities here...

 

[Rand]

I update whereever anyone has a new idea;)

No firewall, there shouldnt be any traffic shaping on either of the two switches, no screensaver, no noteworthy background tasks visible;
No other tasks on FreeNas either.
Potentially the server board could throttle fir thermal reasons (thats currently not deactivated), but disks run around 28° so i dont think that should be it but i will deactivate that stuff in the servers bios for testing (along with any green policies in the switches for the relevant ports)

 

[ixdwhite]

I ran some tests last night on a RAID10 and wasn't able to produce anything like you are seeing when copying files around with Explorer. I'm trying to do some testing with iozone (under cygwin) and IOmeter but its giving me odd results, so I'm trying to figure out what I have set up wrong on it. (I wish FreeBSD or Linux had a decent CIFS client .. grrr)

 

[ixdwhite]

FWIW I was just reminded that there are 2 forms of write tests -- extending an empty file and overwriting an existing file. The performance difference between the former and latter is about 25%. If your test tools create a file the first pass then overwrite it on later passes then that might explain the run to run difference.

I'm going to run a full perf sweep between 9.2.0 and 9.2.1 since there seems to be a massive write performance regression lurking in there somewhere but I need hard numbers as evidence.

 

[Rand]

>FWIW I was just reminded that there are 2 forms of write tests -- extending an empty file and overwriting an existing file. The performance >difference between the former and latter is about 25%. If your test tools create a file the first pass then overwrite it on later passes then that >might explain the run to run difference.

Unfortunately i experience the same on real data as well; and sometimes during a copy; and sometimes on the first copy...

But a nice idea:)

 

[Rand]

Ah i do hear you;)
I have already added an Intel PT adapter to the desktop but didnt get around testing it last night.

So I've tested the intel nic > freenas 780 Mbit... quite a bit better.

So will run some more tests with that one then;)

 

[ixdwhite]

For those of you not following the bug,

I just found that on my test E3-1270 v3 system that turning off HyperThreading fixed the write performance issues entirely. I went from very poor, unstable throughput (20-50MB/s) to saturation levels (110MB/s +) by turning off HTT. Read performance was never a problem.

This likely means there is inter-thread or lock contention that is exacerbated on a fast core system with more than 4 cores (or at least E3-12xx v3). Next step is to run with lock profiling to figure out what is special in that setup and see what can be done to mitigate it.

Haswell cores are particularly fast so this likely doesn't come up on Sandy/Ivy Bridge. It wasn't seen on dual-core Haswell desktop CPUs either (i3-4130T specifically), for the record. I don't have a i5/i7 Haswell chipset board (or CPUs) to test higher end configs yet.

 

[Rand]

intel nic
So will run some more tests with that one then;)

The Intel nic is better - as expected, but not necessarily by the large margin that one might think.
After all other testing is trough i might run some direct comparison to check the actual difference.

Another setting which surprisingly seemed to be more relevant is the Dell Switch Green Ethernet configuration

I deactivated the Link Short Reach Detection (left inactive port detection on) and that reduced performance degradation by quite a margin (now at 80-90 MB/s) instead of 60-70. (Its either that or LACP;)).

Will test HT impact next

 

[Rand]

Well i've transfered around 25 iterations à 64GB with NasPT now and no massive drops visible with HT disabled ... variations +- 3MB/s... Looks good.

 

[Rand]

Lol, new day new fun - can anyone explain *this* pattern?;) (2 disks, but weird speed, fun graph though :p)

 

[cyberjock]

Wow.. a line graph with no x or y units or quantities. Can I guess that's my mom's IQ over time? What did I win?