Performance issues following update to FN11

[Skuloth]

Hey all,

Over the last week I've been dealing with some sudden performance issues with my freenas box. I have a hard time believing its related, but my performance issues began as soon as I updated to freenas 11 from 9.10 u4.

I dont have benchmarks from before hand, but I could fully saturate my 1 gig connection transferring files to my desktop from the server, and now it struggles to hit ~60MB/s. On FN11 the webui was incredibly sluggish taking 15-20 seconds to load panels compared to nearly instant previously.

I dropped back to 9.10u4 after a scheduled scrub completed (which returned no errors) to better webui performance and similar disk performance, but still significantly worse than pre update.

As far as config changes, nothing has been done aside from the obvious update to 11. I dusted out the box between the scrub completion and the drop back to 9.10 and will be running memtest when I take it down tomorrow or wednesday to replace the fans in my case.

Hardware:
Supermicro X10SDV-4c-TLN2f-0
Xeon d-1521
2x16gb samsung DDR4 ecc ram
6x8TB WD reds in raid z2

I've checked smart data and plan to keep an eye on it over the next couple weeks, but saw no issues. Temps are within reason. Will be running memtest like I mentioned.

I'm mainly looking for suggestion about what to look into since I'm out of ideas currently.

Any help would be appreciated
-Skuloth

 

[dlavigne]

Anything interesting in /var/log/messages? What is the capacity of the pool?

 

[Skuloth]

The pool is 6x8tb in z2 so ~29 TB usable

/var/log/messages doesn't show anything particularly interesting, but it does show a bunch of stuff like these 2 lines. Which iirc were related to a known bug.

Code:

Jul 17 00:31:13 freenas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name				 
Jul 17 00:31:13 freenas alert.py: [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name

 

[Ericloewe]

Check the networking with iPerf. We need to narrow this one down.

 

[Skuloth]

Sorry for the delayed response, been running memtest since yesterday evening.

Returned no errors tho.

Code:

------------------------------------------------------------					
Server listening on TCP port 5001											  
TCP window size:  128 KByte (default)										  
------------------------------------------------------------					
[  7] local 192.168.1.25 port 5001 connected with 192.168.1.16 port 2300		
[ ID] Interval	   Transfer	 Bandwidth									
[  7]  0.0-10.0 sec  1.10 GBytes   942 Mbits/sec									

 

[Ericloewe]

So iperf is fine... Unfortunately, that leaves me somewhat devoid of smoking guns.

Though the more interesting question than the capacity of the pool is how full it is.

 

[Skuloth]

12.7 TB used
15.4 TB available

so a little under 50%

 

[Skuloth]

At this point I'm demonstrating how bad at computers I am to myself.

Turned on autotune, cause I couldnt find a compelling reason not to. restarted. Performance is back to normal now.

I have no idea how to feel about this situation, cause I'd really like to know what happened.

 

[Ericloewe]

Well, autotune is generally not useful, but that does make for an interesting data point...

 

[tvsjr]

Sounds like you should start by listing all the tunables that autotune set. Perhaps one will jump out as making sense.

 

[Skuloth]