Performance oddity, explorer lockups.

[diskdiddler]

Hi,

Tried accessing an SMB share on mirrord SSD pool and found windows explorer was doing the whole circle spin business for a solid 30 seconds before accessing it. It remained this way a while.

Same time my extended file copy through explorer to the regular disk array (6x8TB raid z2) had dropped to 300kb/s


This is an atom c3758 system with 8 cores, each core about 20% faster than my old 2 core AMD N54L. I have a Melannox connectx2 card in there and a 10Gb TP-Link 28 port switch.

I would describe my reaction to this ad not happy, very not happy. Having researched my hardware and building the machine slowly carefully and thoroughly before trying to use it.


So, the machine responded to ping at 1ms consistently. Iperf (a few weeks ago) maxed out at over 850Mbit.
I sshd in and copied 3.5GB from my disk array to my SSD array. That was over 250MB/s.

So, what are we looking at here? Bad drivers for the Melannox? Overheating network card? Switch too hot? SMB daemon broken?
Processor issues?

Any ideas? I really had hoped to simply find buying the premium atom setup would result in none of these issues at all.
I've also found that importing exporting jails on an SSD array with this CPU isn't much faster than my old system at all.

Is there an internal disk benchmark?

 

[diskdiddler]

Oh while I'm at it, my SSH connections are disconnecting when left idle. ? I've never seen this before. Is this a new setting?

My old NAS doesn't do this, but perhaps it's an old setting and I have something in my config which disables these timeouts?

 

[diskdiddler]

Well I (kind of) have part of an answer.

If i run iperf server on my desktop and connect to it, from my FreeNAS box as client, I get consistent, 9.5+Gbit/s. (Ran 15 times over)

Code:

[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  11.5 GBytes  9.89 Gbits/sec
------------------------------------------------------------
Client connecting to 192.168.0.126, TCP port 5001
TCP window size: 34.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.7 port 42356 connected with 192.168.0.126 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  11.2 GBytes  9.56 Gbits/sec
------------------------------------------------------------
Client connecting to 192.168.0.126, TCP port 5001
TCP window size: 34.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.7 port 39666 connected with 192.168.0.126 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  11.5 GBytes  9.87 Gbits/sec
------------------------------------------------------------
Client connecting to 192.168.0.126, TCP port 5001
TCP window size: 34.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.7 port 37995 connected with 192.168.0.126 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.1 sec  11.6 GBytes  9.89 Gbits/sec
------------------------------------------------------------
Client connecting to 192.168.0.126, TCP port 5001
TCP window size: 34.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.7 port 36607 connected with 192.168.0.126 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  11.6 GBytes  9.89 Gbits/sec
------------------------------------------------------------
Client connecting to 192.168.0.126, TCP port 5001
TCP window size: 34.9 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.7 port 18930 connected with 192.168.0.126 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0-10.0 sec  11.5 GBytes  9.87 Gbits/sec
root@freenasnew[~]#

However, if I run iperf server, from the FreeNAS machine and have my PC connect to it as a client, the speeds vary wildly.

Code:

[  5]  0.0-39.7 sec  3.93 GBytes   851 Mbits/sec
[  7]  0.0-46.8 sec   286 MBytes  51.3 Mbits/sec
[  4] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 64835
[  6]  0.0-50.5 sec  1012 MBytes   168 Mbits/sec
[  5] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 64848
[  4]  0.0-66.5 sec  3.63 GBytes   469 Mbits/sec
[  6] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 64864
[  5]  0.0-72.8 sec  3.21 GBytes   379 Mbits/sec
[  4] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 64994
[  5] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65110
[  6]  0.0-98.1 sec  1.13 GBytes  98.7 Mbits/sec
[  4]  0.0-108.7 sec   801 MBytes  61.8 Mbits/sec
[  7] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65233
[  5]  0.0-115.1 sec  5.70 GBytes   425 Mbits/sec
[  4] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65246
[  7]  0.0-121.4 sec   834 MBytes  57.6 Mbits/sec
[  5] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65265
[  4]  0.0-141.9 sec  4.14 GBytes   250 Mbits/sec
[  6] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65274
[  5]  0.0-148.3 sec   284 MBytes  16.1 Mbits/sec
[  6]  0.0-157.2 sec  1.06 GBytes  58.0 Mbits/sec
[SUM]  0.0-157.2 sec  28.3 GBytes  1.55 Gbits/sec
[  4] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65291
[  5] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65316
[  6] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65328
[  4]  0.0-27.4 sec  26.4 MBytes  8.07 Mbits/sec
[  5]  0.0-30.9 sec  6.47 GBytes  1.80 Gbits/sec
[  7] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65344
[  6]  0.0-37.4 sec  1.16 GBytes   266 Mbits/sec
[  4] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65351
[  5] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65367
[  7]  0.0-59.8 sec   110 MBytes  15.4 Mbits/sec
[  6] local 192.168.0.7 port 5001 connected with 192.168.0.126 port 65378

So what's the most logical conclusion here, if any?

 

[diskdiddler]

I've done further testing.

A Scrub runs at approximately 350-500MB/s - generally 450MB/s though, for 6x8TB disks, I guess this is ok? I'm not unhappy with that, but I'd have thought it would be faster.


The slowest I've seen a copy from /disk/ to /ssd/ across 17GB was 172MB/s which again, could be faster, but seems ok.


I rebooted the system and iperf is now behaving but this behaviour is not, in any way, ideal to be occurring this close to release.

Anyone seen any speed issues? Especially clicking a folder, in a share and the system timing out like the remote share is on a 16KB/s line, 6 states away?

 

[dlavigne]

Does this issue persist after updating to RELEASE?

 

[diskdiddler]

I am still investigating issues with my NAS, I'm not confident enough to say it's fixed or it's broken unfortunately.

I will say, for something with faster single core performance, 8 cores total, double the memory, I'm utterly shocked how many times it's 'gone weird' on me. My original install is smooth as silk, slow at times but when it's slow it's 'logically' slow or consistently slow.

We'll see. I'll update this thread as things go on. I suspect Rancher has a lot to do with this problem.