Help: Odd write performance

[Asi9]

Hey guys. I'm still pretty new to freenas in the grand scheme of things and slowly learning as I go. I've built up a dedicated box and think I have most of the kinks worked out after a lot of research, headaches, and frustrations, but I cannot for the life of me figure out what exactly is going on with my write performance from my PC to the FreeNAS box.

I've done iperf tests, checked the ifconfig, run the disk tests, switched all cabling out to new CAT6 cables, etc and everything seems good to go. Read speeds are consistent and fully saturating the new i210 card I threw in today:

But the write performance I have no explanation for:

This persists whether it's a large, single file as above (26gb single file) or folders of smaller files. I can't figure out if this is a limitation of my hardware or a configuration issue or what exactly the case is, but I feel like I've hit a wall so any insight would be greatly appreciated. Thanks everybody!

FreeNAS server specs are as follows:
Mobo: Supermicro X8SI6-F
CPU: Intel Xeon X3470
RAM: 22GB Hynix DDR3 ECC
HDDS: 6x HGST Ultrastar 3TB drives in ZFS-Z2
NICs: Intel i210 PCIe card (switched from the onboard Intel NICs because I thought they were possibly the issue)

 

[kdragon75]

What is your dataset sync setting? This looks oddly close to the 5 second default TXG timeout...
Ok hold up, I don't even know what version of FreeNAS your running...

 

[Asi9]

What is your dataset sync setting? This looks oddly close to the 5 second default TXG timeout...
Ok hold up, I don't even know what version of FreeNAS your running...

Sorry about that. I should have stated I am running FreeNAS-11.1-U6.

And I have to admit I'm not sure what exactly your hypothesis is referring to, but as far as the sync setting, it is set to "standard".

 

[Asi9]

Update: I am now seeing an odd cap on server reads as well depending on the client. Could this indicate an issue with the SMB service itself?

 

[Asi9]

Update2: I tried forcing my windows SMB protocol to force a higher version to be used and still no dice. I'm starting to wonder if this is possibly a limitation of FreeNAS itself of if there's some sort of configuration issue server side. Would upgrading to 11.2 possibly solve this? I'm running out of ideas here :\

 

[kdragon75]

Have you done any benchmarking on the pool? Even a simple DD style test? We need to find out where the performance issues are coming from. Lets start with establishing the raw hardware baseline. Try making a new dataset and setting compression to off (important for this test) and running dd if=/dev/zero of=/mnt/<pool_name>/test_dataset/test.dd bs=1M count=26000 (write) & dd if=/mnt/<pool_name>/test_dataset/test.dd of=/dev/null bs=1M (read)[/cmd][/cmd]

 

[Asi9]

Have you done any benchmarking on the pool? Even a simple DD style test? We need to find out where the performance issues are coming from. Lets start with establishing the raw hardware baseline. Try making a new dataset and setting compression to off (important for this test) and running dd if=/dev/zero of=/mnt/<pool_name>/test_dataset/test.dd bs=1M count=26000 (write) & dd if=/mnt/<pool_name>/test_dataset/test.dd of=/dev/null bs=1M (read)[/cmd][/cmd]

I have run some tests in the past, and everything looked to be within acceptable ranges. But it's worth another go at it, soI did as you suggested, and here are the results:

Write:
[root@freenas ~]# dd if=/dev/zero of=/mnt/DatNAS/test/test.dd bs=1M count=26000
26000+0 records in
26000+0 records out
27262976000 bytes transferred in 89.363828 secs (305078425 bytes/sec)

Read:
[root@freenas ~]# dd if=/mnt/DatNAS/test/test.dd of=/dev/null bs=1M
26000+0 records in
26000+0 records out
27262976000 bytes transferred in 93.104760 secs (292820431 bytes/sec)

 

[kdragon75]

Yeah those number look spot on. This is good(ish) news. Now you may want to look into trying to set sync disabled on your SMB dataset to test if that makes any difference.

 

[Asi9]

Yeah those number look spot on. This is good(ish) news. Now you may want to look into trying to set sync disabled on your SMB dataset to test if that makes any difference.

I'll give that a shot, copy a large file, and report back.

 

[Asi9]

It seems setting sync to disabled made things a whole heck of a lot worse...

Read from NAS:

Speeds were very inconsistent, and would sometimes drop down to or below 1MB/s.

Write to NAS:

Think that one about speaks for itself. Windows couldn't even get an estimated time and I even got an "error attempting to copy this file" on the first try.

 

[kdragon75]

Ok something is screwy. If anything sync disabled should make it faster. Time for an iperf from FreeNAS to your machine. Have you verified that the MTU is 1500 all around?

 

[Asi9]

Ok something is screwy. If anything sync disabled should make it faster. Time for an iperf from FreeNAS to your machine. Have you verified that the MTU is 1500 all around?

Oddly...and terrifyingly, those transfer rates have now seem to have stuck even after setting it back to normal and restarting. I really hope things are not stuck this way. Anyway, I tried running an iperf test and now seeing this:

[root@freenas ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.107 port 5001 connected with 192.168.0.104 port 58992
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-12.1 sec 424 KBytes 288 Kbits/sec


[root@freenas ~]# iperf -s -w 128k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 125 KByte
------------------------------------------------------------
[ 4] local 192.168.0.107 port 5001 connected with 192.168.0.104 port 58997
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-12.1 sec 504 KBytes 342 Kbits/sec


And apologies for my ignorance, but where all would the MTU need to be set at? The only place I am personally aware of to set that at, is the options within the interface configuration.

 

[kdragon75]

And apologies for my ignorance, but where all would the MTU need to be set at? The only place I am personally aware of to set that at, is the options within the interface configuration.

That's correct. Each interface would have its own MTU. Your desktop, FreeNAS and possibly, your switch. Though the switch is less important than the endpoints (Desktop FreeNAS etc...

342 Kbits/sec

That's inline with your windows transfer rates... what does ifconfig say on FreeNAS?

 

[Asi9]

That's correct. Each interface would have its own MTU. Your desktop, FreeNAS and possibly, your switch. Though the switch is less important than the endpoints (Desktop FreeNAS etc...


That's inline with your windows transfer rates... what does ifconfig say on FreeNAS?

I have the mtu configured in FreeNAS for 1500, but am unsure where to set that for my desktops LAN. I'm assuming it's somewhere buried in the Intel NIC settings.

and for an ifconfig output:


root@freenas:~ # ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether a0:36:9f:4c:b5:cd
hwaddr a0:36:9f:4c:b5:cd
inet 192.168.0.107 netmask 0xffffff00 broadcast 192.168.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 100baseTX <full-duplex>
status: active

em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:25:90:3a:d6:a4
hwaddr 00:25:90:3a:d6:a4
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier

em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:25:90:3a:d6:a5
hwaddr 00:25:90:3a:d6:a5
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:1b:48:d2:c6:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 200000

epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:21:90:00:06:0a
hwaddr 02:21:90:00:06:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:21:90:00:07:0a
hwaddr 02:21:90:00:07:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

 

[kdragon75]

I have the mtu configured in FreeNAS for 1500, but am unsure where to set that for my desktops LAN. I'm assuming it's somewhere buried in the Intel NIC settings.

Yeah its either in the Intel utils or the properties of the network card device in device manager. but that all looks fine. You have some sort of network issue. Have you done any tuning/tweaking in FreeNAS?
I hate to say it but reboot your router, switch, and both machines.

 

[Asi9]

Yeah its either in the Intel utils or the properties of the network card device in device manager. but that all looks fine. You have some sort of network issue. Have you done any tuning/tweaking in FreeNAS?

Quickly looked it up to double check, and seeing this in the windows command line so guessing I'm good there:
C:\WINDOWS\system32>netsh int ipv4 show int

Idx Met MTU State Name
--- ---------- ---------- ------------ ---------------------------
18 25 1500 connected Ethernet
1 75 4294967295 connected Loopback Pseudo-Interface 1

And if you are referring to tunables and such, the only things I have set are a couple of tunables in the table for Transmission, and an init script to set minimum allowed C-state.

The tunables that seem to have been set by the system are as follows:
kern.ipc.maxsockbuf=5242880
net.inet.udp.recvspace=4194304

Actually, I forgot I added this myself, as otherwise my Shield TV isn't able to access FreeNAS.
sysctl freenas.services.smb.config.server_min_protocol=NT1

 

[Asi9]

I hate to say it but reboot your router, switch, and both machines.

Well, just did a hard reset on both machines and the router connecting the two. I had set the sync back to Standard, but I'm not sure what exactly I did that changed it, but it does seem to be more consistent now. Performance doesn't seem absolutely capped, but it's certainly better than it was and I can live with it.

Write:

Read:

Iperf:
[root@freenas ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.0.107 port 5001 connected with 192.168.0.104 port 50020
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.09 GBytes 936 Mbits/sec

Strangely, I have seen this level of performance and behavior before, and it seems to not last for long after it starts performing normally again. Hopefully it sticks this time. Do you have any other suggestions for things to check on or tweak? Also, thanks so much for all the help!

 

[gpsguy]

I'd save (backup) the configuration.

but I'm not sure what exactly I did that changed it

 

[kdragon75]

Can you provide your zpool list -v? Next time it gets slow, reboot one device at a time until you find the one that fixes it.

 

[Asi9]

Can you provide your zpool list -v? Next time it gets slow, reboot one device at a time until you find the one that fixes it.

That command spits out:

The forums really don't seem to like the formatting of pasting the output of that too much, so just went for an image :p

I'd save (backup) the configuration. :)

And that's a pretty good idea. I'll have to remember how to do that now haha. IIRC that's in the System/General tab.