Locked read speeds from freenas?

[Th3RadMan]

I built a new server and have almost everything perfect, except my read speeds will not go over 145MB/s. I have tried every trick in the book without any change. The write speeds are fine, in excess of 600-700MB/s on some files. But no matter the array (2 raid z2's and ssd raid 1) it will not go over 145MB/s. Any ideas why I might be capped? Everything is running over a 10gb nic.

Specs:
Freenas 11.2
2X xeon 2670
Asus Z9PA-D8
64gb ddr3 ecc ram
norco 4224 chassis
intel X550-T2 NIC
Intel RES2SV240
LSI 9240-8i (flahed to 9211-8i it mode)
6 8TB WD white hdd's
6 10TB WD white hdd's
3 Samsung ssd's
2 sandisk flash drives (boot)

 

[dlavigne]

Did you find the culprit?

 

[Th3RadMan]

Did you find the culprit?

nope, Even after adjusting tunables and parameters All I got was a peak of 154MB/s read. Enen with zfs I was expecting/hoping for at least double... And I just realized I labeled everything on the original post as raid hahaha

 

[dlavigne]

Some details might help to diagnose:
- exact build version (from System -> Information)
- output of ifconfig
- type of share

 

[Th3RadMan]

Build: FreeNAS-11.2-RC1

Shares: All SMB

ifconfig:
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:c3:20:c7
hwaddr a0:36:9f:c3:20:c7
inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether a0:36:9f:c3:20:c6
hwaddr a0:36:9f:c3:20:c6
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 54:a0:50:86:b3:6d
hwaddr 54:a0:50:86:b3:6d
inet 192.168.1.206 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 54:a0:50:86:b3:6e
hwaddr 54:a0:50:86:b3:6e
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:50:0c:79:64:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 20000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: plex
options=8<VLAN_MTU>
ether 02:ff:60:14:fa:09
hwaddr 02:cf:d0:00:07:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

 

[dlavigne]

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000

The jumbo frames on ix0 may be the issue: https://redmine.ixsystems.com/issues/26695 .

I know the issue was resolved for the em, igb, ixgbe, and ixl Intel drivers, but I don't think it has been for the ix drivers.

 

[Th3RadMan]

The jumbo frames on ix0 may be the issue: https://redmine.ixsystems.com/issues/26695 .

I know the issue was resolved for the em, igb, ixgbe, and ixl Intel drivers, but I don't think it has been for the ix drivers.

After some experimenting with jumbo frames I got a tiny bit more performance, I'm up to 160MB/s. Of course I have the odd driver out. Dumb question, is it possible to change the driver type or am I stuck with ix drivers on my nic?

 

[dlavigne]

The driver is tied to the hardware model. The linked manpages for the fixed drivers should provide lists of supported devices (if purchasing another device is an option for you).