Very slow disk I/O

[JimKusz]

Hi:

I'm running a FreeNAS system based on SuperMicro 12-bay (2U) server. I've installed the IT-mode HBA and have 12 SATA drives in two vdevs of raidz2. I use it only for an NFS server for my oVirt VM server cluster. It was working well. However, I've noticed poor performance for a bit now, and I've been unable to isolate it completely. I think it happened at or since I upgraded to 11.2-RELEASE from 11.1.

I'm not sure where to go to track this down. My VM performance is nearly useless now. For example, opening Quickbooks on a VM running on the server now takes over 7 minutes for it to just open (used to be under 2). I have a Zabbix server running on it, and it can take over 5 minutes to load some web pages that are doing a number of DB queries. Most things just run SLOW.

When I did a DD test on the freeNAS box, I found a peak performance of 40MB/s. I'm not sure this is the problem, but I would have expected better performance than that, I thought.

How do I go about finding and fixing the underlying cause of the performance issues?

Thanks!
--Jim

 

[dlavigne]

Were you able to determine the bottleneck? If not, what are the full system specs? Which update version of 11.2? What's the full output of ifconfig (within code tags)?

 

[JimKusz]

Hi:
Thank you for the reply. I haven not been able to make any headway, other than convince myself its not likely something like fragmentation in the filesystem. I'm still at a loss of even pinning down the bad performance reliably. However, I am having increasing problems with my VMs, and have started to move critical systems off my VM stack (which the whole purpose of this freenas box is storage for VMs).

Here's my ifconfig. I've never used tags in a forum before, so hopefully this works right:

Code:

igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:98:49:66
    hwaddr 00:25:90:98:49:66
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 00:25:90:98:49:67
    hwaddr 00:25:90:98:49:67
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 90:e2:ba:88:a0:d4
    hwaddr 90:e2:ba:88:a0:d4
    inet 192.168.8.19 netmask 0xffffff00 broadcast 192.168.8.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
    status: active
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 90:e2:ba:88:a0:d5
    hwaddr 90:e2:ba:88:a0:d5
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:40:a9:49:fe:00
    nd6 options=1<PERFORMNUD>
    groups: bridge 
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 2000

Here's another datapoint/info: I just went to the NAS's website. It took approx. 1.5 minutes to load (I can load remote NASes over SSH-SOCKS proxy forwarding in about 5 seconds).

Running top while this was going on showed a system load of 0.12, and a 98-99% idle CPU. When I did get the GUI up, it reported a fairly stady 1.5Mbps in and 1.0Mbps out on its 10Gbps ethernet interface. VM performance is always poor, but sometimes it gets worse (my Zabbix VM is now starting to report false failures that appear to be caused by storage-related delays and performance problems)

The system is running FreeNAS-11.2-Release-U1 currently. It has:
Intel(R) Xeon(R) CPU E5640 @ 2.67GHz (16 cores)
32 GiB RAM
12 Disk pool, two RAIDZ2's with 6 disks each, each disk is a 2TB WD Red. Its booted from two SSDs plugged into the mobo SATA ports, 75GB each.
Controller: Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
The base system is a SuperMicro 12-bay storage server, supermicro mobo (don't know which model), based on the ICH10 chipset it appears. Here's lspci:

Code:

00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.0 ISA bridge: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.0 ISA bridge: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
06:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

I know there's an update up for the OS now...I'll attempt to apply that (I have to shut down my VM cluster to do that, which means most of my infrastructure, so I have to wait until late).

Oh, my zfs volume has NOT been updated to the latest ZFS features. I researched that, and it sounded like it wouldn't make any improvements, and just in case I might need to downgrade, I figured it best to leave it alone for now).

--Jim

 

[JimKusz]

Oh, one other note: I do have a jail created, but not currently running. It could be that the creation of the jail (installation of plugin) happened about the same time as the performance hit, in case that is related. I know on another FreeNAS system I was running, having jails caused a lot of issues with networking. I don't know if just having it installed could cause these issues; I'm not sure how to thoroughly uninstall (return to the same state as before I installed the plugin, removing all config changes made)

 

[lilarcor]

I was using esxi and use NFS over freenas as the vm storage, the performance was poor. So I gave it up. I read some KBs , you need some ssd as write cache to accelerate it. Not user if yours is same as what I experienced. Just FYI.

 

[Arubial1229]

Mirrors provide much better performance for VMs

 

[JimKusz]

For me, what seems so strange is that it used to work reasonably well, and has slowed down so much. It seems it tops out at 40MB/s, and I can do better than that with a single SATA disk in a desktop....

 

[JimKusz]

Ultimately, I did add an SSD for the write cache, and it did help performance a lot. First SSD has tripped SMART wear thresholds...Time to replace it I suppose...

 

[SweetAndLow]

Ultimately, I did add an SSD for the write cache, and it did help performance a lot. First SSD has tripped SMART wear thresholds...Time to replace it I suppose...

If you're not using a power loss protected ssd then it's pointless and you can just turn off sync writes to make it simpler.

You should also run a dd test read and write plus an iperf test. These are the two first thing to do when performance is a problem.