Freenas unresponsive - where to start with diagnostics

[cwilliams40]

Hi, we've been running Freenas for a while now on the hardware I've listed below. Recently we've been having issues with it being unresponsive - shares unavailable and VM's dropping their iSCSI LUNs. Since thep problems began there have been 2 changes - increased load on the machine; upgrade from freenas 8.3.1 to 9.1.1. Most of the time we are able to SSH onto it and reboot but this isn't a given and sometimes takes 2+ hours to accept the SSH connection. Could someone point me in the right direction for trying to diagnose what is going on

Here's what we have

Freenas 9.1.1
6 x 3 TB mirror and stripe (zfs raid 10)
4 SMB shares
1 NFS share
1 x 5TB zvol presented to VMware ESX5.5 via iSCSI with multipath on 2 separate controllers
1 x LACP lagg contaning 2 nics on seperate controllers

Hardware

Supermicro H8DGi-F
AMD Opteron 6272
14gb ECC ram
6 x 3TB Seagate 7.2k sata (2 are enterprise class, the others consumer)
2 x 256gb samsung 840 pro l2arc
IBM M1015 crossflashed to LSI card in IT mode using

Thanks

 

[cyberjock]

First, you need MUCH more RAM if you plan to run VMs without them going unresponsive. I'd start with 32GB of RAM, but you might 64GB or more. So don't buy a bunch of 2GB DIMMs and expect that 32GB will solve your problems. Your l2arc should never exceed about 5 times your RAM(3-4 is better) because of the index. You've broke this cardinal rule with 2x256GB l2arc disks(even if you assume you mirrored them). In fact, its impossible for you to even be using 150GB of your l2arc because of how little RAM you have. As soon as you start wanting to run VMs on ZFS you are implying that you need to make your server more powerful than just some CIFS/NFS shares. Failure to do so results in systems that become unresponsive and ESXi begins dropping the devices(and that's exactly what you have).

You also need to downsize your l2arc to something less than 5x your RAM or increase your RAM to make the RAM to l2arc ratio less than 5. Even if you go with 5, you are putting alot of stress on your RAM. I'd shoot for a ration of about 4.

You should also check your disks to see if one of them is failing. A failing disk will cause all sorts of problems for FreeNAS.

 

[cwilliams40]

Thanks for the reply. 14gb was all the ecc ram we had lying around at the time and did think it was a little too low, it was really just a proof of concept at first that then became production and we didn't make any changes

The server in question has 16 memory slots (although only 8 work without 2nd CPU) but will order a bunch more ram. As for the l2arc I will take a look, I think I just used the only ssds we had spare at the time

Disks all seem fine, no smart issues, weekly scrub on pool finished ok today in good time

 

[cwilliams40]

.

 

[cwilliams40]

This is now working much better after upgrading to 32gb of ram and downsize l2arc to 120gb

 

[jgreco]

That's great to hear! We love to hear the success stories!

 

[cyberjock]

Yaaah! Word of the day is "rightsizing".

 

[jgreco]

Rightsizing ... 512GB of L2ARC plus 128GB system RAM --> very fun very fast VM datastore!