FreeNAS 9.3 filesystem hang

[cyberjock]

Got some more info from one of our devs...

There is a sysctl variable, vfs.zfs.free_min_time_ms, which controls how many miliseconds the system shall wait before starting a new free. Additionally, you can control how many blocks may be freed in one transaction group by changing vfs.zfs.free_max_blocks.

 

[titan_rw]

Before async_destroy I never noticed the pool 'hang' while it was reclaiming a large dataset. Of course the actual command blocks until the destroy is done but otherwise the pool was fully functional to other io requests.

This was deleting 3-4tb datasets that took 20 minutes or so to reclaim. I didn't mind the previous behavior at all. The actual 'delete this dataset now' command took as long as the operation itself did to complete. This seems totally reasonable. I'd not expect the pool to stall (for other requests) during this time though, and it never seemed to for me.

 

[peterh]

My zpool is a raidz with 2 vdevs, all consisting av SEAGATE ST3000NM0023 0004> Fixed Direct Access SCSI-6 device
( 3TB disks )
sysctl as discussed above is
vfs.zfs.free_max_blocks: 131072
vfs.zfs.free_min_time_ms: 1000

 

[cyberjock]

Before async_destroy I never noticed the pool 'hang' while it was reclaiming a large dataset. Of course the actual command blocks until the destroy is done but otherwise the pool was fully functional to other io requests.

This was deleting 3-4tb datasets that took 20 minutes or so to reclaim. I didn't mind the previous behavior at all. The actual 'delete this dataset now' command took as long as the operation itself did to complete. This seems totally reasonable. I'd not expect the pool to stall (for other requests) during this time though, and it never seemed to for me.

As long as you don't try to do anything that actually needs its own transaction, then things will be "just fine". Unfortunately, I can't tell you what "requires a new transaction" as I think that is partly dependent on the service running (samba, nfs, etc).

 

[peterh]

Hangs again!
Yesterday evening the same machine froze again.
Seems to happened during a vmware-snapshot of 60+ machines after that the fileserver stopped responding to nfs
The first operator that arrived noticed that vmware and all vmware hosts was dead, used the freenas console
interface ( which did work, but the GUI did not work, not responding ) . Rebooting however seemed to not happen,
after 5-7 minutes the operator pressen the reset for a reboot.
After reset it seemed to recover. But the vmware environment is a burdon to restart after happenings like this.

messages shows this from the evening before :
Apr 16 20:00:05 fs01 autosnap.py: [tools.autosnap:266] Snapshot of VM <pysphere.vi_virtual_machine.VIVirtualMachine object at 0x80d429a90> failed
Apr 16 20:00:06 fs01 autosnap.py: [tools.autosnap:266] Snapshot of VM <pysphere.vi_virtual_machine.VIVirtualMachine object at 0x80d78ed90> failed
Apr 16 20:00:08 fs01 autosnap.py: [tools.autosnap:266] Snapshot of VM <pysphere.vi_virtual_machine.VIVirtualMachine object at 0x80ea6f210> failed
Apr 16 20:00:10 fs01 autosnap.py: [tools.autosnap:266] Snapshot of VM <pysphere.vi_virtual_machine.VIVirtualMachine object at 0x80e14a150> failed
Apr 16 20:00:11 fs01 autosnap.py: [tools.autosnap:266] Snapshot of VM <pysphere.vi_virtual_machine.VIVirtualMachine object at 0x80e32d490> failed
( and so on, her eis > 60 hosts )

then we only have "innocent" messages like this :
Apr 17 05:17:11 fs01 nmbd[3403]: [2015/04/17 05:17:11.479559, 0] ../source3/nmbd/nmbd_namequery.c:109(query_name_response)
Apr 17 06:33:00 fs01 nmbd[3403]: query_name_response: Multiple (2) responses received for a query on subnet 131.97.50.134 for name VCN<1d>.
Apr 17 06:33:00 fs01 nmbd[3403]: This response was from IP 131.97.51.23, reporting an IP address of 131.97.51.23.

up until the reset button was pressed. The failed reboot did not result in any log entries.


It seems that FreeNAS is not up to the task of serving vmware . Or ? Is there anything else we can do or do we
need to purchase another netapp ?

 

[peterh]

We have tried to get more detailed information about the problem and how to reproduce, and we can provoke another
possibly related problem :
A running vmware can be "snapshot"-ed. In fact there is a mechanism in freenas to to this.
After a while a number of vmware snapshots has been accumulated. If one use the vsphere client tool
to remove vmware snapshots this will cause the vmware machine ( the virtualized server) to fail, just as if it's disk went bad.
(We use vmware 5.5 ) The vmware machine has to be rebooted. The number of vmware-snapshots was 14 in this case.

 

[cyberjock]

It sounds like someone has done some customizing of VMWare (not sure to what extent because there's multiple ways you can screw yourself out of being capable of doing snapshots). I know 2 companies that snapshot more than 40 VMs, and they have no problems doing it every day with a 2 week retention.

You should *not* have snapshots left over. FreeNAS should be telling VMWare to make snapshots of the VMs, then doing the ZFS snapshot, then telling the VMs to remove the snapshots. If this isn't happening, totally automatically, then someone has done some kind of customizing or otherwise created an environment that is not compatible with this process. Sorry, but I can't help you any more than that because the problem is much too complicated to try to troubleshoot further in a forum.

 

[peterh]

If you read the first post there was messages about snapshots that failed, also a problem that started with FreeNAS.
We run vmware according to the book, and the only thing we have changed is replacing the filestore ( netapp) with FreeNAS.

 

[peterh]

We now have replaced with a netapp filer. Case closed.

 

[voj]

I know it is an old thread..

But about same thing happened to my FreeNAS 9.2.1.9. 2.2 TB free space on pool, 6 TB total usable SATA HDD space, raidz2, async_destroy enabled, SSD caches.
- try to delete 1.2 TB zvol
- pool remained in use for other NFS shares
- after 20 minutes SSH stopped responding, ping stopped working
- going to box console, the box is unresponsive: cannot login, getty does not accept any key input
- reset power
- boot stops on pool import (I would guess it is trying to complete the transaction as per above messages)

So, contrary to what is written above - async_destroy may not help, perhaps due to the metadata issue.

If I do another reset, will it rollback the transaction or try again to complete the transaction?

sysctls:

vfs.zfs.free_max_blocks: 18446744073709551615
vfs.zfs.free_min_time_ms: 1000

 

[wblock]

Please upgrade to a supported version of FreeNAS. Much has changed and improved in the years since 9.2 was released.

 

[jgreco]

Please upgrade to a supported version of FreeNAS.

The problem is that those of us who use FreeNAS for datastores do not necessarily have an easy way to do that. Once deployed, and with a few hundred VM's running, either all the VM's need to be shut down, or the datastore needs to be storage vmotion'ed over to a different datastore. In the practical real world, this can be a nightmare. Storage is a 24/7 operation at many sites.

So in an attempt to at least provide a little as-helpful-as-possible-though-really-more-of-a-postmortem-analysis,

2.2TB available space on a 6TB usable pool would put you over the recommended maximum capacity for a pool being used for block storage on mirrors (~60%, but I prefer lower). RAIDZ2 acts as a badness multiplier in this case, as RAIDZ isn't good for block storage.

A 1.2TB zvol is a massive amount of data, especially if there are any snapshots or other complications, and you really need a ton of memory in order to help to offset the IOPS that are probably being soaked up here. Because it is a RAIDZ, you probably only have approx. a few hundred IOPS available for I/O that needs to involve the pool, and you need to be doing a massive number of metadata updates, this is going to take A Really Long Time.

A mirror based pool has somewhat higher read and write capabilities which is really helpful.

I don't have any really good suggestions for you. I haven't been tracking this particular problem so I don't know if this has been resolved in more recent versions.