FreeNAS loses IP and spikes CPU

[Mark Francis]

Hello,

I'm running FreeNAS-11.0-U3 (c5dcf4416) in a Proxmox hypervisor.

Every so often (and a lot recently) the server will become unresponsive on it's web gui and shares. When I login to my hypervisor, I can see the server showing the web user interface on 0.0.0.0. A reboot will fix this for a small amount of time. It also seems to spike the cpu beyond normal (see image. All spikes are these events). I know I don't have a bad cable because the other VM's remain up and working. It's also not IP conflicts, since this VM's IP is set by DHCP.

I'm more of a linux guy, so am not really sure how to approach troubleshooting this. My first thought was filled systems partitions, which I can see a few tiny ones at 100%, but they are small so I expect this is usual?

Code:

Filesystem												  Size	Used   Avail Capacity  Mounted on
freenas-boot/ROOT/default								   7.7G	725M	7.0G	 9%	/
devfs													   1.0K	1.0K	  0B   100%	/dev
tmpfs														32M	9.4M	 23M	29%	/etc
tmpfs													   4.0M	8.0K	4.0M	 0%	/mnt
tmpfs													   3.3G	 20M	3.3G	 1%	/var
freenas-boot/grub										   7.0G	6.3M	7.0G	 0%	/boot/grub
fdescfs													 1.0K	1.0K	  0B   100%	/dev/fd
tempvol													 105G	104G	839M	99%	/mnt/<redacted>
vol1														5.2T	614G	4.6T	11%	/mnt/<redacted>
tempvol/.system											 1.3G	508M	839M	38%	/var/db/system
tempvol/.system/cores									   859M	 19M	839M	 2%	/var/db/system/cores
tempvol/.system/samba4									  840M	396K	839M	 0%	/var/db/system/samba4
tempvol/.system/syslog-d6d294a404024d4b9f621e6bf1afa53c	 844M	4.4M	839M	 1%	/var/db/system/syslog-d6d294a404024d4b9f621e6bf1afa53c
tempvol/.system/rrd-d6d294a404024d4b9f621e6bf1afa53c		854M	 15M	839M	 2%	/var/db/system/rrd-d6d294a404024d4b9f621e6bf1afa53c
tempvol/.system/configs-d6d294a404024d4b9f621e6bf1afa53c	858M	 18M	839M	 2%	/var/db/system/configs-d6d294a404024d4b9f621e6bf1afa53c

Does anyone have any ideas how I can approach troubleshooting this?

 

[DrKK]

The thing that jumps out at me is the 99% full dataset you have "tempvol". ZFS pretty much grinds to a halt if a dataset gets full like that, because it requires blue sky to do all of its transactions.

 

[Mark Francis]

I have cleared out some data from that share to remove that as a possible cause. This issue has been ongoing for months now, so I think it's unlikely to be the cause.

 

[DrKK]

I have cleared out some data from that share to remove that as a possible cause. This issue has been ongoing for months now, so I think it's unlikely to be the cause.

I agree. But I thought I'd still point it out as a remote possibility.

Once FreeNAS is running under a hypervisor (particularly one that is not ESXi), then, really, it's just too many variables for me, at least, to triage, especially since I don't virtualize anything. One thing you'll note, is that a large percentage of people having problems in the forum, *AND*, a large percentage of the forum help requests with very few replies, are all people with virtualized setups. But everything you said in your original post suggests something with the virtualization.

Maybe someone else will have more useful ideas.

 

[Mirfster]

While I don't use ProxMox, I do virtualize my FreeNAS with ESXi 6.0U3. Also, I run FreeNAS 9.10...

Anyways:

• Have you reserved dedicated memory to the FreeNAS VM (not "shared memory")?
• How does FreeNAS access the drives it uses for Storage (does it have direct access to the controllers/drives)?
• What is the output of: zpool status
• What is the output of: zpool list