TrueNAS Core 12.0-U7 - Crashing when ZFS cache is full while copying files

[MSP1978]

Hello all

i have an issue with my TrueNAS core vm running as a VM on my proxmox server.
The VM has 16GB of RAM and in idle directly after booting the system there is the following RAM:

• Free: 13,5GB
• ZFS Cache: 0,2GB
• Services: 2,3GB

I just setup the system and created the RAIDZ2 datastore out of my 4x4HDDs inside the TrueNAS Core system.

I used the shell with putty to copy over the data from my external backup hdd (NTFS) to the new datastore. But for 3 times, the system / Vm just stopped to work and went offline on proxmox.
As far as i could rebuild, the ZFS cache run full during copying, and when the cache was full, the VM stopped without any report.

I used the following method to copy my date, maybe this could be an issue / reason for the crash...

Code:

kldload fuse
mkdir /mnt/extdrive
ntfs-3g -o ro /dev/da5p2 /mnt/extdrive 'da5p2 is my external USB backup drive'

cp -a /mnt/extdrive/data /mnt/DatastoreZ2/data

Everytime i start this process, the VM keeps stopping...
Maybe anyone has an idea what could cause this crash or maybe has an idea how to prevent the vm from crashing...

With best regards,
Michael

 

[Samuel Tai]

How are you passing through the drives to TrueNAS? As virtual drives or via a PCI-passthrough of a physical HBA to which the drives are connected?

 

[jgreco]

Make sure that you are using PCIe passthru for the drives you're passing to the FreeNAS VM.

Please note that failure to do so can cause all sorts of problems. Also be sure to reserve and lock the memory for the VM. Do not let it swap or be subject to any ballooning mechanism.

Please consider upgrading to the latest version of ESXi 6.7 or 7.0. Proxmox is a newcomer to the virtualization game, and PCIe passthru support is immature. Most of the time, ESXi on recent hardware can virtualize a FreeNAS workload without incident. Proxmox is not recommended, although some people have gotten it to work.

 

[jgreco]

Also see

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[MSP1978]

Hello

wow, thats a fast response, thank you!

I checked your virtualize FreeNAS post some time before i decided to go with proxmox as host system.
As this is only my private datastore i have not really high requirements with the stability, as i'm doing several backup (USB HDD, Cloud backup) for my important files (Documents and Pictures).All other files are not so important.
Also its only me using this fileserver, so there is no high load on system and hardware at all.
The whole thing is a replacement of my HP MIcroserver Gen8 and my backup QNAP NAS... Just for the background of what i'm doing here

At the moment i'm checking the ZFS tuning guide (https://wiki.freebsd.org/ZFSTuningGuide) and this forum, i guess i can / have to go through this limitations to the memory to prevent the system from crashing.

With best regards,
Michael

 

[MSP1978]

Hmm, i did not found the edit option for my previous post to add some more informations, so i have to add them here.

After i changed some settings to the ARC system

Code:

vfs.zfs.arc.max: 8563959808
vfs.zfs.arc.min: 2147483648

i tryed another cp -a job.
But the VM keeps crashing when the memory runs full.
Now the crash situationis as followed: (Taken from the TrueNAS dashboard)

• Free: 2,6GiB
• ZFS Cache: 5,6GiB
• Services: 7,8GiB

So now i'm up to the Services which is growing from 2,4GiB when a reboot was done.
I'm only using SMB, SSH and S.M.A.R.T. services enabled on the system.

Thats really strange for me.

With best regards,
Michael