SOLVED High services memory and low wired memory causing sawtooth transfer speeds

[Lascar12F]

Hi,

Just like in the title, and the accompanying graphs, shorty after boot, the wired memory goes way down, the services go way up to almost all of my memory and stay there.

If I transfer a large file immediately after boot, before the wired memory goes down, I get very consistent, steady, normal network usage in windows task manager and very acceptable transfer speeds. But after the wired memory goes down and the services reach 20 gigs, I get very strange saw-tooth networking use, and slower transfer speeds.
(see both windows task manager screenshots)

TrueNas is the most up to date version (12.0-U2.1) and it's running inside a proxmox hypervisor. There are no VMs or plugins inside truenas itself. The drives are on a LSI HBA passedthrough'ed to the TrueNas VM, and so is the Chelsio 10gig network card, no virtIO anywhere except boot drive.

One thing I can think of is that the RAM isn't acutally ECC like trunas claims, it's regular DDR4, but I assume that's something to do with proxmox, and I don't see how that would cause weird utilization.

I tried disabling services one by one to see if I got some ram back but no. I only have 3 services running anyway, SMART, SMB and SSH

 

[Lascar12F]

Ok, I'll leave this up if anyone has a similar problem, but deactivating " ballooning device " in the VM hardware RAM setting for the TrueNas VM inside Proxmox seems to have fixed this issue.

It looks like Proxmox was trying to get some ram back and TrueNas really doesn't like that.

 

[jgreco]

Oh, yeah, no, you never want a hypervisor to try to steal RAM from ZFS. VMware doesn't have this problem, because you have to reserve the guest's memory when doing PCI passthru, which happens to be exactly the right thing for ZFS.

 

[5hiftyy]

Ok, I'll leave this up if anyone has a similar problem, but deactivating " ballooning device " in the VM hardware RAM setting for the TrueNas VM inside Proxmox seems to have fixed this issue.

It looks like Proxmox was trying to get some ram back and TrueNas really doesn't like that.

I was having this same issue and have spent weeks trying to find the cause on my own, and you just solved it for me. Thanks!!

 

[TrumanHW]

I think I've almost always had FreeNAS + TrueNAS do this cyclical style transfer-speeds increasing then, slow-down.
Increasing then, slow-down.
Increasing then, slow-down.
So I'll def. be watching this thread ... I thought that was just normal (as I'd also seen it on other transfers).

My assumption was some contingent processing of the data before it was released to be written...?

 

[jgreco]

Increasing then, slow-down.
Increasing then, slow-down.
So I'll def. be watching this thread ... I thought that was just normal (as I'd also seen it on other transfers).

This can happen especially when you have a lot of memory and a relatively slow pool. ZFS ambitiously ingests data and then has to throttle back to reflect actual pool performance. The new write throttle helps with, but doesn't entirely fix, this. If you are expecting "flat line" transfer speeds out of ZFS, such as you might get from sequential writes to a conventional HDD, you'll be disappointed.

You can certainly (and easily) aggravate the situation with things like memory ballooning, which should never be used on a ZFS system.