Memory Leak

[Mac Thompson]

Hi

We are getting consistent memory leaks when writing large amounts of data regardless of the sharing protocol we use (ie same on iSCSI, SMB and NFS). We have a SMB share on 3 x RAIDZ vdevs. Its running at about 40TB used on a 150TB zvol. Dedupe and LZ4 enabled. When the writing is large (ie 1Gbps sustained). RAM is suddenly given up as per the graph below. When that happens the UI becomes unusable and the shares drop

.


and the network traffic dies:

Any ideas? Auto tune isnt set. Have tried it but it makes no difference.

FreeNAS-9.10.1-U2
HP DL360p Gen 9 Chelsio 10Gbp NIC, 2 x Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 786GB RAM,
MDS600 SAS disk chasis via a HP H240 HBA with 3 x RAIDZ vdevs consisting of 7 x 8TB 7200rpm SAS drives.
2.4TB stripped L2ARC Intel 3500 NVME SSD
400GB mirrored ZIL Intel 3500 NVME SSD.

 

[SweetAndLow]

That is a normal memory usage graph. The strange part is the sharp free of memory towards the end. What happens if you disable dedup and run the same test. Your system is larger than anything people run here so there might be done other interesting factors.

Sent from my Nexus 5X using Tapatalk

 

[Stux]

That is a normal memory usage graph. The strange part is the sharp free of memory towards the end. What happens if you disable dedup and run the same test. Your system is larger than anything people run here so there might be done other interesting factors.

Sent from my Nexus 5X using Tapatalk

This is the signature behavior of FreeBSDs UMA/VM/ARC bug

https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-3#post-315065

 

[Mac Thompson]

thanks Stux and SweetandLow. Interestingly Iget the same behaviour on other test systems in a virtualised environment and disabling dedupe reduces the severity of the issue, but we will still get it. Im relucantant though to disable dedupe at this stage. We operate two production systems running VMs (the system above is for backup) and they have Build FreeNAS-9.10.1. They dont have dedeupe and dont have a problem. The have mirrored vdevs which is the only notable difference.
On the backup system mentioned above we have multiple replications to it and two types of compression running as well as dedupe. Would vfs.zfs.zio.use_uma = 0 help? What is the effect? Would disabling L2ARC also have an effect?

 

[Stux]

Would vfs.zfs.zio.use_uma = 0 help? What is the effect? Would disabling L2ARC also have an effect?

UWA is an optimization which makes multi-threaded memory allocations more efficient. In theory.

Thus there would perhaps be a performance impact.

Try it and see.

 

[Mac Thompson]

UWA is an optimization which makes multi-threaded memory allocations more efficient. In theory.

Thus there would perhaps be a performance impact.

Try it and see.

looks like there is patch for FreeBSD via a new tuneable
https://forums.freebsd.org/threads/41880/page-3