Memory problem in TrueNAS, don't know how to fix it. Help!

[Itay1778]

So, for the last few months, I have had a "memory" problem in my TrueNAS.
For years everything was stable and working great, but about a month+ ago I started having some strange problems.

1. All of a sudden I have corrupted files in the pool, it is discovered in the monthly scrub that it does. It's happened twice now (hopefully it won't happen again) once it was files I haven't touched in years so at first, I thought maybe that was somehow the problem. But then it also happened with really new files. I follow it but don't understand how it suddenly happens. The drives themselves are perfectly fine. And right now I've taken care of the corrupted files and everything currently looks normal.
- I also had errors in the boot pool but it worked out, probably there was an improper connection because I disconnected and connected, restarted the system, scrub, and everything worked out (for two months everything has been quiet in this part). I thought I'd mention that too.

2. My main problem, things just crashing for me,
This will be a bit long because I'm going to go into detail about everything so sorry in advance.

I noticed in the last few weeks that my Plex and qbit crashing (each has its own jail) at the same time. (I attach a screenshot)

Now if you look at the screenshot you will see that the error is that he has run out of swap but he has a lot of swap space, I checked and swap has never (in the last year) exceeded the 1GB out of the 2 he has, so this cannot be the problem.

The most obvious thing that is wrong is the use of Memory in the Dashboard
The "Services" takes all the RAM (1GB remains for ZFS and zero free space)
And I found what the problem is,
In the Plex jail it is connected to another file server (DSM) in NFS and the second I do umount to the NFS share in the jail the usage of the "Services" drops significantly (from 10GB to 4-3GB)
So he's making all the problems, at least that's what I think. If it's not the NFS I have no idea what's going on here. Because if I think back since I connected the NFS all my problems started...
Now it's probably some kind of NFS cache, the problem is that I haven't found a way to cancel it. From what I read the cache is off by default, so...

I also opened another post about this, but no one responded ):

NFS client in a Plex jail RAM issues

Hi everyone, I have a problem that I noticed in the last few days. For the past 3 months, I've been running an NFS connection (from another storage server) into a jail running Plex and it works great. The problem I noticed is that the RAM in TrueNAS fills up after a few hours to a level where...

www.truenas.com

Another thing that happens at the same time that the jails crash is that my Proxmox performs a backup for one of my VMs, and it seems that in the middle of an SMB transfer to TrueNAS, everything crashes, and the backup fails, but if I restart all my jails and start the backup again, everything works perfectly fine. Plus I have another backup of VMs on other days of the week and they don't cause problems... I don't know if it's related but wouldn't hurt to mention it.

In any case, after reading all this, I would appreciate your help in understanding what is going on here.
Like I said I think it's NFS cache but I have no idea how to disable it...

The command I used to mount the NFS into the jail:
iocage fstab -a Plex IPofanotherstorageserver:/volume1/linux-isos /mnt/linux-isos2 nfs rw 0 0

Specifications:
CPU: i3 540 (I hear them laughing at me, planning to upgrade when I can)
RAM: 16GB (I increased it from 8GB two months ago)
HDD: 3X WD Red 3TB
OS Drive: 120GB SSD
TrueNAS 12.0-U8.1

Waiting for your help. And thanks for your help in advance.

 

[elvisimprsntr]

1. Did you buy some el cheapo RAM? Are all the sticks from the same manufacturer, same specs? Have to tried to revert back to original RAM config?
2. Any changes in thermal cooling or ambient environment?
3. You should consider stepping up to a system with ECC RAM.
4. Are your WD Reds SMR or CMR?

 

[Itay1778]

1. Did you buy some el cheapo RAM? Are all the sticks from the same manufacturer, same specs? Have to tried to revert back to original RAM config?
2. Any changes in thermal cooling or ambient environment?
3. You should consider stepping up to a system with ECC RAM.
4. Are your WD Reds SMR or CMR?

1. No… I have installed 2X 4GB Kingstone 1333Mhz ( they are brand new, their manufacture date was a month before I bought them)
And 2X 2GB team group 1333Mhz
I bought the Kingstons from eBay and ran memtest for about 24H and they pass.
The old config was 1X 4GB and the 2 from team group.
I haven’t tryed to go back to the old RAM config.
2. Yes and no, Yes the room does get hotter but the temperature is still look good. CPU Max 67C
And HDD MAX 41C
3. I know, But before that I need a new case and then I will replace the platform for one that supports ECC.
4.CMR.

 

[elvisimprsntr]

1. No… I have installed 2X 4GB Kingstone 1333Mhz
And 2X 2GB team group 1333Mhz
I bought the Kingstons from eBay and ran memtest for about 24H and they pass.
The old config was 1X 4GB and the 2 from team group.
I haven’t tryed to go back to the old RAM config.

Used RAM of evilBay? If it were me, I would buy new sticks from the same manufacturer, same specs to avoid mixing sticks.

 

[Itay1778]

Used RAM of evilBay? If it were me, I would buy new sticks from the same manufacturer, same specs to avoid mixing sticks.

Yes, you're right, I have more of the Kingston I bought (which, as it seems, are brand new) I'll probably switch to them, just need to get them out of another system.
But I don't think this is what causes the crashes of the Plex, etc.

 

[Itay1778]

Update:
I replaced all the RAM with a non-ECC RAM that passed 4 passes in memtest86 And that seems to have fixed the swap and plex problems.
But now there is another problem - kernel panics, process - smbd crashes the system every 3 days.
I tried to update to the latest version (TrueNAS 13 U2) does not help.
Really at a loss for what to do. Help!

End of the latest msgbuf.txt file in /data/crash/textdump:

Code:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address    = 0x0
fault code        = supervisor read data, page not present
instruction pointer    = 0x20:0xffffffff82708cde
stack pointer            = 0x28:0xfffffe0119640750
frame pointer            = 0x28:0xfffffe0119640800
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process        = 35541 (smbd)
trap number        = 12
panic: page fault
cpuid = 1
time = 1664724602
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0119640510
vpanic() at vpanic+0x17f/frame 0xfffffe0119640560
panic() at panic+0x43/frame 0xfffffe01196405c0
trap_fatal() at trap_fatal+0x385/frame 0xfffffe0119640620
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0119640680
calltrap() at calltrap+0x8/frame 0xfffffe0119640680
--- trap 0xc, rip = 0xffffffff82708cde, rsp = 0xfffffe0119640750, rbp = 0xfffffe0119640800 ---
dnode_hold_impl() at dnode_hold_impl+0x31e/frame 0xfffffe0119640800
dmu_object_alloc_impl() at dmu_object_alloc_impl+0x23c/frame 0xfffffe01196408b0
dmu_object_alloc_dnsize() at dmu_object_alloc_dnsize+0x1c/frame 0xfffffe01196408e0
zfs_mknode() at zfs_mknode+0x1d3/frame 0xfffffe0119640a20
zfs_create() at zfs_create+0x389/frame 0xfffffe0119640ac0
zfs_freebsd_create() at zfs_freebsd_create+0xee/frame 0xfffffe0119640b10
VOP_CREATE_APV() at VOP_CREATE_APV+0x24/frame 0xfffffe0119640b30
uipc_bindat() at uipc_bindat+0x336/frame 0xfffffe0119640d60
sobind() at sobind+0x33/frame 0xfffffe0119640d80
kern_bindat() at kern_bindat+0xc4/frame 0xfffffe0119640dc0
sys_bind() at sys_bind+0x75/frame 0xfffffe0119640e00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe0119640f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0119640f30
--- syscall (104, FreeBSD ELF64, sys_bind), rip = 0x804bfe19a, rsp = 0x7fffffffe3d8, rbp = 0x7fffffffe490 ---
KDB: enter: panic

 

[Davvo]

TrueNAS needs at least 16GB of RAM. You are under minimum requirements, which means instability is to be expected. If you add apps and dockers on top of that...

Since your system requires DDR3, if you don't have enough RAM/density with you It might be the time to upgrade.
You can get a really cheap (50€?) Intel Pentium G4560 from eBay (from Italy too), a used supermicro board for under 200€ (eBay, from Germany) and a single stick of new 16 GB ECC DDR4 for around 90€ (Amazon) iirc?

If you have the budget, you could go with Better CPUs.

 

[Itay1778]

TrueNAS needs at least 16GB of RAM. You are under minimum requirements, which means instability is to be expected. If you add apps and dockers on top of that...

Since your system requires DDR3, if you don't have enough RAM/density with you It might be the time to upgrade.
You can get a really cheap (50€?) Intel Pentium G4560 from eBay (from Italy too), a used supermicro board for under 200€ (eBay, from Germany) and a single stick of new 16 GB ECC DDR4 for around 90€ (Amazon) iirc?

If you have the budget, you could go with Better CPUs.

No… TrueNAS minimum requirements are 8GB of RAM. And I upgrade to 16GB when I troubleshooting (and forgot to edit the original post. Sorry about that.) and if it was a lack of RAM then it was using swap but that not the case. And that not explain the kennel panic from the exact same process every time.

 

[Davvo]

That's in-name only, especially for your use case.
Anyway, I would try a fresh install on latest CORE.