TrueNAS-13.0-U6 random freezes

[Silvan Burch]

hello everybody

Looking for some help in trouble shooting, as I do not exactly know where to start.

The situation is as follows:
My boot drive died and I replaced it. While doing so, I saw how easily the restoring process of TrueNAS is (at that point it was TrueNAS 12.0-U8 or something like that) and I thought, I'd finally try the update to the newest version 13.0-U5 which I have since replaced with the update 13.0-U6.
Everything seemed to work pretty ok, i.e. all the VMs were running as they were supposed to, and I recreated the two jails I had in order to have them on the newest release as well.
However, lately my TrueNAS is hard freezing (i.e. cut power) from time to time, it's around every 3 or 4 days now.
So first thing I thought of was the RAM (got 32GB of ECC), hence I ran memtest. It passed the test four times without any errors.
Second thought: I had changed from an USB installation to a NVMe. My mainboard has no on-board NVMe slot and I bought a cheap PCIe 3.0 x16 adapter card for a Crucial P3 SSD. So I replaced this adapter card with one from ASUS, hoping it might help. Still no change.
I made also sure that swap isn't used - so far, the 4GB of swap allocated have not been used by TrueNAS as far as reportings show (i.e. at least in the last 5 months or so)

I basically try to figure the problem out without any idea why it happens. It seems as if my TrueNAS does not log anything before the freezes happen.

So my two questions:
1. any idea on where to find log files that actually are of help or how it is possible to create logs that show what is happening and why it freezes?
2. just any idea on what the problem could be? I read in these forums that the ethernet interface might be the problem. I got an ASRock Rack C236 WSI with 2 x RJ45 GLAN by Intel i210 and Intel i219 ... but since that was working before the upgrade to TrueNAS 13.0 I'm not sure that the problem lies there.

I appreciate any help on this topic ... I've been using FreeNAS and now TrueNAS for about 8 years and I really don't want to miss my setup ;)

Thank you all and best regards, Silä

PS: let me now for anything needed and I'll gladly add it.

 

[wolfie1_0]

I just upgraded to 13.0-u6 two days ago and noticed that my truenas VM has randomly died twice since the upgrade, so it might not just be you, or even hardware for that matter. I just switched back to booting from my previous instance on u5.3 and will see if that fixes it.

But the only thing that I changed was going to u6

 

[Silvan Burch]

hey wolfie
thanks for your answer. I do not think it is the same problem, though. first of all, it's not about the VMs. I do have two VMs running on the system, but it's not them that freeze up, it's the whole TrueNAS system.
secondly, the problem is occuring on 13.0-U5 and -U6. I just updated and wanted to check if the problem persists after the update.

 

[Silvan Burch]

short update:
I removed an external USB device that I was connecting only for power and thought it might interfere in some way. for two weeks I had no freezes and I started to believe that it might had solved the problem but nope, just happened again.

I really have no clue where to start, sadly

 

[Triox]

Hi, I have the same issue, but ONLY after it updated to TrueNAS-13.0-U6.1.
Mine got so annoying that it sometime froze anywhere from after 5min of uptime to a couple of hours. I did a clean reinstall via USB stick to my NVME drive, which helped it stay up for 1 whole day, but after that it froze again. interestingly, the shell command was still working (I have a monitor hooked up to it) so a restart command solved it, until it froze again...
I tried a clean reinstall and only importing my old pools, not loading any jails or anything, just clean install and letting it stay up. It died in 30min. WebUI not available, not found on my network, ping not working, like it just vanished completely. Again a restart helped to get it alive again. I did not have this problems with the U5 release. They problems showed themselves only after the update to U6.
I'll wait for the next update in hopes that this gets fixed, but if it'll be a long time, I may just do a clean reinstall of the U5 version, which was working stable for me.

My TrueNAS System:
Asus Prime B450M-A II
Ryzen 5 5600G
32Gb od DDR4 3200
Boot/Install drive: M.2 NVME SSD Samsung 970 EVO Plus 500 Gb (MZ-V7S500BW)
2x WD RED 6Tb
1x WD Blue 3Tb
No raid, I use each drive as it's own pool

 

[Silvan Burch]

thanks for sharing ...
in my case, it completely froze as the command shell wasn't even working (have a monitor hooked up as well ;)
as stated before: the biggest problem right know is that I don't even know where to start searching for the problem ... any ideas on your side? any idea what causes the freezes at all?

br

 

[Triox]

Hi, sadly no, since there are no obvious indications as to why or what's causing it. I'm also not an expert, have been using it only for about a month or so, so I'm still learning the ins and outs of it...

 

[Silvan Burch]

well, I'll let you know if I even find something - could be useful :)

 

[Silvan Burch]

don't wanna spoil it but still: now eight days without any problems ... still no idea why but I'll take it

 

[Triox]

hi, for me too and also no idea how or why...
Uptime: 7 days, 6:44 as of 18:19

 

[Triox]

OK I praised it too soon, it just died...
But my terminal is still functioning and it says: "The web interface could not be accessed. Please check network configuration."
Could be an network adapter fault... If I try to configure the network interface I get Failed on "Restarting network" & "Restarting routing" I have my NAS set to a fixed IP with DHCP reservation on my router I find it funny, that restaring the NAS fixes the issue...

 

[Davvo]

First thing I would do with unexpected restarts or hanging is a memory check.

 

[Triox]

I did MemTest with 8 or 10 passes (not sure how many I set up), and got 0 errors

 

[Davvo]

You can peek into the logs to see if there is any useful info.

 

[Triox]

The failover.log file is empty... is there another I can access?
I do have a ticket open about this issue, and I did send them the debug file that they requested.

 

[Davvo]

You could try dmesg.

 

[Silvan Burch]

just to clarify: two different problems probably
Triox still has access to his terminal with output
my case as described above: total freeze, logfiles (incl. dmesg) do not show any sign of why it happens

 

[Davvo]

my case as described above: total freeze, logfiles (incl. dmesg) do not show any sign of why it happens

Check ram, it that does not fix things:

1. reseat the CPU
2. replace the PSU
3. clean install on a fresh drive
4. replace the CPU
5. trash the motherbord

You can skip to point 5. if you run out of patience first. This however supposes there is a hardware issue, and excludes a software issues (like a memory leak, etc).

 

[Silvan Burch]

RAM are checked as stated

I can and will try no. 1 and no. 3 when it happens again

for point 2, 4 and 5: I really had hoped to find some information anywhere before I start to replace stuff as they might be still okay and I can't afford it

 

[Davvo]

Check any power connections and remove any splitter before replacing the PSU.