TrueNas Restarts automatically/randomly.

[cbraafhart]

Hi forums

It has been a while since i have posted something within the forums, due to always working without any problems.
But recently i noticed that after upgrading to Truenas i started to get random restarts.

First i was using USB sticks too boot my Custom Build Nas but now i finally decided to use an SATA SSD for my boot media.
My Machine is as followed:
MOBO: Gigabyte GA-F2A88XM-D3HP
RAM: 32GB of DDR3
CPU: AMD Athlon X4 860K
HDD: 7x 4TB IronWolf
SSD: 1x Kingston 256GB

This build has had no problems in the past at all, i recently change my 550W PSU to a 650W PSU.(old age) after updating to TrueNas my problems started.
i saw the forums has had similar problems, is this a software/OS issue or could this be hardware issues?
Some posted that they changed their whole machine(nearly) with the same thing happening, some disabled SMART(as a recommendation) but without any success.
I hope this is just a software issue and will be fixed within the next update, new parts with the setup i have are very hard to find and when using old tech and replacing it with new ones will become very costly.

Any tips, advice, help would be appreciated or should i just wait?

 

[diedrichg]

Is your RAM on the QVL for the motherboard? Knowing the processor model and RAM model number are critical keys here. My money is it's RAM incompatibility or the timings are set incorrectly. CPU voltage could also be set improperly. If you haven't done it yet, I'd set the motherboard back to defaults and then check the RAM voltage and timings with your model number specs. I'd also make sure that the motherboard isn't automatically overvolting or undervolting the CPU.

 

[sretalla]

AMD(can't remember the model right now)

This may have some bearing on it.

Have you turned off the C-states and cool'n'quiet in the BIOS?

 

[cbraafhart]

Is your RAM on the QVL for the motherboard? Knowing the processor model and RAM model number are critical keys here. My money is it's RAM incompatibility or the timings are set incorrectly. CPU voltage could also be set improperly. If you haven't done it yet, I'd set the motherboard back to defaults and then check the RAM voltage and timings with your model number specs. I'd also make sure that the motherboard isn't automatically overvolting or undervolting the CPU.

I am currently checking my RAM modules 1 by 1 since yesterday. this machine is not turned on on a daily basis but has been working for the last 3 years without any problems, so i am currently trying to find the Root of my problems with all the help i can get from the forums.

My processors is the following: Athlon X4 860K

i think none of the memory modules i have bought are on the QVL of the manufacture and seeing these are hard to come buy nowadays it will be hard to find memories on the QVL list.

Yesterday i did restored bios to default values but it kept doing the same.
Memory timing, memory voltage, cpu voltage and multiplier are all set on auto.

 

[cbraafhart]

This may have some bearing on it.

Have you turned off the C-states and cool'n'quiet in the BIOS?

I will check this later on the day and i will post my findings if this has granted me any success yes or no. my mother board was equipped with 4x8GB of memory but i am currently swapping them out 1 by 1 and using 1 at the same time if any changes or effects could be removed or mitigated by using correct memory modules.

 

[cbraafhart]

My next step that i will be testing is to install windows on a hard drive and see if it does the same after a few minutes/hours then i can rule out that its an hardware issue and not a software issue.

 

[cbraafhart]

Dear Forums,

Turns out that my CPU was in the state of dying. i swapped the current one with an model i previously had before upgrading to the one i had and now i am not haunted by random restarts anymore. Or after testing it for nearly 2 hours it did not restart on its own again.

But i am going to leave my NAS turned on for a week and check if the problem is solved.

 

[ThreeDee]

For your setup .. I think you are better off with the lower TDP 4 core CPU

 

[cbraafhart]

It has been 2 days of rigorous troubleshooting and as my final attempt. I am currently testing my rig by connecting my hard drives one by one to check if they are causing the problem. Or its due to a powersupply issue.

I have memtested my 32GB for approximately 16 hours without any errors. I did 4x4 tests per memory module per slot for 20 minutes each and after that i just left all 4 each in every slot and tested it for 16 hours. No restarts no reboots, mind you with none of my hard drives connected

I have swapped processors and maintained a powered up system without any restarts for 1 day. Reinstalled Truenas and attached my hard drives and after importing my zfs pool it crashed after less then 5 minutes.

So last effort is to check and see after connecting my hard drives one by one which one is the culprit or what seems to be the problem.

If all else fail i will just rebuild the pc with new parts using a HBA recommended by a techie that created the econas 2020 for less then 500$ with some minor upgrades of myself.

 

[cbraafhart]

Okay, i have tested all the hard drives by letting the be powered on for 30 minutes each, i thought one was faulty so i tried booting wihout it(6 / 7 drives) the machine stayed powered on without any reboots for 30 minutes. last i connected the hard drive i guess was faulty but it was not. every test i ran a short SMART test and resulted without any errors.

Okay the big moment came, Import my ZFS pool. this immediately caused my system to reboot and kept rebooting ever since. When i reinstalled the OS with a fresh re-install the machine worked fine without any type of errors and without any reboots for an hour long. as soon as i import my zfs pool, reboots starts.

So my zfs keeps crashing my system and i am getting nervous that i am going to loose my data regarding this. Any suggestions how i could recover or fix my zfs pool?.

 

[cbraafhart]

See attachments for information.
Yesterday i have followed someone result that posted to try to go through the bootloader to fix/recover the data:
https://www.truenas.com/community/threads/zfs-has-failed-you.11951/ (followed this post without any success) machine keeps booting after first attempting to copy files.

 

[ThreeDee]

maybe onboard SATA controller is causing the issues.. check physically inspect your motherboard for blown or bulging capacitors.

Maybe look into an HBA card of some kind .. one in my sig run about $25'ish on ebay and $7 for break out cable x 2

 

[Russell Haley]

Perhaps your pool has become corrupted? Do either of these links help?

Checking ZFS File System Integrity - Oracle Solaris ZFS Administration Guide

This book is intended for anyone responsible for setting up and administering Oracle ZFS file systems. Topics are described for both SPARC and x86 based systems, where appropriate.

docs.oracle.com

Repairing Damaged Data - Oracle Solaris ZFS Administration Guide

This book is intended for anyone responsible for setting up and administering Oracle ZFS file systems. Topics are described for both SPARC and x86 based systems, where appropriate.

docs.oracle.com

 

[Russell Haley]

You could also try to open a damaged pool as readonly?

Code:

zpool import -o readonly=on tpool

From: https://docs.oracle.com/cd/E23823_01/html/819-5461/gbbwl.html#scrolltoc

 

[cbraafhart]

Dear Forum,

I have not given up still and still trying to repair and restore my data to its original form. or at least a few or most important data from it.
I have and currently am following the procedures to repair using the Repairing ZFS Storage Pool-Wide Damage, because i don't think i have problems with a specific file or folder.

Funny thing is i did not import the zpool yet but by checking the zpool status with the specific name it says its a healthy pool.
I am currently scrubbing the pool to check data consistency, last month when importing and using it as an imported disk(even in read only) i could not scrub the pool.

So i still have hope and i still am trying.
Thanks for the feedback and help. I am currently in the process of buy a whole new PC and using a add in card to control my hard drive.
Mind you this PC can be old( bought all the parts back in 2018/2019 but i barely use it as a day to day system. i only power this baby on when i need it to store my data and after that it turns OFF!!. Behind surge protectors and smart surge protectors with circuit breakers(yes very paranoid) to protect it from over current/voltage spikes when theirs a power outage(happens very frequently in my region) .

I will let this run the whole night long and with HIGH hopes it does not crash.
I ran this on Truenas 12.01 U1 now running and trying again with Truenas 12.01 U4