Truenas freeze After 3 minutes and pool with problems

[carlozvr88]

Hello everybody, i've got a truenas that Is freezing After about 2-3 minutes from when It Is online. After 2-3 minutes, the cpu star working hard and the Nas stop replying! The administration Page remain stuck and reply no more. I watched the notifications and It talked about a disk with 8 errors.

In the Nas there are 3 disks: 1 Is for the OS, and 2 disks are in mirroring.

It's strange because now in the notifications i can't find again the messages about the 8 errors on dev ada0.

You can see all the datas of the computer (processor, RAM and truenas version), in the attached pictures that i tooked in the 2-3 minutes the Nas Is Active and working.

I've got the urgent problem to recover the data now, they are important and i have no fresh backups.

Thank you for your attention and help! (If it's possible or course).

 

[jgreco]

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your hardware.

You've basically given no one anything to work with, so the responses will tend to be random guesses rather than anything useful.

 

[carlozvr88]

Hello jgreco, thank you for your reply. Actually i can't understand how i can see the other details about the hardware. It Is a normal computer with a i3 540 processor and 4 gigabytes of RAM. Maybe i can disassemble It and see what motherboard model It has, if it can help.

 

[jgreco]

4 gigabytes of RAM

Ah, there's your problem. TrueNAS Core has a minimum memory requirement of 16GB RAM.

See https://www.truenas.com/download-truenas-core/

TrueNAS Minimum Requirements
64-Bit CPU | 16 GB of RAM Memory

 

[carlozvr88]

Are you sure this Is the problem? For 7 months truenas worked well. It's only now that It is freezing so hard. And if you leave It there for days and days it remains stuck. Usually when you have too Little RAM, the computer become more slow but without freezing. Considering that It seems that there Is a problem with One of the two hard disks, is there a method to save the datas contained in the healthy One?

 

[Whattteva]

Ah, there's your problem. TrueNAS Core has a minimum memory requirement of 16GB RAM.

See https://www.truenas.com/download-truenas-core/

TrueNAS Minimum Requirements
64-Bit CPU | 16 GB of RAM Memory

Oh I must've missed this over the years. I've always thought it was 8 GB min, and 16 GB recommended.

 

[jgreco]

Are you sure this Is the problem?

No, I'm not *sure* it is the problem, but the middleware itself has been known to consume 4-6GB of memory at times. Usually if the system goes semi-catatonic, it is a sign that something critical has run into some sort of major roadblock. Without actually seeing your system and poking around on it, it is difficult to know for certain what that is. The most reliable debugging strategy here on the forums has been to look for things that are out of wack, and then fix them, until the system works as expected.

Usually when you have too Little RAM, the computer become more slow but without freezing.

That's highly dependent on what is actually happening. Userland processes may start getting swapped out, which could result in the "more slow" you're expecting, but kernel resource starvation can lead to all sorts of weird problems, including the NAS going catatonic (pingable but otherwise frozen) or completely frozen.

Considering that It seems that there Is a problem with One of the two hard disks, is there a method to save the datas contained in the healthy One?

Attaching the disks to a different system, you should be able to import the pool (do NOT try to create a new pool!!!) and your data should still be fine. You have a mirror vdev setup and it is designed for this exact circumstance.

Oh I must've missed this over the years. I've always thought it was 8 GB min, and 16 GB recommended.

You're correct. It was probably about a decade ago now that I raised the documented minimum memory requirement to 8GB. At that time, dipping below 8GB was hazardous and sometimes resulted in damaged pools, etc. It isn't entirely clear to me why this was raised to 16GB, though it is safe to say that the middleware has gotten fatter and with RAM being fairly cheap, it's certainly a good idea to have more. I suspect that on a small system with no services, jails, VM's, etc., that one might still be able to get away with 8GB. But not 4GB.

 

[carlozvr88]

Hello jgreco, thank you a lot for all these informations! You're really kind! And Also for the Hope you gave me for the files recovering!

So What do you suggest to do in this case? Is there a guide that i can follow to attach disks to another system and to import the pool?

Have i to setup a new system with only the OS hard disk connected and then in a second Moment i Will connect both the two hard drives to import the pool? Or have i to disconnect the two mirroring hard disks and to connect them to a new computer and then setup a new truenas system from the beginning? When can i find the command to import the pool?

I'm Sorry for all these questions, i'm a computer technician but i'm a Total newbie about truenas!

I Also did a backup of the settings a lot of months ago, i don't know if they can be useful.

And anyway i can do It again, in these 2 famous minutes.

Moreover, Is there something else that i can do to identify Better the issue or to solve the issue without installing a new system? For example i can connect a monitor and a keyboard and see what happens on the monitor. Moreover in these 2 minutes that the system works, i can do everything: i can read the logs (if you Say to me how can i do It, because i don't know i never did It), i can shut down things that burn CPU and Memory, and so on.

I can Also try to add more RAM to the system! Does It cause issues?

Last question: if i Simply disconnect the broken hard drive? What Will happens? Can It solve the situation only to recover the data? I know that there are some procedures to correctly disconnect an hard disk on truenas.

Maybe i must First deactivate It from the control panel? (Another thing that i can try to do in these 2 minutes).

Thank you again for your patience.

 

[carlozvr88]

In the meantime i disassembled the computer. The motherboard Is an Asus p7h55-m/le .

I read in some posts that there are Also things in the BIOS that maybe i must to activate or deactivate?

 

[jgreco]

In the meantime i disassembled the computer. The motherboard Is an Asus p7h55-m/le .

I read in some posts that there are Also things in the BIOS that maybe i must to activate or deactivate?

Perhaps. However, since you're not using anything that resembles recommended hardware, this puts everyone at a disadvantage. What I can tell you is that, as an older 2010-era board, there are some things that might not work correctly.

Asus seems convinced that the unit will only accept 8GB of RAM, and Crucial's memory configurator agrees.

Further, as it is a desktop board, it has a Realtek ethernet chipset that is certain not to work at peak performance. I suspect you can get basic functionality out of it by putting the 8GB RAM in, but even with that, it isn't going to be awesome.

Memory RAM & SSD Upgrades | Crucial.com

Crucial Memory and SSD upgrades - 100% Compatibility Guaranteed for - FREE US Delivery.

www.crucial.com

If it were mine and here in the shop, I would try some 8GB sticks in it just to see if you could make it work on 16GB.

 

[carlozvr88]

Hello jgreco, i Will try to put more RAM in the system and let's see what happens.

What about the other questions? I have another truenas system at home with 2 500 gigabytes disks in mirror where i can put the disks with the datas. How does It work? Is there a guide that i can follow? Have i to connect the two mirroring disks of the not working system to this system? And then? How does the pool import works?

What about disconnecting One of the two mirror disks (hoping the One with the issues, but i Imagine i have to try First One then the other) from the not working asus truenas?

What Will happen if i do It?

Thanks again.

 

[ChrisRJ]

Since you are running a system that is 10+ years old that has worked well so far, chances are that some piece of hardware has finally given up. The PSU would be my prime suspect, probably together with the motherboard. A freeze is usually an indicator of a hardware issue, especially if the setup has worked so far.

For additional background information, which I think would be helpful, please check the recommended readings in my signature.

 

[Patrick M. Hausen]

Move both disks to the other (hopefully stable) system. Import the pool via UI. Simple as that.

Storage > Pools > ADD > Import an existing pool

HTH,
Patrick

 

[carlozvr88]

It's strange because now at home i had a 4 GB bank of RAM. I upgraded the RAM from 4 to 6 GB and the problem remains. Now It freeze faster because maybe the system with more RAM Is more fast (lol).

So i have to remove the two 500 GB mirror hard drives from the other system (yes it's stable) and connect the two 2 terabytes mirror hard dives and go in storage pools add import existing pool and It Will see them?

There Is another thing: the other system Is truenas 13, but it's empty so If it's a problem, i can erase It and reinstall the 12 without any problem.

What about if i physically disconnect the sata cable of the bad drive? What Will happen?

 

[carlozvr88]

So i put a new 4 GB bank of RAM to see if the problem were the RAM banks. Nothing changed. I changed ALL the (three) sata cables. Nothing changed. I've got a new PSU so i can Also try to change It, even if It's strange because i changed the PSU when i set up the system 7 months ago (so the PSU has only 7 months of Life).

The behaviour Is Always the same: i turn on the system, It loads and work for a couple of minutes, then the CPU go to the 50% and the system freeze and stop answering. It continue answer only to the ping.

The Red led of disks remains full turned on for another couple of minutes and then It turn off and never turn on again.

It really seems a problem caused by One of the hard drives (i imagine the damaged One). Can It Be possibile that it freezes the whole system?

This Is why i would like to disconnect the sata cable of the mirroring pool drived. First One and then the other One.

If you Say to me that it's not Dangerous for the datas, then i'll do It.

If it Will not work, i Will try changing the psu. Moreover i have other 3 free sata connectors on the mainboard, so i can try change the
sata connections on the mainboard of all the 3 hard disks!

If nothing work, i Will put the two disks on the other stable system.

What do you think about these tests?

Thank you again to everybody!