truenas crashing when in use

[tursiops]

Hi everyone,
I have a recurring and very problematic issue with Truenas.
As long as the server is idle it works fine, but as soon as I want to transfer data on my shares, it will crash. It can take a few minutes, or 30 minutes, or several hours but it will crash in the end. The only option would be to reset the server using the physical button.
What I've already tried:
- I've updated everything to the latest version
- i've destroyed and re-created my pools
- i've wiped and reinstalled truenas.
- i've tried using smb and nfs shares, same result
- i've tried accessing the nas via network cable or wifi and same thing
- My hardware:
- B550 Aorus pro motherboard
- 24Gb memory
- AMD Ryzen 5 3600 6-Core Processor

Pools details:
first pool with 2 drives of 8tb in mirror
second pool with 3 drives of 4tb in RAIDZ1

Accessing server with fixed address, ipv4.

3 shares on second pool
1 share on first pool

Hope this is information enough, the problem can be reproduced at will.

Thank you for your help.

 

[LarsR]

Your Mainboard has an integrated Realtec NIC and Realtek support on bsd is pretty flaky. First thing i would try is a seperate Intel NIC

 

[tursiops]

Your Mainboard has an integrated Realtec NIC and Realtek support on bsd is pretty flaky. First thing i would try is a seperate Intel NIC

What kind of card that would do 2.5Gb/s would you recommend. I move quite large files that's why i need to have 2.5Gb lan and that's why i chose that board. But it seems it was not the best choice after all...

 

[LarsR]

propably this one https://www.intel.de/content/www/de...dapters/ethernet-network-adapter-i225-t1.html

but i think intel hasn't released this one yet :/

 

[tursiops]

propably this one https://www.intel.de/content/www/de...dapters/ethernet-network-adapter-i225-t1.html

but i think intel hasn't released this one yet :/

Yes it says launched, but i cannot find on sale. Otherwise i can try with a cheap 1Gb card to test your theory. What do you suggest in that case?

 

[LarsR]

Any 1Gb with an intel chip will do the job

 

[tursiops]

Any 1Gb with an intel chip will do the job

Ok i've bought an ethernet gigabit intel card and also done the latest update, but the situation is worse now. I'm getting malloc errors, system is rebooting on its own at the slightest network access.
The strange thing is that, it first stops to ping, but if i go directly on the computer and ping google it still works.
I have tried with 2 sticks of ram instead of 4, and alternated between both but same results.

Thank you for your help.

 

[Captain404]

Maybe a RAM issue ?
What does Memtest86+ say ?

 

[tursiops]

Maybe a RAM issue ?
What does Memtest86+ say ?

Can I run memtest from the shell directly or do I need a boot cd?
Also I should be able to upload the logs to check why the computer is restarting, I'm not familiar with this version of Linux, can you help me out?
Thanks!

 

[Captain404]

You need to create a bootable usb key with this utility and with an ISO.
try this https://www.ultimatebootcd.com/ download ucbc iso and boot on legacy mode not on UEFI mode.
or a live linux distribution iso like ubuntu . How to boot on memtest86+

 

[ThreeDee]

Check for an updated BIOS for your motherboard ..
Try running just 2 sticks of matching RAM in slots A2/B2
Disable unnecessary features like sound, etc.… and Global C-states, Cool & Quit and ErP-Ready in BIOS can be problematic for some so you could try disabling them and see if that helps with stability.

 

[tursiops]

Check for an updated BIOS for your motherboard ..
Try running just 2 sticks of matching RAM in slots A2/B2
Disable unnecessary features like sound, etc.… and Global C-states, Cool & Quit and ErP-Ready in BIOS can be problematic for some so you could try disabling them and see if that helps with stability.

ok i will try that.
Here is my messages file log, tell me if there is something that jumps at you as being wrong, or tell me what log files you would need to find out what is causing the issues.
I have had another weird issue where my nvme drive would cause a interrupt error.

 

[tursiops]

Check for an updated BIOS for your motherboard ..
Try running just 2 sticks of matching RAM in slots A2/B2
Disable unnecessary features like sound, etc.… and Global C-states, Cool & Quit and ErP-Ready in BIOS can be problematic for some so you could try disabling them and see if that helps with stability.

Well disabling all unnecessary features didn't help. It worked for like 10 minutes, and then it just rebooted on its own. :(

 

[pschatz100]

Ok i've bought an ethernet gigabit intel card and also done the latest update, but the situation is worse now. I'm getting malloc errors, system is rebooting on its own at the slightest network access.
The strange thing is that, it first stops to ping, but if i go directly on the computer and ping google it still works.
I have tried with 2 sticks of ram instead of 4, and alternated between both but same results.

Thank you for your help.

Did the problem get worse after installing the new network card?

Are you confident your power supply is OK?

What is your boot device? If you are booting from a flash drive that is flakey or going bad, it can create all sorts of impossible to find errors.

When you say you disabled unnecessary features, did you disable all unnecessary features? I would make certain all power-saving features are disabled, any usb enhancements, sound, the realtek NIC, and any accelerated or optimized motherboard profiles. Literally turn off everything that is not required for the system to run. Gaming motherboards have lots of features you don't need, and they can cause headaches.

 

[tursiops]

Did the problem get worse after installing the new network card?

Are you confident your power supply is OK?

What is your boot device? If you are booting from a flash drive that is flakey or going bad, it can create all sorts of impossible to find errors.

When you say you disabled unnecessary features, did you disable all unnecessary features? I would make certain all power-saving features are disabled, any usb enhancements, sound, the realtek NIC, and any accelerated or optimized motherboard profiles. Literally turn off everything that is not required for the system to run. Gaming motherboards have lots of features you don't need, and they can cause headaches.

Well it did get a bit worse indeed, because before the computer didn't reboot on its own, it just stopped responding.
My power supply is good quality so i'm confident that it's not the problem.
I must say that i'm not a specialist of bios options, specially since as you said there are a lot of them.
I have disabled the audio and the nic also the global c-states as mentionned before.
Otherwise do you have a suggestion for a more compatible motherboard? I chose thise one because it had a 2.5nic but i can live with another motherboard with a 2.5g nic add in card.

 

[ThreeDee]

ASRock Rack > X470D4U

www.asrockrack.com

mobo I'm using right now .. it's been great

or the 10Gb variant

ASRock Rack > X470D4U2-2T

www.asrockrack.com

they are server boards with IPMI.. very handy to have. You can run your current CPU and memory ..but I'd recommend getting some ECC UDIMM's

 

[tursiops]

ASRock Rack > X470D4U

www.asrockrack.com

mobo I'm using right now .. it's been great

or the 10Gb variant

ASRock Rack > X470D4U2-2T

www.asrockrack.com

they are server boards with IPMI.. very handy to have. You can run your current CPU and memory ..but I'd recommend getting some ECC UDIMM's

ok thanks i will try that too

 

[fivearrowsnh]

I have a similar problem. My motherboard has an Intel network port according to the messages at boot up. I’ll try the BIOS setup changes. I’m going on vacation today so it’ll be a couple weeks before I can report back.

 

[tursiops]

ASRock Rack > X470D4U

www.asrockrack.com

mobo I'm using right now .. it's been great

or the 10Gb variant

ASRock Rack > X470D4U2-2T

www.asrockrack.com

they are server boards with IPMI.. very handy to have. You can run your current CPU and memory ..but I'd recommend getting some ECC UDIMM's

So i've followed your advice and bought that mobo with the 10gb nic, but when i boot i get a error 3b from what i've found online it would be a problem with memory. Can you confirm? I have some hyperx fury ddr4 8Gb module, 2 of them.

Thank you!
Rafael.

 

[ThreeDee]

are they in slots A1/B1 (the blue slots)? ..or try with just 1 stick in A1 ..if no go .. try the other stick in A1.

..also, make sure you have the memory all the way "clicked" in ..

When you get it to boot, grab the latest BIOS update and BMC update ..
..and technically you don't need it to boot as you can update through BMC features