Truenas randomly shutting down

[PlasmPlayer]

I am running TrueNAS-12.0-U7
I have a z97 MSI gaming 7 with a 4790k, 16gb ram, and 2 HBA cards connecting up to 32 2.5 inch hard drives.
After installing truenas core and setting up my raid z2's and plex it runs great and there is no smart errors. Randomly either during the day or night. the system will shut down and when i turn it on it will give me a notification that it shut down unexpectedly. I read that it may keep the logs of whats happening before it shuts down on the built in storage, I am a noob at truenas and was hoping someone can help figure out whats causing this issue.

 

[Redcoat]

Welcome to the forums!

Unfortumately, gaming boards are often not suited for use with TrueNAS.

Killer NIC may be a problem (FreeBSD drivers?).

What are your HBA's? Are they LSI-based?

Check /data/crash directory for information, as well as the content of the older /var/log/messages files (messages.1.bz2, etc).

 

[Samuel Tai]

z97 MSI gaming 7 with a 4790k, 16gb ram, and 2 HBA cards connecting up to 32 2.5 inch hard drives.

How large is your power supply? I suspect you may be experiencing sags on one or more rails given the number of drives and HBAs you have connected.

 

[PlasmPlayer]

This board is only temporary until i can get a better board and CPU to use.
Still debating o what i am wanting to run. I have access to a Xeon silver 4208 CPU, but may look into an embeded CPU for lower power consumption, if I can get multiple x16 lanes for my cards.
There is a 1x slot available i can squeeze in an intel nic to see if its a NIC problem.

I am running 2 LSI SAS 9300-16I cards

How to i check for the /data/crash directory? do i need to create an smb share for the root of the pool?

I am running a Seasonic Flagship PRIME 600 Titanium Fanless PSU. the hard drives are 2.5 inch thin laptop drives and i currently have 29 hooked up with 2 500gb ssd for cache and a boot m.2 ssd. They will all eventually be replaced with SSDs over time.

 

[PlasmPlayer]

If it is definitely a PSU underpowered, i have 4 days to return it and get one with more power

 

[Samuel Tai]

How are you powering all your drives? Via a backplane, or using splitter cables?

 

[Samuel Tai]

How to i check for the /data/crash directory? do i need to create an smb share for the root of the pool?

No, you would need to navigate to the directory from the CLI. You can either use the web shell, or SSH to your server via PuTTY. Then cd /data/crash and look for core files with ls.

Other directories to look at are /var/log/, which contains several logs you can scroll through with more:

• more /var/log/console.log
• more /var/log/messages

 

[PlasmPlayer]

I am using 2 icy dock MB516SP-B 16 drive bay. They use 4 sata power cables each and i have mini sas cables connecting them to the LSI controllers.

 

[Samuel Tai]

I suspect your power supply is maxing out. Your storage system alone is drawing at least 300 W. Each HBA uses 28 W steady-state. Each of your 32x 2.5" drives draws up to 5 W. With power inefficiencies, that's almost half of your power supply. Your motherboard, RAM, PCI-E cards, GPU, and CPU will eat up the rest.

 

[PlasmPlayer]

then i may need to look at upgrading my PSU
Attached is the putty output for the commands. The system does not instantly cut off power from what i saw flash by the screen. It looks like it does go through a shutdown process. I looked at the alert again and it says it had an unscheduled system reboot. I did update truenas to the newest version today.

 

[Samuel Tai]

Sorry, there was no attachment.

 

[PlasmPlayer]

Sorry. let me try again.

Wish i had a kill a watt meter to look at the power draw from the wall

 

[PlasmPlayer]

now i think its attached. last time it did not have the zipper icon on the file

 

[Samuel Tai]

Thanks for the PuTTY output. There aren't any core files in /data/crash, and the console log and the messages file don't show any crashes either. The only thing I can see that might be an issue was a CPU firmware update during the update, but that succeeded. This still smells like a power supply draw issue to me.

 

[Redcoat]

To follow Samuel's train of thought, Is there a way to reduce the load on the PSU for a trial? For instance, do you have multiple pools? Could you export one or more pools and remove their drives, for instance?
If possible, disable all main board devices not in use (eg: sound cards, etc.).

 

[PlasmPlayer]

I just have 1 pool in the truenas
I do have 5 drives that are not currently in a pool, I've been waiting on to get more drives to expand the current pool.
Is there a function in truenas to shut it down if the PSU is overdrawn? It does not suddenly turn off like like it would normally would from too much power drawn. I will try to get a screen capture next time it does it. I had a camera pointed at it last night and its hard to read what it says, but i will upload the video

 

[PlasmPlayer]

Truenas random shutdown

Be sure to give a thumbs up or downsubscribe if you like my videos and want to keep up to datecomment on what you thinkSubscribe: http://www.youtube.com/subs...

www.youtube.com

 

[NugentS]

@PlasmPlayer asmPlayer You mention 2.5" drives. Are these SSD or HDD? Almost all 2.5" HDD seem to be SMR (with some exceptions)

Not that SMR would cause crashes I think. Lock ups maybe

 

[NugentS]

@PlasmPlayer - you seem to be having a significant number of timeouts talking to your disks

 

[PlasmPlayer]

they are a mix of Seagate thin HDD and HGST thin HDD