Continuous Reboot

[Mutley]

Hi
So the story begins, my system has been running well for the last couple of years, then about a week ago i was getting continuous reboots after about 2 minutes, Unfortunately I've made one disastrous mistake in my system and I didn't back up my configuration, I'm a newbie I know and no excuses, probably distracted as it was working so well, Originally it was running on a SSD drive Freenas 11.3 but I then decided to run the boot drive on a SuperMicro sataDom 16GB after looking at recommendations, which worked perfect in my Supermicro X10SLL-F motherboard. I open up my Fractal Node 804 to start pulling components out and i notice the sata dom is loose and then finally the solder joint came apart so consequently the sata connection separated from the circuit board. So I assumed this was the fault and replaced the satadom with a WD Green SSD 120GB. I then reinstalled Trunas 12.0-U6 and have server up and running on basic settings "no configuration yet", unfortunately it is still rebooting, having removed ram and unplugging components, I believe it is something to do with my raidZ2 6 WDREDS 6TB pool, If I disconnect them, the server is stable and no rebooting, my original hardware though was connecting the drives via LSI SAS 9207-8i kit and 2 mini sas SFF-8087 forward breakout cables, again I thought i had narrowed down problem to sas card or sas cables, I temporarily changed card to older raid card and standard sata cables but problem still existed and rebooting continues after about 2 minutes. I suppose my questions are do you think I am right and it is the raidz2 pool, when the system is all plugged in, it shows no faults on the dashboard of Trunas and my raidz2 pool is fine no errors but reboots, if I disconnect pool it runs fine, so 2nd question is do I now detach a disk to see if one of the 6 disks is actually the problem, can I actually do this because it reboots after 2-3 minutes, so I'm assuming to reattach drive resilver will take a few hours to get back to normal and 2-3 minute reboots are not healthy. Another new lesson I had learnt was about WD Red drives, again I assumed they were all the same, but since my problem I have 3 WDRED 6TB EFRX Drives and 3 WDRED 6TB EFAX Drives, I don't think that's the problem as its been running fine for 7 months since I changed a drive, I have now bought 2 new WDRED Plus drives to put into pool so I can remove the EFAX Drives once its all up and running. Server is used for Emby and Nextcloud, so not overloaded but all my dvd,'s and cd's are stored on it

 

[Samuel Tai]

Your statement was very hard to follow, but it sounds like you have a hardware failure in your motherboard causing the automatic reboots.

I open up my Fractal Node 804 to start pulling components out and i notice the sata dom is loose and then finally the solder joint came apart so consequently the sata connection separated from the circuit board.

If there's loose solder, it'll result in shorts throughout the motherboard. You'll need to shut down your system, and blow it out thoroughly. Also replace your motherboard, as that's likely not salvageable. Your HBAs and drives can probably be salvaged.

 

[Mutley]

Your statement was very hard to follow, but it sounds like you have a hardware failure in your motherboard causing the automatic reboots.



If there's loose solder, it'll result in shorts throughout the motherboard. You'll need to shut down your system, and blow it out thoroughly. Also replace your motherboard, as that's likely not salvageable. Your HBAs and drives can probably be salvaged.

Appologies its confusing, it was the sata dom that came apart not the motherboard, the circuit board of the sata dom detached from its sata connection

 

[Mutley]

Watching the Dos screen start up I've noticed I am getting a KDB: enter: panic
[ thread pid 2364 tid 101753 ]
stopped at kdb_enter+0x37: movq $0,0x1648ba6%rip)
db:0:kdb.enter.default> write cn_mute 1
cn_mute 0 = 0x1
db:0:kdb.enter.default> textdump dump
textdump complete.
db:0:kdb.enter.default> reset
cpu reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 6

 

[Mutley]

Got some stability now, i have reverted back to Trunas 12.0, so not sure if my install was poor, i did also replace motherboard but that had no effect, Does the config file put back windows shares and user accounts etc

 

[Samuel Tai]

Yes, uploading the config restores all the shares and accounts.

 

[Mutley]

Unfortunately I made one disastrous beginners mistake in my system and I didn't back up my configuration, which i admitted at the beginning of post and is slowly learning the hard way, so having read numerous posts about configs saved automatically, I'm struggling to get a copy of the automatic backups from my pool so I can restore my system.........I believe you are supposed to find them in cd /var/db/system then "ls" the directory to show all files, which I have done and it shows 4 "configs-aaaaaa111111" files, I assume I then "cd" to the configs-aaaaa11111 file but if I "ls" it shows nothing, again reading posts am I to believe its actually a directory not a folder and if so where are all the daily backups kept, how do you then access an individual backup, one of the backups looks like this
configs-0ad0015b3a7f402884da0c3d1f953e06 which i assume is correctly identified as a backup

 

[Redcoat]

There's a utility built into TrueNAS called Midnight Commander - use MC on the CLI to open it. To use it see https://www.linode.com/docs/guides/how-to-install-midnight-commander/ or https://linuxcommand.org/lc3_adv_mc.php . Navigation in it can be a bit opaque, but it will allow you to get to see the config files on your box. They are there, you just aren't seeing them with the method you have used.

Assuming you are using a windows client, better IMHO is WinSCP - https://winscp.net/eng/index.php , docs at https://winscp.net/eng/docs/start

Here's the output from one of my NAS boxes -

You will need to set up SSH access to your TrueNAS box in order to use WinSCP.

 

[Mutley]

Thanks Redcoat for the suggestions and help, after some reading and searching I used WinSCP and the 4 config files I had found were from the new install of Truenas, as the date confirmed being only a couple of days old, I believe any backups that were automatically made, were also saved on the sata dom, which was my original boot pool as I left most settings at default when originally installed, I can't find any hint of being saved on my data pool or jails pool, unless i am still doing something wrong.

 

[Redcoat]

I don't know what else to offer, except to say that "for years" the System Dataset has, by default, been moved to the first pool created (thus moved off the boot pool).

Did you look at the contents of each of the 4 config(long number) files that you found?

 

[Mutley]

I started to manually input settings to get emby up and running by inputting group and user id, I had managed to get Emby up and running but Truenas server has reverted back to rebooting every couple of minutes, I disconnected my jails pool and problem seemed to stop, reconnected and then it started rebooting.

 

[Mutley]

I've ended up deleting my Emby jail and reinstalling, this seemed the only way of fixing my issue, server has been stable for a few days, and no rebooting anymore, i also swapped out my 3 efax drives which are now replaced with the newer wdred plus drives, I've also mirrored my boot drive with another wd green ssd. Not really sure what the initial cause was, whether it was the supermicro sata dom that was starting to fail because of the solder joint, was most likely a factor, for anyone buying a supermicro sata dom they actually take up 2 spaces on the sata connections because they are very wide, and the circuit board of sata dom tends to be resting on parts of the motherboard, so i'd probably now always recommend an ssd over sata dom.