TrueNAS SCALE 22.12.1 Upgrade - Kernel panic - not syncing: No working init found

[vm_boy]

A week or two ago I upgraded from 22.12.0 to 22.12.1. The upgrade seemed to be working fine, but then I've had a series of weather-related power outages which led to other server restarting. When I tried to connect, I realized that the server was working properly. I ended up putting a keyboard and monitor on it to figure out what was happening. I found that it was hanging up during the boot process with this error:

Code:

---[ end Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance. ]---

I have no idea what that means or how to do any of what it is telling me to do. I went to the Linux Documentation I understand that even less.

During the boot-up process, grub does give me the option to use the previous version so I have been able to do that as a temporary fix, but it still seems to be rebooting or something because the next day I will find it is not working again, so I am not sure what is going on there. I just now tried to re-upgrade thinking that might fix whatever kernel/init situation is going on.

If someone could please walk me through the process to fix this, I would really appreciate it.

 

[NugentS]

Its probably hardware. But even if it isn't thats what you need to test first.
Of course, as you don't have any hardware (see forum rules) there isn't a lot I can suggest other than:
1. Run memtest for at least 24 hours and see what happens
2. Make sure you have a config backup, not on the NAS for you to rebuild with

 

[vm_boy]

I was able to get it to boot into the new upgraded version of 22.12.1 so hopefully it has fixed itself. In case the problem persists, here are the hardware specs.

Motherboard: ASUS P8P67 Pro (Revision 3602)
CPU: Intel Core i5-2500k
RAM: 16 GB (G.SKILL Sniper Low Voltage (4 x 4 GB) DDR3 1600)
Disks:
Boot Drives: 2x MKNSSDCR60GB-7 (Mirrored)
Pools:
{Pool 1 - Mirrored}
2x ST4000DM004-2CV104

{Pool 2 - Striped}
1x ST40000DM004-2CV104
1x WDC_WD40EFRX-68N32N0
1x ST5000DM000-1FK178

{Pool 3 - Striped}
Seagate ST1000DM000-9TS15E

{Pool 4 - Striped}
Seagate STEA2000400

 

[vm_boy]

As it turns out, the issue has not been resolved. I was able to get it to boot last night and seemed to be running fine, then I wake up this morning and it's giving me the same kernel panic error message. So I don't understand why it's going from a perfectly booted state to this error state.
I backup/save the configs on another machine everytime I upgrade so I have a current one and some others. I will look into a memtest and see what happens.

 

[NugentS]

How many disks and what PSU are you using?

 

[vm_boy]

The power supply is a Corsair CX600M. I have 2 SSDs, 5 HDDs, and 2 external HDDs

 

[NugentS]

Can you please be more specific about how you have the storage arranged.
What attached to what, and in what array type.
Are pools 3 & 4 the external disks? [I hope so]
I would try removing the external disks (unplug) and see what happens - does it remain stable for a week

 

[vm_boy]

Can you please be more specific about how you have the storage arranged.
What attached to what, and in what array type.
Are pools 3 & 4 the external disks? [I hope so]
I would try removing the external disks (unplug) and see what happens - does it remain stable for a week

All of the internal drives are connected to the motherboard using SATA cables. The disks for Pools 3 and 4 are both external hard drives plugged in through USB ports. The there are (2) internal 60 GB SSDs mirrored as the boot devices. For the pool storage, I have one pool setup with (2) 4TB internal HDDs that are mirrored for special stuff like pictures and such. The other pool is just mass storage that is striped consisting of (2) 4TB internal HDDs and (1) 5TB internal HDD. I've got another 6TB to add to pool 2, but I haven't gotten it in yet.

Regarding the memtest, I was finally able to run it yesterday and it looks like I have one bad bit. The first digit of address 2EBEB5E0 is 4 off of what it's supposed to be, for example: 40000000 instead of 00000000. So maybe I can just pull that one stick and maybe it will run sufficiently until I can get a replacement. I'm not exactly sure how to figure out which stick, but trial and error couldn't be too hard. The HTML report is attached.

 

[NugentS]

Well DDR3 should be quite cheap.

Take half the memory out and re run the tests - if it works, put the sticks in the other slots and re run. You need to be certain its not a bad slot

 

[vm_boy]

It took a while, but after swapping different sticks in and out, I was able to identify the stick of RAM with the one bad bit. It's annoying that I can't use an entire stick because of one little bit, but whatever. I've got the other 3 sticks in now and it booted into TrueNAS 22.12.1. I'll let it do its thing and hopefully I won't have any more issues, and then I'll find another stick or four of RAM...if I'm buying one, I might as well max out my capacity.

 

[NugentS]

for ZFS extra memory is never a bad thing