TrueNas VM Kernel Panic

[nate-wilkins]

Hi fellow NAS users. I'm having the best of luck with my NAS and well why not start the New Year off strong...with a kernel panic! Yay!

I don't know how to debug stuff at this level and even had to ask for help last time this happened but at the FreeNAS OS level - quite likely not the same issue but still thought it was funny.

So my setup is:

- TrueNAS@13.0-U3.1
- Intel(R) Core(TM) i7-2700K CPU @ 3.50GHz
- RancherOS in a Virtual Machine

When booting up the VM (after changing some `ros` configuration at runtime) I get the following kernel panic:

[ ] init:info: [18/21] Starting set proxy env [ ] init:info: [19/21] Starting init SELinux SELinux: Could not load policy file /etc/selinux/ros/policy/policy.29: Read-only file system [ ] init:info: [20/21] Starting setupSharedRoot [ ] init:info: [21/21] Starting sysinit [ ] init:info: Launching System Docker [ 31.895559] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 31.895559] [ 31.897206] CPU: 5 PID: 998 Comm: system-dockerd Not tainted 4.14.73-rancher#1 [ 31.898365] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 13.0 11/10/2020 [ 31.899355] Call Trace: [ 31.899752] dump_stack+0x5a/0x6f [ 31.900391] panic+0xd8/0x238 [ 31.900989] ? css_set_move_task+0xc3/0x128 [ 31.901809] do_exit+0x50a/0x923 [ 31.902365] do_group_exit+0x9f/0x9f [ 31.903050] get_signal+0x42d/0x45a [ 31.903666] ? do_futex+0xdb/0x892 [ 31.904246] do_signal+0x23/0x569 [ 31.904763] ? ktime_get_ts64+0x37/0x76 [ 31.905316] ? _copy_to_user+0x22/0x28 [ 31.905864] ? poll_select_copy_remaining+0xf3/0x109 [ 31.906615] ? SyS_futex+0x12c/0x13c [ 31.907163] prepare_exit_to_usermode+0x91/0xf5 [ 31.907910] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 31.908718] RIP: 0033:0x45d3f3 [ 31.909165] RSP: 002b:00007f19ca36dbf8 EFLAGS: 00000286 ORIG_RAX: 00000000000000ca [ 31.910361] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 000000000045d3f3 [ 31.911427] RDX: 00000000

This is frustrating and I'm not sure how to go about fixing it. I've tried restarting the VM numerous times but it panics every time.
Maybe I'm not meant to tinker with operating systems or virtual machines in general.

But to be optimistic I guess what I'm looking for is:

- Can I fix this without making a new VM?
- If I create a new VM, how do I get the data that I had on this VM?
- Is there a way to possibly fix this by modifying the VM disk data? (I know very little about VMs)

My guess is that Could not load policy file /etc/selinux/ros/policy/policy.29: Read-only file system is the culprit since I was modifying ros configuration and might have messed up the permissions but I'm just not sure because I can't look at the VM disk data or change it to the correct configuration.

 

[sretalla]

Can I fix this without making a new VM?

Can't say for sure... maybe try with a different CPU type specified in the settings.

If I create a new VM, how do I get the data that I had on this VM?

Add that VM disk as an additional disk to a working linux VM and do your data copy

Is there a way to possibly fix this by modifying the VM disk data? (I know very little about VMs)

hard to say without really knowing the problem, but same as the previous answer.

 

[nate-wilkins]

Hi sretalla! Thanks for getting back to me so fast, I'll admit I wasn't expecting a reply that fast and moved on to other things... But thank you!

maybe try with a different CPU type specified in the settings.

I can't seem to find that in the settings under the VM?

Add that VM disk as an additional disk to a working linux VM and do your data copy

Interesting that seems like a good idea.

Looking at the GUI I can't seem to find the volume though. Steps I took:

1. Go to Virtual Machines
2. Create
3. Set Operating System Details
4. Set CPU and Memory
5. Modify Disks (use existing disk image)
6. Try to Select Existing zvol (not listed)

It also looks like the Disk device isn't even listed? I don't know if it was there before but I'd assume so because I was storing stuff on the disk...

I can find the directory of files that was on the VM:

Thanks for the help!

 

[sretalla]

Your disk is a Raw file device, so shows as a RAW device.

That also explains why you can't find it as a ZVOL (since it's just a file in a dataset, not a ZVOL).

You can add it to a VM by adding a device of RAW type and then specifying the file location.

 

[sretalla]

I can't seem to find that in the settings under the VM?

Yes, I was talking about settings available on SCALE... I see that you're on CORE, so you can discount that point.

 

[sretalla]

Also, since you're talking about RancherOS... there's a good thread somewhere with input from @Samuel Tai about that...

HELP: please setting up RancherOS

Hi, I'm following this https://www.truenas.com/community/resources/how-to-rancheros-on-freenas-11-0.50/ I've setup a iohyve pool with; iohyve setup pool=WD20EARS kmod=1 net=em0 this has created a folder '/mnt/iohyve' I wget the iso into my root user folder in shell but when I do cpiso I get...

www.truenas.com

Note Samuel's post about end of support.

Also find a few posts down a how-to that may help.

 

[nate-wilkins]

I did read that RancherOS was ending support which is unfortunate since I spent time setting it up but onto bigger and better things I guess!

Okay that makes sense. I'll try to attach the Raw File device to a new VM to see if I can debug further.

Definitely a start.

 

[sretalla]

I did read that RancherOS was ending support which is unfortunate since I spent time setting it up but onto bigger and better things I guess!

I note that the links from Samuel point to burmilla OS as the (one of the) next steps (and can even be easily upgraded)... unfortunately seems to be stuck at beta4 of their 2.0 version since 2021, so maybe that's dead too. 1.9.5 seems to run fine for now though and would be a step forward from a years old rancherOS install.

 

[nate-wilkins]

@sretalla so I've attached the Raw Disk but not entirely sure how to inspect it.

Do you have any commands I should run/look into for more info?

Edit

I was able to mount the device and am looking into the files that I modified.

mount /dev/sdb1 /mnt/rancher_os

 

[nate-wilkins]

Yup. At least this kernel panic was due to a direct change that I made.

I updated the RancherOS cloud-config.yml with `system-docker` set to insecure registries being a local docker registry - that, funny enough, is started by RancherOS. This was the root cause and removing those lines allowed me to boot back into the VM.

Thanks a lot for the help @sretalla Learned a little bit and feel more comfortable debugging these sort of things.