TrueNAS 13.0-U2 is now available

[morganL]

So I've two screenshots. First one is right before the panic, second is it.

Please "report-a-bug".... especially if its replicable.

 

[Samuel Tai]

So I've two screenshots. First one is right before the panic, second is it.

OK, the panic happened in core_pcpu_init() when trying to load the kernel CPU initialization routine. Typically a problem here is due to the hypervisor reporting the wrong number of cores to TrueNAS. How many cores have you allocated to TrueNAS in ProxMox? How many physical cores do you have in your Ryzen CPU?

 

[jlpellet]

Hi,

just updated from TrueNAS 13.0-U1.1. Sadly U2 doesn't come up completely, I had to boot into U1.1 environment again. I tried twice, but U2 always makes the kernel panic.
Unfortunately I just don't know how to provide details. How can I help here?

Environment: TrueNAS 13.0-U1.1 in KVM on Proxmox. I'm using this environment since 12.0-U7 and updated always went smooth.

Bye,
Chris

No obvious answers but would try the following:
1) do a manual update rather than in-line update
2) ensure no more than 3 boot environments before the update
3) scrub the boot pool before updating
Good luck.
John

 

[Frosty0571]

Hi,
I have the same issue on Proxmox 6.4-13, running the update from TrueNAS-13.0-U1.1.
Getting the same kernel panic and switching back to U1.1 is working great.
CPU is configured as "2 (1 sockets, 2 cores) [host] [cpuunits=1000]".

 

[Samuel Tai]

Hi,
I have the same issue on Proxmox 6.4-13, running the update from TrueNAS-13.0-U1.1.
Getting the same kernel panic and switching back to U1.1 is working great.
CPU is configured as "2 (1 sockets, 2 cores) [host] [cpuunits=1000]".

core_pcpu_init() changed in upstream FreeBSD. Try configuring your CPU as 2 (2 sockets, 1 core).

 

[Frosty0571]

core_pcpu_init() changed in upstream FreeBSD. Try configuring your CPU as 2 (2 sockets, 1 core).

Thanks Samiel, I'll try!
Do you have any reference for the change of `core_pcpu_init()` in the FreeBSD code? Would be good to know.
Any link in the release notes of U2 would also be great.

 

[Samuel Tai]

Do you have any reference for the change of `core_pcpu_init()` in the FreeBSD code? Would be good to know.

src - FreeBSD source tree

cgit.freebsd.org

 

[adrianwi]

I've upgraded from U1 to U2 without any issues. Thanks to everyone involved!

 

[TrueChris]

OK, the panic happened in core_pcpu_init() when trying to load the kernel CPU initialization routine. Typically a problem here is due to the hypervisor reporting the wrong number of cores to TrueNAS. How many cores have you allocated to TrueNAS in ProxMox? How many physical cores do you have in your Ryzen CPU?

To answer all questions:
1. I didn't upgrade the pool since installing 13.0
2. I did a successful scrub.
3. The system has 24GB of RAM.
4. The system doesn't use a thumb drive but a virtual 10GB harddisk.

The TrueNAS VM gets two cores in one socket from Proxmox. The host CPU is a Xeon E5-2630 v2 (12 core). Virtualized CPU type is equal real CPU.

In the meantime I trimmed the boot environment to one (U1.1) before the update and switched from above CPU configuration to 2 sockets, 1 cores and also 1 sockets, 1 cores but behaviour stayed the same.

I don't use any Jails or VM inside TrueNAS.

So I just created https://ixsystems.atlassian.net/browse/NAS-117975

Regards,
Chris

 

[pkerwien]

To answer all questions:
1. I didn't upgrade the pool since installing 13.0
2. I did a successful scrub.
3. The system has 24GB of RAM.
4. The system doesn't use a thumb drive but a virtual 10GB harddisk.

The TrueNAS VM gets two cores in one socket from Proxmox. The host CPU is a Xeon E5-2630 v2 (12 core). Virtualized CPU type is equal real CPU.

In the meantime I trimmed the boot environment to one (U1.1) before the update and switched from above CPU configuration to 2 sockets, 1 cores and also 1 sockets, 1 cores but behaviour stayed the same.

I don't use any Jails or VM inside TrueNAS.

So I just created https://ixsystems.atlassian.net/browse/NAS-117975

Regards,
Chris

I have a very similar setup but with Intel Xeon CPU E5-2620 v4 CPU, 4 vCPUs, 16GB RAM for the TrueNAS VM and saw exactly the same problem when I tried to update from 13.0-U1.1 to -U2 yesterday.

 

[pkerwien]

I have a very similar setup but with Intel Xeon CPU E5-2620 v4 CPU, 4 vCPUs, 16GB RAM for the TrueNAS VM and saw exactly the same problem when I tried to update from 13.0-U1.1 to -U2 yesterday.

Changing the CPU model from host to kvm64 will make the system boot.

 

[Lefuneste]

Same issue here on Proxmox 7.2-7 on Intel i7 8800K CPU hypervisor. KDB panic after upgrading to U2. Changing the CPU model from HOST to KVM64 allowed the system to boot properly.

 

[mav@]

core_pcpu_init() function is a part of hwpmc(4) CPU profiling subsystem. It seems Proxmox virtualizes most of the PMC hardware, but not
MSR_DEBUGCTLMSR CPU register, that got used in 13.0-U2 to get more accurate profiles. It can be that switching CPU model you change emulation of that register. I've tested my work on VMware and had no problems there. But just to be safe I think I'll just block that new functionality when inside VMs. Any way profiling inside VM is not the best idea, it can't be very precise.

PS: Should be patched in next nightly build. Confirmation would be good.

 

[Volts]

next nightly build

Nightlies don't seem to be making it to the public, or maybe I'm using the wrong train name.

TrueNAS-13.0-Nightlies --> 202205050459

https://update-master.ixsystems.com/TrueNAS/TrueNAS-13.0-Nightlies/LATEST

TrueNAS-13.1-Nightlies --> 202206240813

https://update-master.ixsystems.com/TrueNAS/TrueNAS-13.1-Nightlies/LATEST

 

[SweetAndLow]

Upgraded from 13.1-U1.1 and everything is going great. Original install was 9.3 and I have upgraded ever since :) same usb key from 2015.

 

[House Of Cards]

Upgraded from 13.0-RELEASE to U2 and now none of my jails have DNS resolution. I ping google.com and I get an error that says "cannot resolve", and "host name lookup failure". Ughhhh.... TrueNAS shell can ping. Jail shells cannot.

 

[House Of Cards]

Multiple reboots not solving the DNS issue. Boot back into 13.0-RELEASE and everything works again.

 

[Samuel Tai]

How is your jail networking set up?

 

[House Of Cards]

How is your jail networking set up?

VNET and Berkely Packet Filter are enabled. VNET interface to auto, manually configured 172.16.2.0/24 address with default route of 172.16.2.1 and no IPv6. Network Properties with vnet0:bridge0 and ip4.saddrsel and ip6.saddrsel selected. Also checked are allow_set_hostname, allow_raw_sockets and allow_mlock.

It's been like this forever. Nothing different except for the update to TrueNAS.

All the jails run 12.2-RELEASE-p13 except one on 12.3-RELEASE.

 

[SweetAndLow]

VNET and Berkely Packet Filter are enabled. VNET interface to auto, manually configured 172.16.2.0/24 address with default route of 172.16.2.1 and no IPv6. Network Properties with vnet0:bridge0 and ip4.saddrsel and ip6.saddrsel selected. Also checked are allow_set_hostname, allow_raw_sockets and allow_mlock.

It's been like this forever. Nothing different except for the update to TrueNAS.

All the jails run 12.2-RELEASE-p13 except one on 12.3-RELEASE.

1. update your jails
2. could it be Realtek RTL8111H networking?