Kernel panic on boot when SMP is enabled on QNAP TVS-473

[angryant]

Hi,

I've a problem with running FreeNAS in QNAP TVS-473. This is a box with quite exotic CPU:

Code:

CPU: AMD Embedded R-Series RX-421BD Radeon R7		(2096.11-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x660f01  Family=0x15  Model=0x60  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x2febbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC,<b25>,DBE,PTSC,MWAITX>
  Structured Extended Features=0x1a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2>
  XSAVE Features=0x1<XSAVEOPT>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics

FreeNAS panics on boot with stack trace pointing at function: native_start_all_aps() function which indeed can panic if start_ap() returns an an error.

Code:

if (!start_ap(apic_id)) {
/* restore the warmstart vector */
*(u_int32_t *) WARMBOOT_OFF = mpbioswarmvec;
panic("AP #%d (PHY# %d) failed!", cpu, apic_id);
}

Code taken from here: https://github.com/freebsd/freebsd/blob/release/11.1.0/sys/amd64/amd64/mp_machdep.c

FreeNAS works like a charm for weeks if I start it with "set kFreeBSD.kern.smp.disabled=1" option, but I'm leaving a lot of performance on the floor which bothers me.

Question #1: anyone knows what this could be and how to fix it?

Question #2: how to build custom kernel which is identical as the one built by FreeNAS? (I'm happy to do all necessary tests to fix it)

 

[Chris Moore]

Hi,

I've a problem with running FreeNAS in QNAP TVS-473. This is a box with quite exotic CPU:

Code:

CPU: AMD Embedded R-Series RX-421BD Radeon R7		(2096.11-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x660f01  Family=0x15  Model=0x60  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x2febbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC,<b25>,DBE,PTSC,MWAITX>
  Structured Extended Features=0x1a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2>
  XSAVE Features=0x1<XSAVEOPT>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics

FreeNAS panics on boot with stack trace pointing at function: native_start_all_aps() function which indeed can panic if start_ap() returns an an error.

Code:

if (!start_ap(apic_id)) {
/* restore the warmstart vector */
*(u_int32_t *) WARMBOOT_OFF = mpbioswarmvec;
panic("AP #%d (PHY# %d) failed!", cpu, apic_id);
}

Code taken from here: https://github.com/freebsd/freebsd/blob/release/11.1.0/sys/amd64/amd64/mp_machdep.c

FreeNAS works like a charm for weeks if I start it with "set kFreeBSD.kern.smp.disabled=1" option, but I'm leaving a lot of performance on the floor which bothers me.

Question #1: anyone knows what this could be and how to fix it?

Question #2: how to build custom kernel which is identical as the one built by FreeNAS? (I'm happy to do all necessary tests to fix it)

You might want to submit a ticket on redmine if you want a developer to look at this.
https://redmine.ixsystems.com/projects/freenas

 

[angryant]

You might want to submit a ticket on redmine if you want a developer to look at this.
https://redmine.ixsystems.com/projects/freenas

Thank you. Filed: https://redmine.ixsystems.com/issues/34621
Added some more detailed info there.

In case someone runs into the same issue: there are two workarounds which worked for me (both require to set some boot arguments via Grub bootloader):

1. set kFreeBSD.kern.smp.disabled=1
2. set kFreeBSD.hint.lapic.18.disabled=1
   set kFreeBSD.hint.lapic.19.disabled=1

The first one disables multiprocessor support altogether. The second one disabled just two last cores. FreeNAS then can utilize two cores, which is not ideal but still better than one.

 

[John007]

Thank you. Filed: https://redmine.ixsystems.com/issues/34621
Added some more detailed info there.

In case someone runs into the same issue: there are two workarounds which worked for me (both require to set some boot arguments via Grub bootloader):

1. set kFreeBSD.kern.smp.disabled=1
2. set kFreeBSD.hint.lapic.18.disabled=1
   set kFreeBSD.hint.lapic.19.disabled=1

The first one disables multiprocessor support altogether. The second one disabled just two last cores. FreeNAS then can utilize two cores, which is not ideal but still better than one.

Hi Angryant,

I've been following this and I'm getting the exact same error.
Could you please educate me where do you actually set these parameters?
Do you somehow edit the installation files in the USB drive that I created, or is it done on the "KB" prompt which when Panic Happens, the system show?

If that's where you do it, then when I do the set command with your paramets, it says "Unrecognized something"... i can check the exact error.

But if you can tactically guide me to the exact solution? I would appreciate it. Thanks.

 

[angryant]

Hi John007,

I did it on boot, by typing the args in. When Grub presents kernel selection screen you press 'e' which takes you to an edit screen where you have opportunity to add these entries (just look for lines starting with set kFreeBSD) and place additional ones there. Then boot the system using this new config (I believe Grub displays proper keyboard shortcuts at the bottom of the screen).

Since this was a major PITA I decided to install stock Linux-based OS on this box. I'm not proud of it but I didn't want to spent too much time on fighting with this. Overall this box unfortunately turned out to be a piece of junk for me (one drive bay failed shortly after warranty expired).

 

[John007]

Hi sir,

Thank you so much for the response. I was able to get to that point with Grub. Did a lot of reading on it.

Overall, this box, I totally agree, is complete gargbage. I bought it 2.3 years ago. TVS-873 (8 Bay) with 64 Gigs of RAM. Eight enterprise NAS drives with 8 TB each and TWO Samsung EVO M.2 SSD at 500 GIG each.

I was running Qtier. and then literally a month ago, SERIOUSLY 7 weeks outside my warranty, 3 drive failure. After a lot of support talks, looks like the actual NAS box pooped. They will not cover but thankfully Seagate had a 3 year warranty so they covered me.

But, since I spent so much money on this box and it has HOT SWAP bays, out of which 5 still work, I decided to use it, but NOT with QTS anymore. After further review of my Firewall logs, they collect WAY TOO MUCH data. And if I block it, the QTS pretty much cripples and doesn't really work that well.

Lo and behold, I decided, let's try FREE NAS on it. Hence I was here.

But after 18 hours of experimentation, I decided to just Load UBUNTU on it.
I took out the original DOM DRIVE from the board, and just put it in a drawer.
I could boot UBUNTU from USB, but decided to run it on the two M.2 SSD I have in RAID1. So far, so good :) :)

I want to get most out of this system. But let's see.

In the meanwhile, I'm spinning up an old Dell Poweredge to see if I like FreeNas.