Kernel Panic on ZFS import

[mtucker502]

I'm getting a kernel panic when I boot FreeNAS. I traced this down to when the zfs import takes place. I've tried various zfs debug options and importing the zpool as read only and it still won't import.

A zdb scan had around 12k hours to finish so I stopped that :)

That's when I noticed Freenas is only showing 4GB of RAM installed. BIOS still shows all 4 2GB DIMMs but vmstat/top both show only 4GB. I believe this is why I'm seeing the kernel panic (though not 100% sure). I ran memtest86 overnight thinking I had a memory issue but after multiple passes there are no errors.

[root@freenas] ~# vmstat -h
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 ad0 in sy cs us sy id
0 1 0 1768M 2567M 966 1 12 16 872 0 0 0 32 1591 451 5 1 94

[root@freenas] ~# top -d1
last pid: 8945; load averages: 0.00, 0.00, 0.01 up 0+00:39:49 07:41:59
33 processes: 1 running, 32 sleeping
CPU: % user, % nice, % system, % interrupt, % idle
Mem: 411M Active, 148M Inact, 328M Wired, 63M Cache, 2501M Free
ARC: 136M Total, 29M MFU, 98M MRU, 144K Anon, 1376K Header, 7362K Other
Swap:


I'd like to try and figure out why the memory is showing at only 4Gb and anything else the gurus can think of before I attempt an import -XFR as a last resort.

Thoughts?

 

[Darren Myers]

full systems specs will help us

 

[mtucker502]

I'll be honest, I'm not sure exactly what command output to send to give the most concise information needed. I looked for a sticky that detailed "required information" but didn't see one. (Maybe a good idea for a sticky if not already).

Tell me what you need and I'll get it asap.

 

[Darren Myers]

motherboard, cpu, ram, psu, HBA/RAID card. All of those should have a make and model, such as:

Motherboard :Supermico x10-something something
CPU: intel or AMD with model
RAM: amount, kind, ecc, non ecc
PSU: what make/model/brand/wattage
HBA/RAID: LSI 9211-8i, m1015, etc

 

[Robert Trevellyan]

And of course, exact FreeNAS version, because one thing that can cause this is running a 32-bit build.

 

[SweetAndLow]

I'll be honest, I'm not sure exactly what command output to send to give the most concise information needed. I looked for a sticky that detailed "required information" but didn't see one. (Maybe a good idea for a sticky if not already).

Tell me what you need and I'll get it asap.

It's at the top of the page in red text:

https://forums.freenas.org/index.php?threads/forum-rules.22553/#post-134570/

 

[mtucker502]

Base Info:

• Motherboard: ASRock P45R2000-WiFi
• CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz
• RAM Size: 8GB (4x2GB)
• Hard Drives: 8 2 TB WD Green in RAIDZ2
• HBA: Onboard & 1 Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)
• PSU: I'm honestly not sure without cracking open the case. I don't remember.
• FreeNAS-9.3-STABLE-201506292332

I'm running 64bit:

Code:

[root@freenas] ~# uname -m
amd64

My memory shows as only capable of 4 GB: (line 11)

Code:

[root@freenas] ~# dmidecode -t memory
# dmidecode 2.12
SMBIOS 2.5 present.

Handle 0x000E, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None
Maximum Capacity: 4 GB
Error Information Handle: Not Provided
Number Of Devices: 4

Handle 0x0010, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x000E
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 2048 MB
Form Factor: DIMM
Set: None
Locator: DIMM0
Bank Locator: BANK0
Type: SDRAM
Type Detail: Synchronous
Speed: Unknown
Manufacturer: Manufacturer0
Serial Number: SerNum0
Asset Tag: AssetTagNum0
Part Number: PartNum0

Handle 0x0012, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x000E
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 2048 MB
Form Factor: DIMM
Set: None
Locator: DIMM1
Bank Locator: BANK1
Type: SDRAM
Type Detail: Synchronous
Speed: Unknown
Manufacturer: Manufacturer1
Serial Number: SerNum1
Asset Tag: AssetTagNum1
Part Number: PartNum1

Handle 0x0014, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x000E
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 2048 MB
Form Factor: DIMM
Set: None
Locator: DIMM2
Bank Locator: BANK2
Type: SDRAM
Type Detail: Synchronous
Speed: Unknown
Manufacturer: Manufacturer2
Serial Number: SerNum2
Asset Tag: AssetTagNum2
Part Number: PartNum2

Handle 0x0016, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x000E
Error Information Handle: Not Provided
Total Width: 64 bits
Data Width: 64 bits
Size: 2048 MB
Form Factor: DIMM
Set: None
Locator: DIMM3
Bank Locator: BANK3
Type: SDRAM
Type Detail: Synchronous
Speed: Unknown
Manufacturer: Manufacturer3
Serial Number: SerNum3
Asset Tag: AssetTagNum3
Part Number: PartNum3

I've attached all the command output.

 

[Robert Trevellyan]

My memory shows as only capable of 4 GB: (line 11)

I have a feeling that line is saying the max size per DIMM is 4GB, not the max supported total RAM, since underneath it shows 4 2GB 'devices'.

That's when I noticed Freenas is only showing 4GB of RAM installed.

Are you seeing this on the System Information page in the GUI?

At this point your best bet might be to move your array to a better system with more RAM, see if you can import it there, and if so, backup your data before proceeding. Your current system lacks ECC support and your secondary HBA has a poorly supported chipset.

 

[Darren Myers]

thats a bad motherboard for this case....supports DDR2 and DDR3.. http://www.newegg.com/Product/Product.aspx?Item=N82E16813157131 maximum supported RAM is 8Gb. From a HW reviews site: http://www.hardwarecanucks.com/foru...srock-p45r2000-wifi-motherboard-review-2.html

Memory [/B]
- Dual Channel DDR3/DDR2 memory technology
- 4 x DDR3 DIMM slots
- Supports DDR3 1333/1066 non-ECC, un-buffered memory
- Max. DDR3 capacity: 8GB, max DDR2 capacity: 4GB
- 2 x DDR2 DIMM slots
- Supports DDR2 1066/800/667 non-ECC, un-buffered memory

 

[mtucker502]

I understand I should be running more RAM, and ECC RAM at that. This is an old desktop that I repurposed. I'm shopping for my next NAS which will have lots of ECC RAM.

The system page shows 4GB: http://imgur.com/i89Ms4d. I didn't notice this until I tried a fresh install of FreeNAS. I thought at first I had downloaded a 32-bit version...if one even still exists. But uname shows it's amd64...

This system has been running fine for ~2 years now. I actually started with 4GB and quickly realized I needed more so maxed out the mainboard. I have a gut feeling (maybe a hopeless cause) that if I can find out why FreeBSD/NAS only sees 4 GB of ram that I can get it back up to 8 GB and at least import the volume without a panic. Any ideas?

 

[Robert Trevellyan]

That screenshot actually shows quite a bit less than 4GB. Do you have RAM allocated to VRAM in the BIOS?

Did memtest detect and test all 8GB?

Is it DDR3?

Are you sure FreeNAS used to see 8GB?

Is there anything weird about your pool, such as ... <shudder> ... deduplication?

 

[mtucker502]

1. That screenshot actually shows quite a bit less than 4GB. Do you have RAM allocated to VRAM in the BIOS?
   1. I don't have anything special configured in the BIOS for VRAM
2. Did memtest detect and test all 8GB?
   1. Yes
3. Is it DDR3?
   1. Yes, DDR3
4. Are you sure FreeNAS used to see 8GB?
   1. Positive
5. Is there anything weird about your pool, such as ... <shudder> ... deduplication?
   1. I did have compression enabled a few months back on a few of the datasets, but I've since deleted those datasets.

 

[Robert Trevellyan]

I did have compression enabled a few months back on a few of the datasets, but I've since deleted those datasets.

I wouldn't expect compression to cause any problems, so I wouldn't hesitate to enable that again once this is resolved.

I've asked all the obvious questions I can think of. The only other thing I can think of is just some kind of incompatibility with FreeBSD and your hardware. Have you tried reverting to the version you were using before?

 

[mtucker502]

I tried 9.2.1.2 and it has the same kernel panic and memory issue. I can't figure this one out :/

Here is screenshot of the kernel panic:
http://imgur.com/Z71PkMC

I'm running ZDB and I have seen one error so far:
http://imgur.com/bcp1Lvn

I've read that depending on the bootloader options used ZDB will also repair the ZFS volume. I can't find anything in the documentation or man pages for ZDB that indicate this but I've got nothing but time at this point.

Disclaimer: In the above screenshots I was giving nas4free a shot to see if I had any better luck with importing......I didn't.



Edit: replaced images with imgur links.

 

[pirateghost]

Please don't use postimg. They try to infect mobile devices with their shitware

 

[Darren Myers]

can you take a picture of the BIOS screen and possibly the memory section of the BIOS screen so we can see all the RAM and the size DIMM's

 

[mtucker502]

Please don't use postimg. They try to infect mobile devices with their shitware

I've never heard of them before this morning but Chrome keeps crapping on with imgur. Figures I had to use IE/Firefox.

 

[mtucker502]

can you take a picture of the BIOS screen and possibly the memory section of the BIOS screen so we can see all the RAM and the size DIMM's

I'll have to wait until ZDB is finished; it still has 41 hrs to go.

 

[danb35]

I've never heard of them before this morning but Chrome keeps crapping on with imgur. Figures I had to use IE/Firefox.

Well, you could just upload the pictures here rather than to some external host. If you're using the reply pane at the bottom of the thread, click the More Options... button, which will take you to a new reply window. Below that window will be a button for Upload a File.

 

[Robert Trevellyan]

I'm running ZDB and I have seen one error so far:
http://imgur.com/bcp1Lvn

Those CAM status errors are a classic symptom of problems with a Marvel-chipset HBA, though they could just as easily indicate a bad cable.