Fatal Trap 12 at boot advice

[Xycor]

Hello, I am new to FreeNAS and Linux. I was hoping with more experience could help me figure out if a failure is occurring on the hardware or FreeNAS side.

The box runs FreeNAS 9.1.2.2. It is a member of our ActiveDirectory network which probably has 30,000+ users. The box has about 30TB of storage on 10x 4TB Seagate NAS disks. The share is CIFS. The dual 8-core processors are Xeon's from about 2009. 16GB of RAM which I was going to increase if needed. sys volumes are on a separate 2TB drive.

The box has been working correctly for a several days across a couple of restarts. Friday evening I increased the load on the box as I began populating it as a backup of other data. The box dropped off the network within a few minutes. Unfortunately on Saturday our lab suffered a power outage so I could not get anyone there to inspect the console.

I am 1000+ miles from the actual machine so my ability to troubleshoot is limited. A co-worker took this screen shot this morning. This error now occurs every time the machine boots.

In short the error is:

Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 07
fault virtual address = 0x2b8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff818ab9b5
stack pointer = 0x28:0xffffff83705ed6b0
frame pointer = 0x28:0xffffff83705ed6d0

...

current process = 534 (txg_thread_enter)

Suggestions?

Thank You

 

[cyberjock]

Good thing you are new to FreeNAS and Linux.. cause FreeNAS/FreeBSD isn't linux. ;)

Fatal trap 12 usually means there was an improper interaction between the hardware and software. Usually this means bad or incompatible hardware with the OS. About the only thing you can ascertain with 100% certainty is that "something is broke". But, I think that's pretty obvious. ;)

If your AD domain has 30k users I'd definitely get more RAM. 16GB of RAM may not be enough for a domain of that size along with a server operating under normal conditions. There may be something going on with not enough RAM for the system to remain stable. 32GB is what I'd try for...

Unfortunately, your distance is going to make troubleshooting the problem much harder than you want. You're probably going to have to touch the system to fix it.

And your tunables are clearly garbage(I'm betting you enabled autotune at some point...). Notice the middle of the picture has your system update a few settings to... the exact same setting.

Autotune should only be used in accordance with the manual.. especially the part about "temporarily".

 

[Xycor]

Ah... oops. Well I guess calling FreeBSD Linux just highlights how new I am.

Thanks for the autotune suggestion. I'll RTFM. This is a learning box for me.

I'll go this route:
1. Add 16GB of RAM
2. If it fails ask someone to swap the MB/CPU/RAM or do it myself on the next visit.

 

[cyberjock]

That sounds like a good plan. ;)

 

[Deleted member 34036]

Hi,

I am also having a similar problem. Just to crosscheck is it a problem with the ram being malfunctioned or less amount of ram being used. Attached please find the snapshots of error and system info. Because I guess, less amount of ram may slow its performance, but won't degrade the system. i may be entirely wrong. Configured the system in raidz2,no deduplication. (30 x 3 TB = 90 TB raw physical drives space), Ram is : 32 GB DDR3-1600 ECC RDIMM

May kindly suggest.

Thanks,

regards,

 

[cyberjock]

Both bad RAM or insufficient RAM can be a failure mode.

From what we've seen, 8GB of RAM is the minimum to prevent possible damage to a pool that is a direct result of insufficient RAM. If you have some absurdly high amount of pool space compared to RAM it may be higher. But so far there's been no solid info of "how big of a pool" it would take. I don't think your pool is big enough for me to consider that.

Bad RAM is just bad. That's why ECC is so awesome.

Has the box been working fine and this just started? Have you tried upgrading to 9.2.1.4.1? I know there's been a boatload of fixes for FreeNAS in recent weeks.

 

[Deleted member 34036]

Hi,

Yes the box has been working fine since a month until I got this error this morning. I have not upgraded since I installed FreeNAS-9.2.1.2-RELEASE-x64. Yes I have been using ECC ram only. I had created a single pool in zfs raidz2 configuration with one drive of 120 GB ssd for cache. and kept some 5 drives as spares attached to the pool.

Just for your information I would like to mention here that I have one more system which is running perfectly fine with intel i7 based processor & motherboard with 16 GB normal ram. It is running perfectly fine since a year with freeNAS 8.3.1 version.

But this time, I thought to build my new system with a Intel E5 Xeon grade processor and with 32 GB ECC Ram and in raidz2 config for more for redundancy.
May kindly suggest, if I have to reinstall and do all the configs once again; as I don't have the last saved configuration also. Or is there any solution to get back my pool.

If its a FreeNAS issue, kindly suggest Which version to install.

Please suggest!

Thanks
regards,

 

[cyberjock]

sharfraj zaman:

If you don't have your config file, you can try to get it off the USB stick by mounting it in a FreeBSD machine and downloading the file /data/freenas-v1.db. If that's not possible or the file is corrupt you will have to recreate the configuration from scratch.

As for your pool, I don't know. Using an L2ARC with a system that has just 32GB of RAM isn't what I'd call a very well designed system. That may or may not have contributed to your problems. I never recommend an L2ARC until you have 64GB of RAM at the minimum, and 96GB+ is more of my "recommended" RAM. Don't ask why, I've explained it at least 20 times this year. Feel free to search if you want to understand it. I'm done trying to explain what I've already discussed plenty of times to people. The manual says to maximize your motherboard RAM before an L2ARC, and I'll stand by that comment in the manual.

I do offer data recovery services. It's not dirt cheap, but not excessively expensive either. But if you have a backup that's recent you should probably just use that. If you want to discuss data recovery options with me send me a PM.

 

[Deleted member 34036]

Hi,

I solved the problem Fatal trap 12 page fault while in kernel mode!!

Formatted the pendrive, installed the FreeNAS version once again, booted up, & AutoImport the Volume, Created the same groups and users and I got my cifs share perfectly with no data loss!! I dumped some 2 TB of data before the system showed this error and performing the above mentioned steps, I got my data as it is - Perfectly no data loss - Thank GOD!!

But one thing I would like to know from you, I am getting slow transfer speeds, as I am copying from USB hard disk to FreeNAS system (30MB/sec). Copying some 300GB size files. Could you please give me some idea how to increase CIFS performance, so that I get get some decent speed, touching 100 MB/sec, if possible.Kindly guide.

Thanks,

Good day!!

 

[gpsguy]

sharfraj zaman please start a new thread regarding your performance issue. Copying from USB will be slow.

I'm more concerned about some statements you've made. "I had created a single pool (30 x 3Tb disks) in zfs raidz2 configuration with one drive of 120 GB ssd for cache. and kept some 5 drives as spares attached to the pool." That's far too wide.

Do a google search of the forum using "site:forums.freenas.org wide pool"

 

[Deleted member 34036]

Hi gpsguy,

Regarding the statements I mean to say: I have configured the system in raidz2 setup. While configuring the pool, I took some 30 x 3 TB hDD and took the SSD as cache and kept another 5 disk as spares but attached to the pool. Finally did the cifs share. Have I made a mistake in configuring the system. please guide, as I am new to FreeNAS.

For a starter like me, by the system described above, which I have configured I understand: I will have a single pool with some 72 TB of usable space, One 120GB SSD cache will help for faster read/write and the spare drives in the event of any failure from the raidz2 setup pool/volume will take over the bad drive/ malfunctioned drive automatically. Have I understood something wrong. Please clarify. Although I am arranging for upgrading the Ram as advised by cyberjock. In the meantime Please guide me if I have configured the system correctly.

Thanks

 

[gpsguy]

Please run "zpool status -v" without the quotes from a command line and post the results in code tags. This will show us how you configured your pool.

BTW, the spare feature doesn't work. Maybe in a future version. Right now it's a manual process.

Sent from my phone

 

[Deleted member 34036]

Hi gpsguy Please find the results that came: And kindly advice how should I configure my system in raidz2.

[root@freenas ~]# zpool status -v
pool: Volume
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKS
UM
Volume ONLINE 0 0
0
raidz2-0 ONLINE 0 0
0
gptid/b7e901bd-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9bc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4i9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901ld-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9yc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4f9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901ad-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9bc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4b9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901zd-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9nc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4m9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901bd-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9vc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4o9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901id-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9sc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4g9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901xd-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9vc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4j9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901ld-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9pc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4q9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b828e9wc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4a9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901fd-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b828e9kc-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0
gptid/b86ed4e9-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
gptid/b7e901bd-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0
0

0
cache
gptid/b8a58d58-cd2d-11e3-b58c-386077dfcc33 ONLINE 0 0


spares
gptid/8d17da63-cd2e-11e3-b58c-386077dfcc33 AVAIL
gptid/8d17da64-cd2e-11e3-b58c-386077dfcc33 AVAIL
gptid/8d17da65-cd2e-11e3-b58c-386077dfcc33 AVAIL
gptid/8d17da66-cd2e-11e3-b58c-386077dfcc33 AVAIL
gptid/8d17da67-cd2e-11e3-b58c-386077dfcc33 AVAIL
0

errors: No known data errors
[root@freenas ~]#


Thanks,

 

[cyberjock]

You didn't put it in code tags. Copy and paste it again, and put it in code tags. The formatting has made your output useless to us.

But, I will say that the fact you *appear* to have a 30 disk vdevs tells me you should stop right now and not go any farther with FreeNAS. You are clearly not experienced enough to handle a FreeNAS server. Pay someone with more experience and knowledge to handle this before you end up in a situation where you lose your data.

Edit: Actually.. stop trying to hijack someone else's thread and go back to your own. Post in your already existing thread instead of dropping into someone else's. Thanks.

 

[Deleted member 34036]

Sorry! But as the question was raised in this thread only, so I thought to reply in the same thread, just to keep a continuation.
Thanks,