"Stopped at kdb_enter+0x3a: movl" - After system freezes he doesn't boot anymore

[charlie04]

Hello guys, first of all, follows my system specs:

I'm using Intel old server with 7x1tb SATA disks + only 2gb of server memory
Freenas release is 9.2.1.7-RELEASE-x64 (fdbe9a0)

During the boot up It got stuck for about 2 minutes (don´t remember if that´s a regular timing):

And then, after a flash, It came to:

I´m quite a bit rusty about Linux, but that sound like a panic.

I'm already unplug all disks and plug again, reboot several times and the problem persists.

If someone can help me I appreciate it.

Regards.

 

[dlavigne]

Try burning a fresh USB stick and booting from that.

 

[charlie04]

Hello dlavigne, you mean run any chkdisk using freenas boot iso ? Witch option I need to run on boot ?

Thanks

 

[dlavigne]

No, I mean it looks like your USB stick is bad. You need to buy a new stick, install FreeNAS 9.2.1.7 on it, boot from the new stick, and restore your config.

 

[charlie04]

Can I use CD-Room drive to boot and try to restore my config ?

 

[dlavigne]

Nope. This is why you're supposed to save a copy of your config whenever you make config changes.

 

[charlie04]

I didn't make any config change since I put them running, few months ago.

He freezes several times in last months but I supposed that is memory gap according to other threads here. I'm waiting new memories to increase server, but last night during one more freeze the server shows this strange message. I have important data stored in ZFS, lost data is not an option on this case.
How do I can perform any type of sys fix on this case? Burn a fresh copy of freeNAS and try to restore it ?

 

[dlavigne]

Yup. If it is the USB stick that failed, your storage data is not affected (this is why the OS is installed on a separate media than the storage disks). You will have to recreate your config from scratch though if you don't have a saved copy and will have to reimport your storage disks after the install.

 

[charlie04]

I guess the system wasn't installed on a separate media, I used stripe array default to install freenas. The system is on the same array that all storage, thats the reason I've asked you about try some recovery using CD-Rom boot. If I try to reinstall or upgrade using CD any chance to fix the problem?

 

[dlavigne]

Nope. Time to buy a beer to cry into.

 

[charlie04]

damn, I need to find a solution...any test/recovery can be performed on "db>" ?

 

[gpsguy]

Don't give up yet.

First of all I'd shut it down until you either have 8Gb of RAM for it or have another suitable machine that you could use.

Using ZFS on FreeNAS with 2Gb of RAM (1/4 the minimum) is just asking for trouble.

This being said, the FreeNAS OS cannot be installed on your storage volume(s). Did you have the OS installed on a flash drive (as recommended) or perhaps ONE of your SATA drives?

If the latter, then you'd need to replace it - but given that you don't seem to know what you did, don't mess with the hard drives now.

Install 9.2.1.7 on a new 4Gb+ Gb flash drive and configure your machine to boot from the flash drive.

If we can get to that point, we can run some command line options to learn about your disks, pool, etc.

Don't run random commands you find on this forum or the Internet. Please wait for assistance in this thread.

Please don't hook up any drives, until you have a machine with 8Gb RAM.


Sent from my phone

 

[charlie04]

Thanks for support gpsguy.

It's a terrible situation, it happens at one remotely site, there is no IT guy with knowledge onsite, I'm trying to do my best with no iLO/Drac available plus one person locally trying to help me.
The scenario is: there is no memory spare to increase on server now. The only thing that I can do is install freenas on flash drive and put on server to boot them, what you think ? I guess its the only possible action now..

 

[charlie04]

Update: I got it boot the server using a bootable flashdrive, I'm seeing all my conf as local boot have but with a slowly performance, now I need to know how to fix the problem in local freenas installation.

Thanks.

 

[gpsguy]

With just 2Gb of RAM it's going to be slow. ZFS liked RAM and with less than 4Gb pre-fetching (caching) is disabled.

And, as I said before running with less than 8Gb is like playing with fire. Check the docs for the minimum requirements. If you want to proceed without upgrading - backup any important data, while you can still access it.

Don't worry about the original installation right now. Once FreeNAS loads - it runs from RAM, so a hard disk is unnecessary.

Since it appears that you have little or no documentation on how the box was configured, you need to work on that.

I can give you some commands to run, but I won't be able to do so, for a couple of hours, when I have access to my pc. I've been typing these messages on my phone.





Sent from my phone

 

[charlie04]

I really know thats isn't a good configuration, I have 4 others freeNAS on my environment, one per site, using a good config and never had problems, but when this specific system was configured on this site we do not have resources and we needed any form to store legacy systems, by te way, I'm working on memory upgrade.

I disabled ISCSI connection between freeNAS and win server, maintain only NFS that I have most important data to use minimum performance required. I'm using NFS to share storage with XenServer, I have legacy VMs on this storage no need for a lot of months, just today when someone needed the system crashes....Murphys Law.

I'll perform backup of important data that some people are annoying me and work on system troubleshooting. When you have some time post the commands that you have to help me.

Thanks for support.

 

[gpsguy]

If you were using iSCSI, even more memory is generally required. For your drives, probably 16Gb or so. And, for best performance, one should use striped mirrors.

Saying "I used stripe array default to install freenas. The system is on the same array that all storage..." it sounds like you don't know much about your configuration. If you are using a stripe for your data, combined with 1/4 of the minimum RAM, you need to keep backups of import data. This is a dangerous situation.

If you have 5 FreeNAS servers, you should know already know these commands.

camcontrol devlist
gpart show
zpool status

Run them from the command line, capture the output, study them, and figure out what you have.

If you weren't booting off a flash drive before, my guess, is that it was one of your hard disks? But, which one? If you didn't have it documented, I didn't want you/your local guy randomly pulling disks (not knowing whether your storage is striped, some form of RAIDz, etc) and potentially loosing all your data if you reinstalled FreeNAS on the wrong hard disk.

Folks normally run FreeNAS 8-9.2.x.x from a flash drive, since it only uses ~2Gb of space and the rest of the space is unusable. It also frees up a SATA port.