My freenas wont start up. Fatal Trap 12

[Magnus Svensson]

Hiya.

My hardware :
Build FreeNAS-9.3-STABLE-201602031011
Platform Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz
Memory 16270MB
I dont remember the motherboard, but its an Asus 1155 socket. Memory are DDR3 ECC.

I've been runing freenas for a few month now without any major problems.
But today one occurred. I was watching a movie on plex when it froze.
When I tried to access the web ui I got no response from it.
When I went to look on the computer it was a black screen and nothing could wake it up.

After a reboot I got an error message on fatla error.

Being a total green novice on freebsd I have no clue on what to do now.
Are my vdev on my raid span dead or are it my boot disk thats dead ?
Should I try to reinstall freenas and see if that helps ??

Regards Magnus Svensson

 

[danb35]

It's unlikely your data disks are dead. It's possible your boot device has died, but IIRC error 12 often indicates memory problems. Try doing a clean install of FreeNAS to a new USB stick, upload a saved config file, and see what happens. If you continue to have problems, I'd suggest letting memtest86+ run a few days.

 

[Sakuru]

Personally I would recommend running Memtest86+ first.

 

[Mirfster]

Check your BIOS, see if it shows any memory chips not functioning correctly. If needed remove those chip(s). See if FreeNas then gives you the same errors. Always make sure you have a backup. Then proceed with the suggestions by everyone else. FreeNas only needs 8GB to run, so ensure you're data is same first (as well as config) then you can proceed.

 

[Magnus Svensson]

Thanks.

I don't think that I have an current config file saved. Is there any way to get one from this state?
Or I'd it possible to import the disks into a new installed Freenas?

I'm going to try an replace the memory and see if it acts the same first.
Got a spare set of 16gb ram.

Thanks for all the advice

Regards Magnus Svensson.

 

[Magnus Svensson]

I've tried to replace the ram but it gives the same error.
But I tried the menu option Boot Environment.
There on "TrueOS (Initial-Install) - 2016-03-06 12:29" it booted up.
All other boot options that I can pick it fails on that part.

It stops on Begining ZFS volume import. So are the data on the boot device corrupt?
I can see all the memory in the bios just fine, bu boot disk and backup disk are there too.
I've been in the raid controller and checked all the disks and they are fine too.

So Would it help to reinstall the server?

Regards Magnus Svensson

 

[Mirfster]

raid controller

What do you mean by "Raid Controller"? Do you have a Hardware Raid card in there passing drives to FreeNas? Tell use about your hardware configuration and how you have things setup.

 

[Magnus Svensson]

I havea RocketRaid 2340 controller thats handeling a 16x 2TB samsungs disks.

The server are around 5 years old and it started out as an windows server for the first 4 years. But I wanted more security and option to set up a SFTP server.
I ran Ubuntu for almost a year, the drivers for the RR3240 controller was a pain to get working and after an accident with purging old drivers from /boot half of the raid drivers went out the door. So instead of installing and trying to get the controller to work in Ubunto so did I migate to freenas that have drivers in the kernel.

But as the system are today so do I have a 320GB boot disk, 1.5T backup disk for photos and my plex database.
Then the raid controller with all my disks on it.

I've been looking for a good way and cheep to replace the RR card, but I havent found any good cards for it.
Also I have around 20T of data on them atm so its little hard to have to start over again getting all the files.

 

[Robert Trevellyan]

Looks like pool corruption to me. You should still run a memory test.

Then the raid controller with all my disks on it.

Exactly how is that RAID card configured?

Can you get to any kind of shell and run camcontrol devlist? If so, please post the output, between CODE tags.

 

[Magnus Svensson]

Its a Raid 5 with 1 spare.

Here are the output from the camcontroll devicelist :

The last string on the fatal error are : vdev queue io to issue. I've also had it booting ut with a new set of ram with the same problem.
But I'll start a memtest and see how it turnsout.

 

[Spearfoot]

Its a Raid 5 with 1 spare.

RAID5? Do you mean RAIDZ1?

Or did you somehow install FreeNAS on your system with a RAID controller, passing disks to FreeNAS through a RAID subsystem of some kind?

 

[Magnus Svensson]

No, the boot disk are a 320gb sata disk, then the raid span comes as a own volume.
But its a raid 5, 1 spare configured in the raid controller. Freenas, Ubuntu and windows sees it as one big disk.

 

[Spearfoot]

No, the boot disk are a 320gb sata disk, then the raid span comes as a own volume.
But its a raid 5, 1 spare configured in the raid controller. Freenas, Ubuntu and windows sees it as one big disk.

I see. That's not a supported configuration for FreeNAS. With FreeNAS, you're supposed to use a HBA (Host Bus Adapter) -- not a RAID controller -- and pass each individual disk through to FreeNAS. By giving FreeNAS a RAID array to work with, instead of individual disks, you've disabled all of the tools it would have given you for disk administration.

You will have to use some other means of troubleshooting your RAID adapter and what seems to be a problem with it and your RAID5 array, because FreeNAS isn't designed to be used the way you've used it and doesn't have facilities for troubleshooting the kind of problem you have.

You say the RAID array is available to Ubuntu and Windows? Perhaps you can use those operating systems to troubleshoot your RAID array.

Good luck!

 

[Magnus Svensson]

The raids array are available in windows, i switched to Ubuntu as i wanted a SFTP and some more security and easyer to configure the tools as sab, couch and sonarr.
But the RR2340 card arent supported nativly in Ubuntu so it took me a few days to get the drivers to work. But as Ubuntu have a minimal /boot and it can fill up quite fast, so i had a bootpurge script that I ran when the server got really slow and /boot droped below 3% space.
Somehow after 7 month using that script some driver for the raid card got purged. I tryed to install the drivers with my old notes but I had to reformat the whole boot disk and to start over.
I know that FreeBSD had support for this drivers so I migated to FreeNAS for that :)

Yeah, I've learned that much that for it to work flawless so do I need to replace the card, but I havent found any card thats in my current economical range, and the swerver have been working really har to get the 20T that I've lost 2 times on 1.5 year lol
I'am going to do that on server 2.0 :)

I can see the Raid Array on the list too. Its the "HPT DISK 0_0 4.00"
I've also been in the raid adapters bios and all the disks are working and are healthy.

I don't have a saved config file that's fresh at all. Could FreeNAS import the old volumes if i reinstalled it and import the jails ?
Or do I have to start over from a fesh format on the disks ?

 

[Sakuru]

Could FreeNAS import the old volumes if i reinstalled it and import the jails ?

Yes

 

[danb35]

Yes

No, I don't think so. Since he's using a hardware RAID5, he'll need to destroy the array and build a ZFS pool using RAIDZ.

 

[Spearfoot]

Well, his FreeNAS-on-RAID5 was 'working'... until it failed. He might be able to re-install FreeNAS and import his ZFS/RAID5 volume. If he can do so, his first order of business should be to make a good backup... because he needs to re-build his system w/o the hardware RAID5 volume. It's a wonder this server ever worked at all...

 

[Robert Trevellyan]

Does da0 show up under View Disks in the FreeNAS GUI?

vdev queue io to issue

This suggests ZFS is unable to get a timely response from the RAID sub-system. Perhaps the RAID card is failing. Perhaps one of the drives is failing in a way that the RAID card doesn't report, and making it unresponsive.

What model are the drives?

My guess is the pool is lost, but I suppose there's an outside chance of recovery if you can somehow recover the RAID.

I see nothing to indicate a bad boot pool.

 

[Magnus Svensson]

So...
I've reinstalled FreeNAS and it works really good. But I've taken your advice and reformated the raid array.
Made a halfhearted try on "zpool import -f Share"
Sadly as wife are between jobs so would she kill me if I purchased a sata controller for 200 euros atm.

But Its on the list for server 2.0
The only 2 thing that's really sting now are all the photos that I've lost on a Single 1.5T disk that I had as backup for them.

So the next time I know that i need to suck-it-up and take it crying like a man LOL for around 3 more years. Then server 2.0 will be on the next to do list.
Then I'm planing to build it on 4 bay usb/eSata cases and set up each as a RaidZ1, I have one 4 bay now that I need disks to.


Thanks for the help guys, I've actually learnt a lot on this one.

Regards Magnus Svensson

 

[SweetAndLow]

No, the boot disk are a 320gb sata disk, then the raid span comes as a own volume.
But its a raid 5, 1 spare configured in the raid controller. Freenas, Ubuntu and windows sees it as one big disk.

If you are using raid your data is gone at this point. A proper hba controller is $100usd off eBay. Super cheap compared to the cost of your entire system.