SOLVED Freenas 9.3 was running fine until a sudden kdb panic

[HimuraMOdo]

Hi there,

Well, my Freenas 9.3 was running fine for a couple months until I went on my vacation last week. A couple days into my vacation, I received an email from freenas saying one of my raidz pool is unavailable as follows:
The volume Backup (ZFS) state is UNAVAIL: One or more devices are faulted in response to IO failures.

Then I tried to log into my freenas web GUI remotely, no response. But I can access my virtual machine remotely. I figured nah, nothing to worry about. It might just need some reboot that's all.

When I got home today, go down stairs to my Freenas box, pressed enter trying to get back to console. Nope. It simply kept on repeating some lines which are too fast for me to read. At this time, I can still access my virtual machine but not my zpool data.

So I hit the reset on the box.

Next thing I saw, the machine rebooted and try to load freenas then panic and halted. See attached picture.
I've searched the forum for this kind of panics. But they usually come in right after installing Freenas. And the root cause is usually not having enough RAM. But I'm not. Here is my machine hardware:
AMD FX8310
16G DDR3(non ECC, I know... It's an old machine and won't support ECC anyway.)
4X1TB HDD
1X3TB HDD
Dual 32G USB stick for Freenas.
Virtual machine was using about 4~6GB of Ram I think.

So, What is causing this? Anybody has an idea? Is it just bad luck that I had a random ram error since I haven't really reboot the machine for a long time and crashed the heck out of it?
How should I fix this? Reinstalling the Freenas?

 

[DrKK]

Virtual machine?

I don't follow. Is your FreeNAS virtualized? Or, you have a jail with PHPVirtualBox?

 

[cyberjock]

That looks like a corrupted zpool.

You are using AMD, non-ECC RAM, may have virtualized FreeNAS andmay not have given FreeNAS the minimum 8GB of RAM we call for in the manual. So don't take offense, but I'm done with this thread.

You've ignored at least 2 of our very strong warnings about how to not losing data, and potentially as many as 4. If you didn't take our advice before, I'm not sure why you'd be asking us for more advice now...

 

[Robert Trevellyan]

How should I fix this? Reinstalling the Freenas?

Assuming your answer to DrKK is "the latter"...

Sometimes a bad or corrupted USB stick causes this kind of problem, so you don't have much to lose by trying a clean install of FreeNAS to a new USB stick and uploading your saved config. Maybe do this after running a few passes of memtest.

 

[HimuraMOdo]

Virtual machine?

I don't follow. Is your FreeNAS virtualized? Or, you have a jail with PHPVirtualBox?

Yes, I have a jail with the virtual box running windows.

 

[HimuraMOdo]

That looks like a corrupted zpool.

You are using AMD, non-ECC RAM, may have virtualized FreeNAS andmay not have given FreeNAS the minimum 8GB of RAM we call for in the manual. So don't take offense, but I'm done with this thread.

You've ignored at least 2 of our very strong warnings about how to not losing data, and potentially as many as 4. If you didn't take our advice before, I'm not sure why you'd be asking us for more advice now...

None taken. Thanks for the reply anyway.
I've given Freenas 12GB of ram so I think I'm not violating that requirement. As for the AMD CPU and non-ECC ram, well, like I said it's an old machine and I don't plan on spending a lot of money on it to get the server grade hardware.

And it turns out it's a corrupted USB stick issue.

 

[HimuraMOdo]

Assuming your answer to DrKK is "the latter"...

Sometimes a bad or corrupted USB stick causes this kind of problem, so you don't have much to lose by trying a clean install of FreeNAS to a new USB stick and uploading your saved config. Maybe do this after running a few passes of memtest.

You are right.

I'm surprised that the ram is causing the problem since it was bought only a couple months ago brand new. But to be safe, I would run a few memtest just in case.

Thank you for your advice!

 

[Robert Trevellyan]

As for the AMD CPU and non-ECC ram, well, like I said it's an old machine and I don't plan on spending a lot of money on it to get the server grade hardware.

@cyberjock jumped to some incorrect conclusions, but that doesn't mean your choice of hardware is good for FreeNAS. Your data, your choice, no sympathy, is the usual refrain in this forum. By the way, "a ton of money" is not required to get server grade hardware. Dell's PowerEdge T20 and T110ii and Lenovo's TS140 and TS440 are all relatively inexpensive options.

And it turns out it's a corrupted USB stick issue.

Have you actually confirmed this? Is it by any chance a no-name or Kingston stick? What brand is your RAM?

 

[HimuraMOdo]

@cyberjock jumped to some incorrect conclusions, but that doesn't mean your choice of hardware is good for FreeNAS. Your data, your choice, no sympathy, is the usual refrain in this forum. By the way, "a ton of money" is not required to get server grade hardware. Dell's PowerEdge T20 and T110ii and Lenovo's TS140 and TS440 are all relatively inexpensive options.

Have you actually confirmed this? Is it by any chance a no-name or Kingston stick? What brand is your RAM?

I didn't really confirm this. I simply reinstall the OS and import the config file with another USB stick. then everything is back to normal. So I just assume it's the USB stick issue.
The stick is Sandisk 32GB, and the Ram is Crucial Ballistix 8GB X2.

And thanks for the server hardware info. I didn't realize they come in ~$450. I was thinking about more than a grand.

 

[Robert Trevellyan]

I simply reinstall the OS and import the config file with another USB stick. then everything is back to normal. So I just assume it's the USB stick issue.

Good enough.

And thanks for the server hardware info. I didn't realize they come in ~$450.

TS140 starts at $239 on Amazon, and the T20 starts at $299 direct from Dell. Of course, you do have to add more RAM and some hard drives, but still very affordable. I paid $275 for my used T110 and it came with 16GB of RAM and 2x 2TB hard drives. I sold the hard drives because they had about 3 years on the clock, then put those $$ and a few more back into it, but again, very affordable. The grand plus systems that people here build are aimed at more storage or higher performance, or both.