Freenas crashes when restoring from Veeam

[Billy Sweet]

Hello,

I did a search, but was unable to find anything similar.

Build FreeNAS-9.2.2-ALPHA-8e59888-x64
Platform AMD FX(tm)-4130 Quad-Core Processor
Memory 24452MB

I am using ZFS in a 3+1 configuration. I have a CIFS share setup, with AD authentication. Veeam is able to successfully backup to my NAS, though it did completely freeze once. After rebooting, the job was able to finish.

When I go to do a recovery at the file level through Veeam, FreeNAS crashes every single time. When this happens, I cannot ping the NAS. The shell is still accessible from the device locally, and says something about ada0 being detached. I tried to do a "zpool online" for ada 0/1/2/3 and it says they do not exist. Rebooting fixes everything, but the problem happens again as soon as I try to do a file level recovery.

I was on the latest stable, but figured I'd try out the 4/30 nightly to see if there was a difference. Both have the same issue.

Any ideas?

The backup jobs we are working with are in excess of 1TB.

Thank you in advance. If any log information is needed, I may need some assistance in retrieving that.

 

[cyberjock]

If ada0 is being detached, that sounds like some kind of hardware problem. Presumably ada0 is failing.

 

[Billy Sweet]

If ada0 is being detached, that sounds like some kind of hardware problem. Presumably ada0 is failing.

First and foremost, thank you for taking the time to reply to my thread. :)

I don't think ada0 is failing. I can do anything under the sun, without any issues. I've moved terabytes back and forth through Windows Explorer via CIFS without issue, and the NAS has stayed up for weeks at a time with no errors. All of my disks are healthy, and the issues only happen when I use Veeam. I can even do a hot pull (I know, not the smartest thing, but just to see) of ada0, and still be good from my +1 redundancy.

I've ran the same Veeam backup/restore jobs on my HP SAN with no issues, so I am left to believe that the fault lies within FreeNAS.

 

[cyberjock]

Can you post the SMART data for ada0? I've seen lots of detaches.. and it's *always* because of a hardware problem.

 

[jgreco]

Detaches would indicate a hardware problem. Nothing the NAS does should cause a detach. There's a sticky in the hardware forum about things to do while burning in your NAS, which you might want to revisit after replacing the faulty controller or port...

 

[Billy Sweet]

I'm going to run the short and long tests. I'll post information for both when they are complete. Thanks.

 

[jgreco]

The long and short tests may pass just fine. Disconnects are a communications issue.

 

[cyberjock]

I'm with jgreco. They'll likely pass just fine as disconnects are a communication issue.

 

[jgreco]

But if it did happen to fail then suspect power quality issues since a bad PSU or cabling would be a likely cause.

 

[Billy Sweet]

You were both correct, it passed the short and long test, "Completed without error." What should I do next?

 

[cyberjock]

Well, start identifying what your hardware problem is.

I'd try replacing the SATA cable first since that's easy and cheap.

other than that, try a spare PSU in the machine and then try a different SATA controller.

 

[Billy Sweet]

I changed the SATA a few days ago, so I know that isn't the issue, but I do believe power is at fault, and my poor configuration is to blame.

I had all four of my drives connected to a single power line that has four SATA power connectors. Considering the size of these drives, I assume that I was pulling too much power through a single line, and it would cause a spike, which would cause all of the drives to crap out. This would explain why the local console was still available after the drives went down.

Does this sound about right? I'm putting it through its paces now, to see if it fails again.

Edit: I forgot to say that I now have the power distributed to two drives per line.

 

[jgreco]

It wouldn't come as a big shock if that was the problem.

There's a nice sticky on hardware burn-in in the hardware forums. You are encouraged to follow all the bits of that that do not involve writing to disks. No NAS box should be deployed before you have full faith in the hardware's stability...

 

[Billy Sweet]

You're absolutely right, and I am going to take a look at that now. I haven't experienced a failure yet, but there are still like 4-5 VMs to go, and if they are successful, I'm going to continue to hammer at 100% through the weekend.

Thank you both for your input, and looking at your post counts, your dedication to the forums. I'll let you know what happens.

 

[jgreco]

We like people willing to take the advice freely offered. Shockingly simple eh :)

 

[Billy Sweet]

So, I've had my FreeNAS box under an almost constant heavy load, and it has been stable since I switched up the SATA power cables. Thanks again.

 

[jgreco]

Great to hear! Enjoy!