FreeNAS 11.1-U6 crashing when performing scrub

[fuxxy]

Dell PowerEdge r610
LSI SAS9207-8e in IT mode

Dell PowerVault MD1000 (unified mode)
10x HGST 6TB
4x WD Black 1TB

At the beginning of the year, I was having no issues. I was running 4x 6TB & 4x 1TB drives in the powervault. The 6TB were in bays 0/1/2/3 with the 1TB in 11/12/13/14.

I purchased 6 more 6TB drives. It seems after these drives were added, I starting having issues. I created a new pool with the 6 drives, and added the 4 old drives to the pool. During the resilvering process while adding each drive, I had mpt driver resets constantly, eventually causing the system to reboot unexpectedly. I eventually got all the drives added.

Now when doing the weekly scrub the system reboots during the scrub.

Up until yesterday, I had a different HBA (still used the mpt driver). I replaced the HBA thinking that it was a hardware error. Replacing the HBA made no improvement.

 

[Alecmascot]

Power Supply ?

 

[cods69]

Power Supply ?

+1 on that. After adding more devices, I'd be swapping the PSU out for a much bigger one to test, at least temporarily.

 

[fuxxy]

The r610 has dual redundant power supplies. Server is also running ESXi, with FreeNAS as a VM with the controller in PCI passthrough.

Wouldn't faulty hardware affect all the VMs running? I haven't had any issues with the other VMs running on this host.

The HDDs were added to the PowerVault, not the r610. The PowerVault also has dual redundant power supplies. I can probably test by running the PowerVault on a single PSU, the odds of both PSUs being faulty are slim.

 

[cods69]

The HDDs were added to the PowerVault, not the r610. The PowerVault also has dual redundant power supplies. I can probably test by running the PowerVault on a single PSU, the odds of both PSUs being faulty are slim.

I think that PowerVault handles about 480w constant so in theory if it's working well, it should be enough, but not knowing what else is in there, it's hard to tell.
All I can suggest without knowing that hardware too well is stripping out the new pool and see if it starts behaving again. Maybe leave the new drives in idling while removing the new pool, for the usual elimination.

 

[Hossa]

Hello,

I have the same problem!
Thing is, I always thought, that I was "only" having Unauthorized Reboots.....as I posted here:
Unauthorized Reboot - 11.1-RELEASE

but after more research I realized I am having reboots, because of the SCRUBS and additionally I am having the trouble with the unauthorized reboots.
Short term solution (for a few months now) I deactivated automatic scrubs, to prevent the system from rebooting all the time.

So here is my situation:
I did the Update from 11.1 to 11.1U1 about a week after release....
I have three pools:
'Storage01'
'Storage02'
'Storage03'
Scrubbing of Storage01 and Storage02 finishes without problems.
So I am expecting it has nothing to do with the power supply!
Storage01 and Storage02 are much bigger pools (consisting of up to 6 disks), while Storage03 is only a Mirror with 2 disks!

As soon as I start scrubbing Storage03 I get random reboots, sometimes leading to a failure in the pool; one disk drops out and a resilvering is needed....
I already replaced the "failing" disk with a new one and also connected the disk to a different HBA.
Still the scrub is creating random reboots!

I updated to all Versions in between:
FreeNAS-11.1-U1
FreeNAS-11.1-U2
FreeNAS-11.1-U3
FreeNAS-11.1-U4
FreeNAS-11.1-U5
FreeNAS-11.1-U6

It (system rebooting many times, because of pool scrub) happens with every Version! :-( At least the "unauthorized reboot bug was resolved!)

Now I am really at the end of my ideas....

I really hope someone can help me out!

Cheers
Hossa

 

[Hossa]

@fuxxy:
Any news from your side?


Did you try:
FreeNAS-11.2-RC1 or FreeNAS-11.2-RC2 ?

Cheers
Hossa

 

[Hossa]

Hello,

just to make sure, if someone runs into a similar problem.

In my case it was the HDD controller.....I guess some day in the past FreeNAS changed to way it communicated with it.
It took me weeks to find out....but in the end I replaced the one I used with another one, and suddenly everything worked again! ;-)

Cheers
Hossa