ZFS Encrypted Drive Relocks

[Cosmo_Kramer]

Hello,
I am on FreeNAS 9.2.0 and I have one WD Green drive that is encrypted. Occasionally, that ZFS pool will randomly relock itself without the NAS turning off.

Any ideas why that could be?

I have my logs sent to Splunk and I searched for errors, but I didn't catch anything. Is there a specific keyword I should search for?

Thank you for your time.

 

[cyberjock]

Can you post your system specs?

Are you sure the system isn't just crashing and rebooting and you aren't aware of the reboot?

 

[Cosmo_Kramer]

Can you post your system specs?

Are you sure the system isn't just crashing and rebooting and you aren't aware of the reboot?

I believe it is crashing. I've witnessed it do so when I stopped a jail and it hung. Afterwards, the system was still up but the drive was locked. I tried unlocking it but every time I tried to restart the jail, the process repeated. I rebooted it and everything worked as expected afterwards. Strangely enough, there weren't any logs in the Messages footer.

I believe it started doing this after I reverted back to 9.2.0 from 9.2.1.x but it may be a coincidence.

My System Specs are as follows:
Motherboard: ECS Elitegroup H61H2-I3
CPU: Intel Core i3-3220 Dual-Core Processor 3.3 Ghz (hyper-threaded)
RAM: 12 GB of Kingston Hyper X Blu 1600 MHz (1x8 GB and 1x4 GB)
PSU: Corsair Builder Series CX 430
Drives: 2x2 TB WD Reds in a ZFS Mirror
1x1 TB WD Green (Encrypted)

I also have 3 Jails if that matters.

Thank you for your help.

 

[cyberjock]

Not to be a hardass, but ECS is the last brand you want to be in a server. The joke I've always heard is ECS stands for Every Customer is a Sucker. I wouldn't be the least bit surprised if your hardware just flat out sucked. Good CPU for performance. Overall though, ECC RAM is recommended.

I'd try a RAM test first as that's the easiest to do. Other than that, I have no suggestions except to get a better quality motherboard and preferably a setup that supports ECC RAM.

 

[Cosmo_Kramer]

Not to be a hardass, but ECS is the last brand you want to be in a server. The joke I've always heard is ECS stands for Every Customer is a Sucker. I wouldn't be the least bit surprised if your hardware just flat out sucked. Good CPU for performance. Overall though, ECC RAM is recommended.

I'd try a RAM test first as that's the easiest to do. Other than that, I have no suggestions except to get a better quality motherboard and preferably a setup that supports ECC RAM.

Okay, thanks for the heads up. I'll let you know what the RAM test results in.

 

[Cosmo_Kramer]

Okay, thanks for the heads up. I'll let you know what the RAM test results in.

I ran a memory test with memtest86 and it didn't find any errors. I completely shutdown the NAS (not reboot) and started it back up, and it now seems to be running stable.
I'm looking into buying new gear for ECC support as you suggested, but if it does crash again with new hardware, do you have any suggestions?

 

[cyberjock]

Not unless you have an error message or something to go with.

 

[Cosmo_Kramer]

Not unless you have an error message or something to go with.

So I found this in /data/crash:

Code:

[root@freenas /data/crash]# zmore info.0                                       
Dump header from device /dev/dumpdev                                           
  Architecture: amd64                                                         
  Architecture Version: 1                                                     
  Dump Length: 121856B (0 MB)                                                 
  Blocksize: 512                                                               
  Dumptime: Thu Mar 13 09:01:46 2014                                           
  Hostname: freenas.local                                                     
  Magic: FreeBSD Text Dump                                                     
  Version String: FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
    root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/
home/jkh/checkout/freenas/FreeBSD/src/sys/FREENASUnrecoverable machine check exc
eption  Panic String: Unrecoverable machine check exception                   
  Dump Parity: 2579047542                                                     
  Bounds: 0                                                                   
  Dump Status: good      

This seems to be a hardware error if I am not mistaken. My new motherboard and ECC RAM is coming on Tuesday, I'm hoping that will fix it.

 

[Cosmo_Kramer]

So I found this in /data/crash:

Code:

[root@freenas /data/crash]# zmore info.0                                     
Dump header from device /dev/dumpdev                                         
  Architecture: amd64                                                       
  Architecture Version: 1                                                   
  Dump Length: 121856B (0 MB)                                               
  Blocksize: 512                                                             
  Dumptime: Thu Mar 13 09:01:46 2014                                         
  Hostname: freenas.local                                                   
  Magic: FreeBSD Text Dump                                                   
  Version String: FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
    root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/
home/jkh/checkout/freenas/FreeBSD/src/sys/FREENASUnrecoverable machine check exc
eption  Panic String: Unrecoverable machine check exception                 
  Dump Parity: 2579047542                                                   
  Bounds: 0                                                                 
  Dump Status: good      

This seems to be a hardware error if I am not mistaken. My new motherboard and ECC RAM is coming on Tuesday, I'm hoping that will fix it.

So I ended up setting up the new motherboard and the new RAM. It turns out I had accidentally bought registered RAM, which my motherboard didn't support. So for a few days, while waiting for the correct RAM to come, the new motherboard was running with the non-ECC RAM just fine. So apparently it was the motherboard that caused the crashes. I still switched out the old RAM for ECC, but this should help people with this error in the future.