Reccuring crashes after upgrading to 9.2.1.6

[atakacs]

Folks,

I'm having a rather annoying problem with FreeNAS crashing after having updated to 9.2.1.6.

This is a fairly standard setup: Dell R420 server, 32Gb RAM, 4x 2TB WD red disks. It was deployed about a year ago with v 9.0.xxx (don't remember which exactly) and was working flawlessly since then - absolutely zero problem. FreeNAS is booted from an usb stick.

Against my best judgement I have recently applied v 9.2.1.6 and now the machine will regularly crash after 6-7 days of run time. I can't really pinpoint a trigger for the crash - it seems to happen overnight when the system is pretty much idle.

The crash log look like this

Code:

kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=132620288, length=4096)]error = 5
kernel: g_vfs_done():kernel: g_vfs_done():kernel: ufs/FreeNASs1a[WRITE(offset=265314304, length=4096)]error = 5
kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=729321472, length=4096)]error = 5
kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=729337856, length=4096)]error = 5
kernel: ufs/FreeNASs1a[WRITE(offset=265314304, length=4096)]error = 5
kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=729321472, length=4096)]error = 5
kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=729337856, length=4096)]error = 5
kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=132616192, length=4096)]error = 6
kernel: Device FreeNASs1a went missing before all of the data could be written to it: expect data loss.
kernel: g_vfs_done():ufs/FreeNASs1a[WRITE(offset=132616192, length=4096)]error = 6
kernel: Device FreeNASs1a went missing before all of the data could be written to it: expect data loss.

Code:

2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 15 bc a7 00 00 08 00 NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): CAM status: SCSI Status Error NULL NULL NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI status: Check Condition NULL NULL NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present) NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): Error 6, Unretryable error NULL NULL NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 g_vfs_done():ufs/FreeNASs1a[WRITE(offset=729329664, length=4096)]error = 6 NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 15 bc a7 00 00 08 00 NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): CAM status: SCSI Status Error NULL NULL NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI status: Check Condition NULL NULL NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present) NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): Error 6, Unretryable error NULL NULL NULL NULL NULL NULL
2014-07-14 00:06:17 0 2 172.16.36.253 g_vfs_done():ufs/FreeNASs1a[WRITE(offset=729329664, length=4096)]error = 6 NULL NULL NULL

Code:

2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 18 80 a7 00 00 08 00
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 18 80 a7 00 00 08 00
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI status: Check Condition
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): Error 6, Unretryable error
2014-07-19 23:23:59 0 2 172.16.36.253 g_vfs_done():ufs/FreeNASs1a[WRITE(offset=822128640, length=4096)]error = 6
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI status: Check Condition
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
2014-07-19 23:23:59 0 2 172.16.36.253 (da0:umass-sim0:0:0:0): Error 6, Unretryable error
2014-07-19 23:23:59 0 2 172.16.36.253 g_vfs_done():ufs/FreeNASs1a[WRITE(offset=822128640, length=4096)]error = 6 NULL NULL NULL
2014-07-20 00:00:00 0 2 172.16.36.253 vnode_pager_getpages: I/O read error
2014-07-20 00:00:00 0 2 172.16.36.253 vnode_pager_getpages: I/O read error
and reboot
2014-07-21 08:50:03 0 6 172.16.36.253 kernel boot file is /boot/kernel/kernel
2014-07-21 08:50:03 0 2 172.16.36.253 Copyright (c) 1992-2013 The FreeBSD Project.
2014-07-21 08:50:03 0 2 172.16.36.253 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
2014-07-21 08:50:03 0 2 172.16.36.253 The Regents of the University of California. All rights reserved.

I have suspected a defective USB key but replacing it does not seem to help (last crash above was with new key - seems very similar to previous one...). Again this seems to be time based (after approx 6-7 days, not to the minute though) with what was a perfectly working setup so far.

Any idea / suggestion / pointer welcome. And yes I am considering downgrading, if at all possible...

 

[cyberjock]

Your suspect about a defective USB key is well put. It could be the USB chipset on your motherboard isn't compatible with FreeNAS 9.2.1.6 since you already tried a different key.

The fact that da0 is throwing errors seems to suggest that da0 is also failing. So you may have two problems if da0 isn't your USB stick. What USB stick are you using? Is it a name brand like sandisk, corsair, or kingston? If not I'd try one of those first.

As for downgrading as long as you haven't done a zpool upgrade from the command line and as long as you have your config file from before you upgraded you should be able to simply install whatever you had installed and upload the config. Now for the other problem. You are claiming that 9.0.x was installed. There was no 9.0.x, so you definitely need to know what you had installed. If your config file isn't the same version or older than the one you install you are going to have problem.

If you don't have your config file you can always recreate the settings after importing your pool. If you are on AD this could suck.

If you did a zpool upgrade you are seriously screwed. You'd know if you did a zpool upgrade because you'd have to invoke the command yourself from the command line. It cannot be done from the WebGUI or is it automatic in any way.

 

[atakacs]

Thanks for you answer.

I will give a try to a "name brand" USB stick first. Also look into what exact version was installed...
As for the zpool it was definitely not upgraded, which is good news :)

More to come...

 

[Yatti420]

Maybe a flakey SATA cable aswell?

 

[joeschmuck]

Looking at the data you provided it does appear to be a boot drive failure and I'm curious on how you changed your software from 9.1.1 (most likely given the time period and you stated) to 9.2.1.6, did you do a GUI upgrade or a clean install?

As for troubleshooting the problem... Try to a different USB port (you only have two, right?) What USB Flash drive model are you using? I would not recommend anything less than 8GB and a good name brand doesn't hurt. Do not use SD cards in a USB adapter, SD cards can become corrupt.