FreeNAS Extremely Slow to Respond + Error: database is locked

[ericv]

First, sorry as I tried getting in on the help for this issue, but there doesn't seem to be a lot of help related to my issue in the Bug section of the forum.

I am running FreeNAS-9.10-STABLE-201604181743 (74ef270). I know there is an update available, waiting for feedback on this before upgrading.

When I login to FreeNAS the UI is very slow. Say 2-3 minutes for the pages to load. My plugins like Plex, btsync seem to be working just fine. When I ssh into the box, it is also really slow.

{"message": "Error: database is locked", "error": true, "events": []}

After I get this message, I can click around and it loads, but it is really slow. My plugins like Plex, btsync seem to be working just fine. When I ssh into the box, it is also really slow.

My /var/log/messages
https://www.dropbox.com/s/33xsu8ytcul9tns/messages.txt?dl=0

[too big to just post the text here]

When I click on the alert light (top right, not flashing or anything) i get this..

https://www.dropbox.com/s/phgm2d7igqd3bw1/error.png?dl=0

CPU usage is not high.

https://www.dropbox.com/s/rpvok6b1zkpdn2k/CPU.png?dl=0

My storage volumes all say HEALTHY.

 

[ericv]

Attempted to upgrade and it didn't work. The upgrade failed.

 

[Glorious1]

Hopefully one of the more experienced members will respond. Did you try rebooting? The messages seem to indicate a problem with drive access, but if the volumes say Healthy, I don't know.

 

[ericv]

Hopefully one of the more experienced members will respond. Did you try rebooting? The messages seem to indicate a problem with drive access, but if the volumes say Healthy, I don't know.

I've tried rebooting. I've also reverted back to FreeNAS-9.3-STABLE-201604150515 and it works, but I cant' update any of my plugs which means I have to re build them all and I am not on a version of FreeNAS getting the latest updates. I'll probably try to upgrade from 9.3 back to 9.10. I am surprised that no one chimed in yet. Thanks for replying.

 

[depasseg]

My guess is your boot devices (or jails or system dataset) are da0 and da1. Is that true? If so, it looks like a communication problem to those 2 devices, which could mean a couple different things - bad cables, bad controller, or overheating/failing devices. What devices are they?

What is the output of camcontrol devlist and zpool list -v (please post in code tags to retain indentations) Also you can copy and paste screenshots directly into the post. :)

Code:

Apr 30 03:04:06 beast (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 71 14 53 00 00 0c 00
Apr 30 03:04:06 beast (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Apr 30 03:04:06 beast (da1:umass-sim1:1:0:0): Retrying command
Apr 30 03:24:42 beast (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 01 44 a3 ef 00 00 33 00
Apr 30 03:24:42 beast (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Apr 30 03:24:42 beast (da1:umass-sim1:1:0:0): Retrying command
Apr 30 03:24:46 beast (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 d1 57 c6 00 00 1f 00
Apr 30 03:24:46 beast (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Apr 30 03:24:46 beast (da0:umass-sim0:0:0:0): Retrying command
Apr 30 03:56:59 beast (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 01 d6 0c 66 00 00 06 00
Apr 30 03:56:59 beast (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Apr 30 03:56:59 beast (da1:umass-sim1:1:0:0): Retrying command
Apr 30 03:57:03 beast (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 71 1b 4f 00 00 04 00
Apr 30 03:57:03 beast (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Apr 30 03:57:03 beast (da0:umass-sim0:0:0:0): Retrying command
Apr 30 04:41:46 beast (da1:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 01 da 42 e0 00 00 01 00
Apr 30 04:41:46 beast (da1:umass-sim1:1:0:0): CAM status: CCB request completed with an error
Apr 30 04:41:46 beast (da1:umass-sim1:1:0:0): Retrying command
Apr 30 04:41:53 beast (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 60 74 09 00 00 04 00
Apr 30 04:41:53 beast (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Apr 30 04:41:53 beast (da0:umass-sim0:0:0:0): Retrying command

 

[ericv]

My guess is your boot devices (or jails or system dataset) are da0 and da1. Is that true? If so, it looks like a communication problem to those 2 devices, which could mean a couple different things - bad cables, bad controller, or overheating/failing devices. What devices are they?

What is the output of camcontrol devlist and zpool list -v (please post in code tags to retain indentations) Also you can copy and paste screenshots directly into the post. :)

I was likely posting an update right when you were posting... I reverted back to FreeNAS-9.3-STABLE-201604150515 and everything is fine other than my plugins need rebuilt as they won't update...

Here is the results...

camcontrol devlist

Code:

<Samsung SSD 840 PRO Series DXM04B0Q>  at scbus0 target 0 lun 0 (pass0,ada0)

<Samsung SSD 840 PRO Series DXM04B0Q>  at scbus1 target 0 lun 0 (pass1,ada1)

<ST4000VN000-1H4168 SC46>          at scbus3 target 0 lun 0 (pass2,ada2)

<ST4000VN000-1H4168 SC44>          at scbus4 target 0 lun 0 (pass3,ada3)

<ST4000VN000-1H4168 SC46>          at scbus5 target 0 lun 0 (pass4,ada4)

<ST4000VN000-1H4168 SC46>          at scbus6 target 0 lun 0 (pass5,ada5)

<ST4000VN000-1H4168 SC46>          at scbus7 target 0 lun 0 (pass6,ada6)

<ST4000VN000-1H4168 SC46>          at scbus8 target 0 lun 0 (pass7,ada7)

<PNY USB 2.0 FD 1100>              at scbus10 target 0 lun 0 (pass8,da0)

<PNY USB 2.0 FD 1100>              at scbus11 target 0 lun 0 (pass9,da1)

zpool list -v

Code:

NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Apps                                     117G  11.3G   106G         -    29%     9%  1.00x  ONLINE  /mnt
  mirror                                 117G  11.3G   106G         -    29%     9%
    gptid/e211e534-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
    gptid/e22afa30-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
Storage                                 21.8T  6.55T  15.2T         -     9%    30%  1.00x  ONLINE  /mnt
  raidz2                                21.8T  6.55T  15.2T         -     9%    30%
    gptid/b3fd6950-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
    gptid/b4466aee-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
    gptid/b4919038-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
    gptid/7bb9fcb6-dd80-11e4-baea-0cc47a49c8de      -      -      -         -      -      -
    gptid/b521019e-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
    gptid/b566e33b-cffe-11e4-87af-0cc47a49c8de      -      -      -         -      -      -
freenas-boot                            29.8G  7.11G  22.6G         -      -    23%  1.00x  ONLINE  -
  mirror                                29.8G  7.11G  22.6G         -      -    23%
    gptid/ee523df1-d02d-11e4-8a7e-0cc47a49c8de      -      -      -         -      -      -
    gptid/ee78f0f8-d02d-11e4-8a7e-0cc47a49c8de      -      -      -         -      -      -

 

[depasseg]

OK, so we know know that your boot devices are the ones causing the issues. Since they are usb, it' possible they are overheating and possibly failing. I would make sure you have a current configuration backup and then try to reinstall the same version of freenas onto a different device (and restore the config) to see if that helps. This is very likely the reason your upgrade failed, and possibly related to the issues of the jails not being upgradable (nor any other system change).

 

[ericv]

OK, so we know know that your boot devices are the ones causing the issues. Since they are usb, it' possible they are overheating and possibly failing.

How can you tell? What told you that? I'm just curious.

 

[depasseg]

The errors from your log are for da0 and da1, which are your PNY USB devices. And your boot device is a mirror, so I assumed that was your boot mirror.

 

[ericv]

@depasseg - Sorry for the lack of response. I had a death in the family.

Ok, I get you. I haven't had any issues since moving back to FreeNAS-9.3-STABLE-201604150515.

I am going to try to upgrade this week to see what happens.

Quick question about my USB sticks....

Could I try getting 2 new ones, replacing 1 with a blank one so the mirror copies over to the new one and then do the same with the last old one?

 

[depasseg]

No need to apologize. I'm sorry for your loss.

Yes, there is a way to do that, but I can't remember if it's using the GUI or not. I recently did this to upgrade to a large boot device.