FreeNAS is kinda responsive, but not working right - what to do?

[mikesoultanian]

I was doing some testing and pulled out three drives on a RAID10 volume to simulate a failure. Sure enough, I got a failure. I reinserted the drives to see what would happen, refreshed, and the volume remained unavailable. Ok, whatever, there was nothing on that volume anyways so I proceeded to delete the volume, recreate it, and vow never to do that again. I tried deleting the volume and now it's just hanging on the "detach volume" message for the past 15 minutes. If I open another browser and look at the log, these are the relevant lines:

Code:

Aug 17 00:06:43 FMTSAN1 notifier: geli: No such device: /dev/da0p1.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Device da0p1.eli destroyed.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Detached da0p1.eli on last close.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Device da1p1.eli destroyed.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Detached da1p1.eli on last close.
Aug 17 00:06:43 FMTSAN1 notifier: geli: No such device: /dev/da1p1.
Aug 17 00:06:43 FMTSAN1 notifier: geli: No such device: /dev/da2p1.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Device da2p1.eli destroyed.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Detached da2p1.eli on last close.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Device da3p1.eli destroyed.
Aug 17 00:06:43 FMTSAN1 GEOM_ELI: Detached da3p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da3p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da28p1.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da28p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da28p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da29p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da29p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da29p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da30p1.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da30p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da30p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da31p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da31p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da31p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da55p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da56p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da57p1.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da55p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da55p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da56p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da56p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da57p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da57p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da58p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da59p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da60p1.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da58p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da58p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da59p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da59p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da60p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da60p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Device da61p1.eli destroyed.
Aug 17 00:06:44 FMTSAN1 GEOM_ELI: Detached da61p1.eli on last close.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da61p1.
Aug 17 00:06:44 FMTSAN1 notifier: geli: No such device: /dev/da64p1.
Aug 17 00:06:45 FMTSAN1 GEOM_ELI: Device da64p1.eli destroyed.
Aug 17 00:06:45 FMTSAN1 GEOM_ELI: Detached da64p1.eli on last close.
Aug 17 00:06:45 FMTSAN1 notifier: geli: No such device: /dev/da65p1.
Aug 17 00:06:45 FMTSAN1 notifier: Stopping syslog_ng.
Aug 17 00:06:45 FMTSAN1 syslog-ng[49789]: syslog-ng shutting down; version='3.6.4'
Aug 17 00:06:45 FMTSAN1 GEOM_ELI: Device da65p1.eli destroyed.
Aug 17 00:06:45 FMTSAN1 GEOM_ELI: Detached da65p1.eli on last close.

That "syslog-ng shutting down" seems ominous - any ideas what is going on? I tried changing a setting in system | general (change the logging level to something different) and it said "Error: database is locked". So now it seems like I managed to lock my database somehow... I'm not even sure what that means.

Any ideas on how to proceed? I really would rather not reboot as I have a few production VMs running on my other volume, but I can inform the users if needed. Should I just wait? I guess I'll go to bed and see if the message is gone when I wake up...

Thanks!
Mike

 

[dlavigne]

Were you able to fix this?

 

[mikesoultanian]

Were you able to fix this?

I haven't tried it again because it's on our production system. It's a bit disconcerting though, so I was hoping someone might have an idea what's going on.

Thanks!