zpool forgetting pools on reboot?

[zbuc]

Whenever my FreeNAS system reboots (and these have been hard reboots -- e.g. once I was trying to adjust networking settings for a jail via the web UI and it caused a complete system crash) zpool forgets all my pools.

However, database rows still exist in the web UI's sqlite backend for the now-missing pools.

If I try to view the volume list in the web UI, the AJAX request winds up getting a 500 error because "zpool list" and "zpool -h Volume" can't find the volumes that are in the database.

If I select "import volume" on the web UI, it finds my volumes and prompts me to import them, however when I try importing them it says "A volume with this name already exists", presumably because of the leftover database rows. I assume if I delete them it will allow me to import, but I haven't tried.

I'm running 9.3 beta.

I have three questions:

1) How do I make it so my zpools aren't forgotten on system reboot?
2) After the pools have been forgotten, what is the correct way to get the system to find them again?
3) How do I upgrade from 9.3 beta to 9.3 stable?

Thank you!

 

[pjc]

1) Your system shouldn't be rebooting. There's something very very wrong. What hardware are you running?

2a) Can you import your pools from the command-line? It sounds like not. What commands are you trying, and what results are you getting?

2b) What version of 9.3-BETA are you running? I don't think you should be getting the duplicate name message any more. I think it's safe to delete the pools from the GUI and re-import them, though that may cause some issues with any shares you have configured to use them. But if your pools won't mount via command-line, the GUI won't be able to import them either.

3) The latest beta update should automatically update you to stable.

 

[zbuc]

1)
CPU: 2X Xeon E5620 2.4Ghz Quad Core.
RAM: 16GB (2X 8GB DDR3 ECC).
HDD: 4x WD Red 3TB
Chassis: SC826TQ-R800LPB
PSU: Supermicro 800W Redundant PS.
MB: X8DTN+
RAID: LSI 9240-8i Flashed to 9211-8i IT Mode.

Is it expected that pools are forgotten on system reboot? What if I wanted to do something that required system shutdown, e.g. installing 16 GB more RAM? Is it expected that I would lose my pools after that?

2a) I don't have access to the machine right now, but I'm new to zfs in general. What should I try to import them from the command line?
2b) I can't delete them from the GUI because the AJAX call to get the list of pools results in a 500 error. I'm not sure of the version, I'm at work right now -- but I downloaded it 4 days ago. How do I check it?

Thank you

 

[pjc]

1) Hardware looks fine at first impression, but a reboot means a major system failure. Did you get your MB secondhand (it's a few generations old)?

Pools aren't forgotten on reboot, but if you're getting pool corruption due to catastrophic system failure, they won't be importable on reboot.

2a) "zpool import" (no arguments) will tell you every pool that ZFS sees and what state it's in (whether it can be imported or not). Let us know that it says.

2b) I haven't seen a 500 error before in that table, even when it couldn't import my pool(s). What does it say with the relevant timestamp in /var/log/debug.log?

Version is in the System Information tab. It'll be something like "FreeNAS-9.3-BETA-201412011304".

 

[zbuc]

It is a second hand motherboard.

zpool import with no arguments gets nothing, but if I specify the cache file I see more and maybe get a hint:

[root@nasty] ~# zpool import -c /boot/zfs/zpool.cache
pool: Main
id: 2441345264389373979
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://illumos.org/msg/ZFS-8000-3C
config:

Main UNAVAIL insufficient replicas
mirror-0 UNAVAIL insufficient replicas
10435691659748225642 UNAVAIL cannot open
4008464825575558348 UNAVAIL cannot open
mirror-1 UNAVAIL insufficient replicas
15097806857163525390 UNAVAIL cannot open
17112385111169799926 UNAVAIL cannot open
[root@nasty] ~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 3.97G 1.83G 2.14G - - 46% 1.00x ONLINE -
[root@nasty] ~# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 9 21:01:57 2014
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors
[root@nasty] ~# df
Filesystem 1K-blocks Used Avail Capacity Mounted on
freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-BETA-201412090235 3070105 954385 2115719 31% /
devfs 1 1 0 100% /dev
tmpfs 32768 5432 27336 17% /etc
tmpfs 4096 8 4088 0% /mnt
tmpfs 3835948 208608 3627340 5% /var
freenas-boot/grub 2123674 7954 2115719 0% /boot/grub
[root@nasty] ~# zfs get mountpoint,canmount
NAME PROPERTY VALUE SOURCE
freenas-boot mountpoint none local
freenas-boot canmount off local
freenas-boot/ROOT mountpoint none inherited from freenas-boot
freenas-boot/ROOT canmount off local
freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-BETA-201412090235 mountpoint / local
freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-BETA-201412090235 canmount noauto local
freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-BETA-201412090235@2014-12-09-21:00:36 mountpoint - -
freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-BETA-201412090235@2014-12-09-21:00:36 canmount - -
freenas-boot/ROOT/default mountpoint legacy local
freenas-boot/ROOT/default canmount noauto local
freenas-boot/grub mountpoint legacy local
freenas-boot/grub canmount on default
freenas-boot/grub@Pre-Upgrade-FreeNAS-FreeNAS-9.3-BETA-201412090235 mountpoint - -
freenas-boot/grub@Pre-Upgrade-FreeNAS-FreeNAS-9.3-BETA-201412090235 canmount - -


The weird thing is that I do see my disk drives in /dev/...

[root@nasty] ~# ls /dev/da*
/dev/da0 /dev/da0p1 /dev/da0p2 /dev/da1 /dev/da1p1 /dev/da1p2 /dev/da2 /dev/da2p1 /dev/da2p2 /dev/da3 /dev/da3p1 /dev/da3p2 /dev/da4 /dev/da4p1 /dev/da4p2

 

[SweetAndLow]

Use code tags please so people can actually read your posts with output correctly

 

[zbuc]

Also, I updated to stable, build is FreeNAS-9.3-STABLE-201412091831

And now everything seems to be working O_O

 

[pjc]

You *might* have been encountering https://bugs.freenas.org/issues/6848 and https://bugs.freenas.org/issues/6868. Your beta version predates the fix. If everything is magically working now, maybe that was it.

But I never encountered "UNAVAIL" messages.

Do you happen to have the debug.log entries from when you got that 500 error?