zpool shows tiny size after full

[herklots]

I ran out of storage space (copying file to server through CIFS) on an encrypted RAIDZ1 pool comprising 5x 2TB drives under FreeNAS 9.2.1.4-RELEASE 64 bit with 16GB RAM.

The usual fixes cp /dev/null <filename> and cat /dev/null > <filename> gave out of space errors.

Rebooted, mounted, and the pool showed odd size under Storage tab in GUI:
Name Used Available Size Status
Five2T 4.0 KiB (8%) 754.0 KiB 823.5 KiB HEALTHY

ls -laR showed . and .. only

Did cp /dev/null <filename> and rm /dev/null <filename>, no errors.

Rebooted, mounted, and the pool showed odd size still.

Unmounted pool, AutoImported it again with key and password. No change.

zpool list shows this:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
Five2T 9.06T 8.92T 145G 98% 1.00x ONLINE /mnt

Ran zpool status which took the expected time:
pool: Five2T
state: ONLINE
scan: scrub repaired 0 in 10h34m with 0 errors on Tue Apr 22 06:22:26 2014
config:
NAME STATE READ WRITE CKSUM
Five2T ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/d4788eb2-9453-11e3-9bf1-d43d7e5555e1.eli ONLINE 0 0 0
gptid/f833fe71-9537-11e3-a0ed-d43d7e5555e1.eli ONLINE 0 0 0
gptid/dc7f5a80-9453-11e3-9bf1-d43d7e5555e1.eli ONLINE 0 0 0
gptid/dfc3141f-9453-11e3-9bf1-d43d7e5555e1.eli ONLINE 0 0 0
gptid/e3182cf1-9453-11e3-9bf1-d43d7e5555e1.eli ONLINE 0 0 0
errors: No known data errors

Help!

 

[warri]

What exactly is your problem and question?
That the Web-GUI is displaying wrong values, while zpool list displays the correct values - and how to fix that?

The trick to delete files when your pool is full is to use echo, which has been described in some other threads before. You cp of /dev/null may or may not work, I haven't tried it myself.

 

[herklots]

What exactly is your problem and question?
That the Web-GUI is displaying wrong values, while zpool list displays the correct values - and how to fix that?

The problem is the pool appears under Storage, on GUI command line, and externally to be under 1Mbyte, not 8TByte.
How to fix that without losing the pool contents, and understand what's going on.

Thanks for the echo comment.

 

[warri]

zpool list seems to display the correct amount:

Five2T 9.06T 8.92T 145G 98% 1.00x ONLINE /mnt

What is the output of ls -al /mnt/?

 

[herklots]

zpool list seems to display the correct amount:

What is the output of ls -al /mnt/?

I can tell you in about an hour. Any other commands to run while I am there?

 

[warri]

Not quite sure yet. I've seen pools which reported a size of 823.5 KiB, but that was only the case when they couldn't be imported properly (and they usually had the status UNKNOWN instead of HEALTHY). Also I don't have any experience with encrypted pools.

 

[herklots]

Not quite sure yet. I've seen pools which reported a size of 823.5 KiB, but that was only the case when they couldn't be imported properly (and they usually had the status UNKNOWN instead of HEALTHY). Also I don't have any experience with encrypted pools.

[root@freenas ~]# ls -al /mnt/ total 5 drwxr-xr-x 4 root wheel 512 Apr 22 11:25 . drwxr-xr-x 20 root wheel 512 Dec 20 14:37 .. drwxrwxr-x 2 root operator 512 Apr 22 11:22 .snap drwxr-xr-x 2 root wheel 512 Apr 22 11:25 Five2T -rw-r--r-- 1 root wheel 5 Dec 20 14:32 md_size

 

[herklots]

zpool list seems to display the correct amount:


What is the output of ls -al /mnt/?

[root@freenas ~]# ls -al /mnt/
total 5
drwxr-xr-x 4 root wheel 512 Apr 22 11:25 .
drwxr-xr-x 20 root wheel 512 Dec 20 14:37 ..
drwxrwxr-x 2 root operator 512 Apr 22 11:22 .snap
drwxr-xr-x 2 root wheel 512 Apr 22 11:25 Five2T
-rw-r--r-- 1 root wheel 5 Dec 20 14:32 md_size

 

[herklots]

Anybody else got any thoughts or suggestions?

 

[eraser]

Are your files missing or are you just concerned about the Web GUI displaying an incorrect total size?

Just to clarify, can you confirm that after your third (last?) reboot the Web GUI still shows an incorrect total size of "754.0 KiB", but zpool list shows the expected size of "9.06T" ?

Your output of "ls -al /mnt/" appears to show that your "Five2T" pool is mounted. I suppose the next thing to do is look inside that folder to see if your expected files/datasets are there?

ls -la /mnt/Five2T/​

...or to recursively list subdirectories add "R" to your options:

ls -laR /mnt/Five2T/​

 

[herklots]

Are your files missing or are you just concerned about the Web GUI displaying an incorrect total size?

Just to clarify, can you confirm that after your third (last?) reboot the Web GUI still shows an incorrect total size of "754.0 KiB", but zpool list shows the expected size of "9.06T" ?

Your output of "ls -al /mnt/" appears to show that your "Five2T" pool is mounted. I suppose the next thing to do is look inside that folder to see if your expected files/datasets are there?

ls -la /mnt/Five2T/​

...or to recursively list subdirectories add "R" to your options:

ls -laR /mnt/Five2T/​

Yes, the Web GUI still shows an incorrect total size of "754.0 KiB", but zpool list shows the expected size of "9.06T"

ls shows no files in the pool, although they are clearly somewhere. It's as if the system is pointing to the wrong part of the pool.
[root@freenas ~]# ls -alR /mnt/ total 5 drwxr-xr-x 4 root wheel 512 Apr 22 11:25 . drwxr-xr-x 20 root wheel 512 Dec 20 14:37 .. drwxrwxr-x 2 root operator 512 Apr 22 11:22 .snap drwxr-xr-x 2 root wheel 512 Apr 22 11:25 Five2T -rw-r--r-- 1 root wheel 5 Dec 20 14:32 md_size /mnt/.snap: total 2 drwxrwxr-x 2 root operator 512 Apr 22 11:22 . drwxr-xr-x 4 root wheel 512 Apr 22 11:25 .. /mnt/Five2T: total 2 drwxr-xr-x 2 root wheel 512 Apr 22 11:25 . drwxr-xr-x 4 root wheel 512 Apr 22 11:25 ..

 

[herklots]

Better listing:

[root@freenas ~]# ls -alR /mnt/
total 5
drwxr-xr-x 4 root wheel 512 Apr 22 11:25 .
drwxr-xr-x 20 root wheel 512 Dec 20 14:37 ..
drwxrwxr-x 2 root operator 512 Apr 22 11:22 .snap
drwxr-xr-x 2 root wheel 512 Apr 22 11:25 Five2T
-rw-r--r-- 1 root wheel 5 Dec 20 14:32 md_size

/mnt/.snap:
total 2
drwxrwxr-x 2 root operator 512 Apr 22 11:22 .
drwxr-xr-x 4 root wheel 512 Apr 22 11:25 ..

/mnt/Five2T:
total 2
drwxr-xr-x 2 root wheel 512 Apr 22 11:25 .
drwxr-xr-x 4 root wheel 512 Apr 22 11:25 ..

 

[eraser]

Well this is something I haven't seen before, but hopefully someone else on here has and can chime in.

When you mounted your pool using the Web GUI were there any interesting messages that showed up in the output of 'dmesg' or in your '/var/log/messages' file ?

Can you show us the relevant output from running "zpool history"?

 

[herklots]

Well this is something I haven't seen before, but hopefully someone else on here has and can chime in.

When you mounted your pool using the Web GUI were there any interesting messages that showed up in the output of 'dmesg' or in your '/var/log/messages' file ?

Can you show us the relevant output from running "zpool history"?

I mounted another pool One4T (it is fine) to cp /var/log/messages to, so log may show that too.
/var/log/messages attached as messages.txt

zpool history:
2014-02-15.09:39:37 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-02-15.09:39:37 zfs set aclmode=passthrough Five2T
2014-02-15.09:39:42 zfs set aclinherit=passthrough Five2T
2014-02-16.02:22:06 zpool scrub Five2T
2014-02-22.14:38:47 zpool import -f -R /mnt 4163289545347891677
2014-02-22.14:38:48 zfs inherit -r mountpoint Five2T
2014-02-22.14:38:48 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-02-22.14:38:49 zfs set aclmode=passthrough Five2T
2014-02-22.14:38:54 zfs set aclinherit=passthrough Five2T
2014-02-27.18:50:58 zpool import -f -R /mnt 4163289545347891677
2014-02-27.18:51:00 zfs inherit -r mountpoint Five2T
2014-02-27.18:51:00 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-02-27.18:51:00 zfs set aclmode=passthrough Five2T
2014-02-27.18:51:05 zfs set aclinherit=passthrough Five2T
2014-03-05.19:27:01 zpool import -f -R /mnt 4163289545347891677
2014-03-05.19:27:02 zfs inherit -r mountpoint Five2T
2014-03-05.19:27:02 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-03-05.19:27:02 zfs set aclmode=passthrough Five2T
2014-03-05.19:27:08 zfs set aclinherit=passthrough Five2T
2014-03-09.14:14:28 zpool import -f -R /mnt 4163289545347891677
2014-03-09.14:14:29 zfs inherit -r mountpoint Five2T
2014-03-09.14:14:29 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-03-09.14:14:30 zfs set aclmode=passthrough Five2T
2014-03-09.14:14:35 zfs set aclinherit=passthrough Five2T
2014-04-21.07:13:44 zpool import -f -R /mnt 4163289545347891677
2014-04-21.07:13:45 zfs inherit -r mountpoint Five2T
2014-04-21.07:13:45 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-04-21.07:13:46 zfs set aclmode=passthrough Five2T
2014-04-21.07:13:51 zfs set aclinherit=passthrough Five2T
2014-04-21.16:48:31 zpool import -f -R /mnt 4163289545347891677
2014-04-21.16:48:33 zfs inherit -r mountpoint Five2T
2014-04-21.16:48:33 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-04-21.16:48:34 zfs set aclmode=passthrough Five2T
2014-04-21.16:48:39 zfs set aclinherit=passthrough Five2T
2014-04-21.19:34:30 zpool import -f -R /mnt 4163289545347891677
2014-04-21.19:34:32 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-04-21.19:46:30 zpool import -f -R /mnt 4163289545347891677
2014-04-21.19:46:32 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-04-21.19:47:43 zpool scrub Five2T
2014-04-22.07:19:35 zpool export Five2T
2014-04-22.07:23:32 zpool import -f -R /mnt 4163289545347891677
2014-04-22.07:23:34 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-04-22.11:25:06 zpool import -f -R /mnt 4163289545347891677
2014-04-22.11:25:08 zpool set cachefile=/data/zfs/zpool.cache Five2T
2014-04-23.20:18:44 zpool import -f -R /mnt 4163289545347891677
2014-04-23.20:18:47 zpool set cachefile=/data/zfs/zpool.cache Five2T

 

[warri]

I noticed two things, firstly this looks like an error:

Apr 23 20:18:48 freenas notifier: cannot set property for 'Five2T': out of space

Not sure which property freenas wants to set though.

Also, your zpool history is missing zfs inherit -r mountpoint Five2T in the newer output. No idea if that is important.

And yes, it looks like some empty folder is mounted over the original mount point, or the mounting of the pool failed, but the mount point was created successfully?

If nobody else has any idea, I'd probably try mounting try pool manually via CLI - and hopefully you'll get an helpful error message.

 

[herklots]

I noticed two things, firstly this looks like an error:

Not sure which property freenas wants to set though.

Also, your zpool history is missing zfs inherit -r mountpoint Five2T in the newer output. No idea if that is important.

And yes, it looks like some empty folder is mounted over the original mount point, or the mounting of the pool failed, but the mount point was created successfully?

If nobody else has any idea, I'd probably try mounting try pool manually via CLI - and hopefully you'll get an helpful error message.

Makes sense. If so, killer advice for anyone filling a zpool is to do the delete-a-file fix before unmounting the pool or rebooting.

What is the command for mounting an encrypted RAIDZ1 pool?

If the problem is that a zpool needs spare space to mount, is there any way to create space before mounting?

 

[cyberjock]

One.. you should NEVER be filling your pool/datasets beyond 95% full. Some people do it anyway, but if you fill it to 100% you are in serious trouble.. as you are seeing firsthand.

There is no "command" for mounting encrypted pools. The GUI handles that for you.

If your pool truly is 100% full, you've probably lost your data. There are some ideas for how you might be able to get access to the data, but it would be read-only so you'd have to have a place to copy the data so you could erase the disks and reformat them. I don't know the commands as it would take some research to even attempt it, and I'm not even sure it would work.