ZVol (not zpool) missing from GUI and not mount point

[GEOptic]

Hello Ladies, Gentlemen,

I use to work with NatApp/IBM storage appliances, so I understand the matter a bit.

I'm not new to FreeNAS, but it has been a while since I used it. (9.3).
Installed it a few months ago (april), and it serves us well.

Now, one of our subpool zvol isn't listed anymore in the GUI, and the mount point disapeared as well.
Seems to appended after a VP powered off every server in the room trying to get back a VM …

Here, I have ixSystems hardware :

Code:

Build		 FreeNAS-11.1-U5	
	
		Platform		 Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz	
	
		Memory		 65387MB

ZPool:

root@stk:/ # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Vol1 21.8T 10.3T 11.5T - 17% 47% 1.00x ONLINE /mnt
freenas-boot 29.8G 1.68G 28.1G - - 5% 1.00x ONLINE -


ZFS list

root@stk:/ # zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
Vol1 2.31T 11.7T 0 192K 0 11.7T
Vol1/.system 2.31T 533M 0 510M 0 23.9M
Vol1/.system/configs-b3147af497d748f98f845f65b942be53 2.31T 22.1M 0 22.1M 0 0
Vol1/.system/cores 2.31T 767K 0 767K 0 0
Vol1/.system/rrd-b3147af497d748f98f845f65b942be53 2.31T 176K 0 176K 0 0
Vol1/.system/samba4 2.31T 711K 0 711K 0 0
Vol1/.system/syslog-b3147af497d748f98f845f65b942be53 2.31T 176K 0 176K 0 0
Vol1/ge1-home 5.33T 5.59T 0 2.57T 3.02T 0
Vol1/jails 2.31T 176K 0 176K 0 0
Vol1/nfs_backup 430G 2.58T 0 2.58T 0 0
Vol1/smb_test 10.0G 176K 0 176K 0 0
Vol1/zVol_iSCSI 4.16T 3.55T 136G 1.57T 1.85T 0
freenas-boot 27.1G 1.67G 0 64K 0 1.67G
freenas-boot/ROOT 27.1G 1.66G 0 29K 0 1.66G
freenas-boot/ROOT/11.1-U5 27.1G 1.66G 840M 861M 0 0
freenas-boot/ROOT/Initial-Install 27.1G 1K 0 1K 0 0
freenas-boot/ROOT/default 27.1G 2.24M 0 2.24M 0 0
freenas-boot/grub 27.1G 6.84M 0 6.84M 0 0


Ls mount point: (Oops! ge1-home not listed)

root@stk:/ # ll /mnt/Vol1/
total 2
-rw-rw-rw- 1 root wheel 0 Jun 14 16:56 .windows
drwxr-xr-x 2 root wheel 2 Jul 9 10:27 jails/
drwxr-xr-x 4 root wheel 4 Oct 4 08:30 nfs_backup/
drwxr-xr-x 2 root wheel 3 Jun 13 16:01 smb_test/


Some errors / warnings (probably due to the above situation)

Code:

	Snapshot Vol1/ge1-home@auto-20181031.0600-3w failed with the following error: cannot create snapshot 'Vol1/ge1-home@auto-20181031.0600-3w': out of space

But as seen above, about 50% space available on Vol1/ge1-home!

Code:

stk kernel log messages:

> WARNING: 1.1.2.1 (iqn.good-value-here): no ping reply (NOP-Out) after 5 seconds; dropping connection

But ping OK (ip is not the real one)

root@stk:/ # ping 1.1.2.1
PING 1.1.2.1 (1.1.2.1): 56 data bytes
64 bytes from 1.1.2.1: icmp_seq=0 ttl=64 time=0.175 ms
64 bytes from 1.1.2.1: icmp_seq=1 ttl=64 time=0.271 ms



Vol1/ge1-home is an iSCSI share extent. And data are there from the initiator point of vue … from the users as well! ;)

I'm a bit puzzled. And stressed… I'd rather not reboot …

Any ideas!? Déjà-vu?


Thanks!

 

[kdragon75]

If ge1-home is a zvol (block device) it will not show up in the file system as a mounted volume. If you look is /dev/zfs/zvol/ (I may have the path wrong) you will see the block device there.

 

[GEOptic]

Tks kdragon75 ! :)

You are right … ge1-home is listed in /dev/zvol/Vol1 !!!


root@stk:~ # ll /dev/zvol/Vol1
total 0
crw-r----- 1 root operator 0x69 Sep 28 16:36 ge1-home
crw-r----- 1 root operator 0x134 Sep 11 12:30 zVol_iSCSI
crw-r----- 1 root operator 0x65 Oct 20 12:00 zVol_iSCSI@auto-20181020.1200-2w
… some more snapshots ...


You maid me realized that iSCSI are not in the left pane ...
Any idea why Snapshot job says no space available?

 

[kdragon75]

NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
Vol1/ge1-home 5.33T 5.59T 0 2.57T 3.02T 0

I suspect its the USEDREFRESERV

 

[GEOptic]

I suspect its the USEDREFRESERV

Oh … I did not pay attention to this … but I should have!
I never set reservation … and reservation is 0 everywhere on the pool/children…

Well I guess I'll learn more about USEDREFRESERV : https://forums.freenas.org/index.php?search/51900032/&q=USEDREFRESERV

 

[GEOptic]

zfs get and zfs set helped me solve my SnapShot out of space issue!!

I changed refreservation to 3.1T just above the previous USEDBYREFRESERVATION 3.0T value.
I assumed that if it is in fact USED, I cannot / shouldn't set it below that value.
Then, the 3.0T changed to 522G … So I have further readings ahead!!! ;)


root@stk:~ # zfs get reservation,refreserv,usedrefreserv Vol1/ge1-home
NAME PROPERTY VALUE SOURCE
Vol1/ge1-home reservation none default
Vol1/ge1-home refreservation 5.59T local
Vol1/ge1-home usedbyrefreservation 3.00T -

root@stk:~ # zfs set refreservation=3.1T Vol1/ge1-home

root@stk:~ # zfs get reservation,refreserv,usedrefreserv Vol1/ge1-home
NAME PROPERTY VALUE SOURCE
Vol1/ge1-home reservation none default
Vol1/ge1-home refreservation 3.10T local
Vol1/ge1-home usedbyrefreservation 522G -

root@stk:~ # zfs list -o space Vol1/ge1-home
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
Vol1/ge1-home 5.14T 3.10T 0 2.59T 522G 0