ZFS vol grow indefinitely! (or 1.01TB)

[Tekkie]

All

I have a weird situation which I believe is down to a bug in the system, but I wanted to check here before submitting it.

I was under the impression that when I create a 'logical' ZFS vol on my RAIDZ2 array of a given size lets say 512GB that this size is not only the initial size of the ZFS vol but also its final size.

Over the past weeks however I've noticed a steady decline in free space on my RAIDZ2 and I couldn't explain until I ran 'zfs status' and it reported all my ZFS vols being between 1.5MB and 1.01TB in size rather than the 512GB I created them initialy.

This implies to me that FreeNAS does sparse initialization of ZFS vols which wouldn't be a problem if it were clearly documented!

Now I have two options...
1) I copy down all the data from these vols to somewhere else, destroy them and then re-create new vols and copy back the data letting it grow the vols.
2) I find a way of freezing the vols that I have now and prevent them from growing.

PS. What will happen when my RAIDZ2 array runs our of space because of these extents on the ZFS vols?

 

[Tekkie]

More details on the problem...

Code:

NAME                   USED  AVAIL  REFER  MOUNTPOINT
backup1                242G  49.2G   242G  /mnt/backup1
storage1              7.10T  31.1G  15.5G  /mnt/storage1
storage1/backup        172G  31.1G   172G  /mnt/storage1/backup
storage1/home         8.05G  31.1G  8.05G  /mnt/storage1/home
storage1/movies_vol0  1.01T  31.1G  1.01T  -
storage1/movies_vol1   766G  31.1G   766G  -
storage1/movies_vol2   512G   543G  1.05M  -
storage1/movies_vol3   512G   219G   324G  -
storage1/movies_vol4  1.01T  31.1G  1.01T  -
storage1/music_vol0    495G  31.1G   495G  -
storage1/music_vol1     64G  34.7G  60.3G  -
storage1/pictures     35.7G  31.1G  35.7G  /mnt/storage1/pictures
storage1/series_vol0   512G   543G  1.05M  -
storage1/series_vol1  1.01T  31.1G  1.01T  -
storage1/series_vol2   512G   287G   256G  -
storage1/series_vol3   512G   381G   163G  -
storage1/software     46.8G  31.1G  46.8G  /mnt/storage1/software

The above is the output of zfs list note the size of those volumes it like they were auto grown by FreeNAS (ZFS) to be 1.01TB rather than to 'stay' at 512GB as the config screen show below shows:


How to recover from this situation?

 

[ProtoSD]

kinda makes me wonder if this is related to SilverJS's scrubs appearing to run indefinitely, here:
http://forums.freenas.org/showthread.php?1463-Scrubs-don-t-stop...&highlight=scrub

That might explain his problem, but I'm not sure his volumns are similar to yours.... Are you using snapshots?

 

[Tekkie]

I did but only on one particular folder in that list:

Code:

storage1/pictures     35.7G  31.1G  35.7G  /mnt/storage1/pictures

And that one is outside of the ZFS vols that keep growing at infinitum. :( I just added 10GB of data yesterday onto one of the iSCSI mapped vols and lost another 10GB from /mnt/storage1/

What is the command issued in FreeNAS8 to create the 'logical' volumes?

 

[Tekkie]

Windows 'sees' FreeNAS UI size of 512GB!

Code:

NAME                   USED  AVAIL  REFER  MOUNTPOINT
storage1/movies_vol0  1.01T  31.1G  1.01T  -
storage1/movies_vol1   766G  31.1G   766G  -
storage1/movies_vol2   512G   543G  1.05M  -
storage1/movies_vol3   512G   219G   324G  -
storage1/movies_vol4  1.01T  31.1G  1.01T  -
storage1/music_vol0    495G  31.1G   495G  -
storage1/music_vol1     64G  34.7G  60.3G  -
storage1/series_vol0   512G   543G  1.05M  -
storage1/series_vol1  1.01T  31.1G  1.01T  -
storage1/series_vol2   512G   287G   256G  -
storage1/series_vol3   512G   381G   163G  -

All of these ZFS vols are mapped to iSCSI extents and this is were things get even more scary.

Each zfs vol is seen as a unique drive on my Windows 7 HTPC, take storage1/series_vol1 as an example, zfs list reports it as being 1TB in size... guess what Windows 7 'sees'? 512GB! Windows 7 does not see these drives as anything but 512GB every single one of them, although on the FreeNAS system they are clearly not 512GB in size.

 

[Tekkie]

zfs list and zpool list dont agree....

I think I may have found a possible explanation: http://download.oracle.com/docs/cd/E19082-01/817-2271/gcfhk/index.html

Though not recommended, you can create a sparse volume by specifying the -s flag to zfs create -V, or by changing the reservation once the volume has been created. A sparse volume is defined as a volume where the reservation is not equal to the volume size. For a sparse volume, changes to volsize are not reflected in the reservation.

That's not the problem apparently at 'zpool history' tell me:

Code:

2011-07-03.12:52:20 zfs create -V 512GB storage1/movies_vol0
2011-07-03.12:52:30 zfs create -V 512GB storage1/movies_vol2
2011-07-03.12:53:01 zfs create -V 512GB storage1/movies_vol3
2011-07-03.12:53:12 zfs create -V 512GB storage1/movies_vol4
2011-07-03.12:53:49 zfs create -V 512GB storage1/series_vol0
2011-07-03.12:53:57 zfs create -V 512GB storage1/series_vol1
2011-07-03.12:54:14 zfs create -V 512GB storage1/movies_vol1
2011-07-03.12:54:26 zfs create -V 512GB storage1/series_vol2
2011-07-03.12:54:53 zfs create -V 512GB storage1/series_vol3
2011-07-03.13:11:56 zfs create -V 256GB storage1/music_vol0
2011-07-03.13:12:10 zfs create -V 64GB storage1/music_vol1

zpool list:

Code:

NAME       SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
backup1    296G   242G  53.8G    81%  ONLINE  /mnt
storage1  10.9T  7.99T  2.88T    73%  ONLINE  /mnt

It reports /mnt/storage1 as having 2.8TB free, but why does zfs list report this:

Code:

NAME                   USED  AVAIL  REFER  VOLSIZE
storage1              7.10T  28.5G  15.5G        -

'df -m' also reports 28GB free space:

Code:

Filesystem          1M-blocks   Used Avail Capacity  Mounted on
storage1                44705  15889 28816    36%    /mnt/storage1

 

[Tekkie]

More background info on zfs vols

http://dlc.sun.com/osol/docs/content/ZFSADMIN/gazvb.html explains most of the details surrounding zfs vol reservations & quotas, however I still don't get it. :(
zfs get all

Code:

NAME                 PROPERTY              VALUE                  SOURCE
storage1/music_vol0  type                  volume                 -
storage1/music_vol0  creation              Sun Jul  3 13:11 2011  -
storage1/music_vol0  used                  496G                   -
storage1/music_vol0  available             28.1G                  -
storage1/music_vol0  referenced            496G                   -
storage1/music_vol0  compressratio         1.00x                  -
storage1/music_vol0  reservation           none                   default
storage1/music_vol0  volsize               256G                   -
storage1/music_vol0  volblocksize          8K                     -
storage1/music_vol0  checksum              on                     default
storage1/music_vol0  compression           lzjb                   inherited from storage1
storage1/music_vol0  readonly              off                    default
storage1/music_vol0  shareiscsi            off                    default
storage1/music_vol0  copies                1                      default
storage1/music_vol0  refreservation        256G                   local
storage1/music_vol0  primarycache          all                    default
storage1/music_vol0  secondarycache        all                    default
storage1/music_vol0  usedbysnapshots       0                      -
storage1/music_vol0  usedbydataset         496G                   -
storage1/music_vol0  usedbychildren        0                      -
storage1/music_vol0  usedbyrefreservation  0                      -

Its been a long time since I was this confused by a system...

 

[gcooper]

The tricky thing about ZFS is the multiple ways to express things in subtle but similar ways. I'd look at the usedbysnapshots, usedbydatasets, usedbychildren, and usedbyrefreservation fields, as well as the refreservation fields. The zfs manpage is pretty clear about how things play with one another in the usedby categories.