SOLVED Why can't I snapshot my dataset?

SRSR333 · May 21, 2017

Long story short: the snapshots on one of my datasets refuse to complete, and give me an 'out of space' error, as shown in the e-mail I receive from my system:

Code:

Hello,
	Snapshot tank/media@auto-20170522.0000-1w failed with the following error: cannot create snapshot 'tank/media/general@auto-20170522.0000-1w': out of space
cannot create snapshot 'tank/media/home@auto-20170522.0000-1w': out of space
no snapshots were created

The /tank/media dataset has capacity exactly 2.0 TB (not TiB), and has two child datasets, home and general, each with a 1.0 TB quota. Both datasets are filled to more than half their capacity, with ~ 600 GB of data in each dataset. According to the ZFS File System Documentation:

Snapshots can be created almost instantly, and they initially consume no additional disk space within the pool. However, as data within the active dataset changes, the snapshot consumes disk space by continuing to reference the old data, thus preventing the disk space from being freed.

Why can't I create a snapshot of the /tank/media dataset, if snapshots aren't supposed to use any space at all initially? Manually attempting to create one also returns the same 'out of space' error.

SweetAndLow · May 21, 2017

Output of zfs list and zpool status. Freenas version and hardware specs required

Sent from my Nexus 5X using Tapatalk

Arwen · May 21, 2017

Make that zfs list -t all in code tags. (And the zpool status as well.)

SRSR333 · May 21, 2017

SweetAndLow said:
Output of zfs list and zpool status. Freenas version and hardware specs required

Arwen said:
Make that zfs list -t all in code tags. (And the zpool status as well.)

Ask and ye shall receive:

Code:

[root@freenas] ~# zfs list -t all                
NAME                                             
       USED  AVAIL  REFER  MOUNTPOINT            
freenas-boot                                     
      1.26G  13.0G    64K  none                  
freenas-boot/ROOT                                
      1.25G  13.0G    29K  none                  
freenas-boot/ROOT/9.10.2-U2                      
      7.11M  13.0G   637M  /                     
freenas-boot/ROOT/9.10.2-U2@2017-04-15-22:09:50  
       344K      -   637M  -                     
freenas-boot/ROOT/9.10.2-U3                      
      1.25G  13.0G   637M  /                     
freenas-boot/ROOT/9.10.2-U3@2017-04-14-06:02:35  
      2.97M      -   636M  -                     
freenas-boot/ROOT/9.10.2-U3@2017-04-14-21:08:37  
       407K      -   637M  -                     
freenas-boot/ROOT/9.10.2-U3@2017-04-21-18:10:02  
       658K      -   637M  -                     
freenas-boot/ROOT/Initial-Install                
         1K  13.0G   636M  legacy                
freenas-boot/ROOT/Wizard-2017-04-14_21:08:37     
       122K  13.0G   637M  /                     
freenas-boot/ROOT/default                        
       131K  13.0G   637M  legacy                
freenas-boot/grub                                
      6.56M  13.0G  6.56M  legacy                
tank                                             
      1.57T  1.06T    96K  /mnt/tank             
tank/.system                                     
      20.6M  1.06T    96K  legacy                
tank/.system/configs-fe5739137b3643d3a6327c8c1d9b
1a34  3.18M  1.06T  3.18M  legacy                
tank/.system/cores                               
       828K  1.06T   828K  legacy                
tank/.system/rrd-fe5739137b3643d3a6327c8c1d9b1a34
      11.1M  1.06T  11.1M  legacy                
tank/.system/samba4                              
       384K  1.06T   384K  legacy                
tank/.system/syslog-fe5739137b3643d3a6327c8c1d9b1
a34   5.11M  1.06T  5.11M  legacy                
tank/documents                                   
       279G   559G    88K  /mnt/tank/documents   
tank/documents@auto-20170516.2300-1m             
          0      -    88K  -                     
tank/documents@auto-20170519.1037-3d             
          0      -    88K  -                     
tank/documents@auto-20170519.1437-3d             
          0      -    88K  -                     
tank/documents@auto-20170519.1837-3d             
          0      -    88K  -                     
tank/documents@auto-20170519.2237-3d             
          0      -    88K  -                     
tank/documents@auto-20170520.0237-3d             
          0      -    88K  -                     
tank/documents@auto-20170520.0637-3d             
          0      -    88K  -                     
tank/documents@auto-20170520.1037-3d             
          0      -    88K  -                     
tank/documents@auto-20170520.1437-3d             
          0      -    88K  -                     
tank/documents@auto-20170520.1837-3d             
          0      -    88K  -                     
tank/documents@auto-20170520.2237-3d             
          0      -    88K  -                     
tank/documents@auto-20170521.0237-3d             
          0      -    88K  -                     
tank/documents@auto-20170521.0637-3d             
          0      -    88K  -                     
tank/documents@auto-20170521.1037-3d             
          0      -    88K  -                     
tank/documents@auto-20170521.1437-3d             
          0      -    88K  -                     
tank/documents@auto-20170521.1837-3d             
          0      -    88K  -                     
tank/documents@auto-20170521.2237-3d             
          0      -    88K  -                     
tank/documents@auto-20170522.0237-3d             
          0      -    88K  -                     
tank/documents@auto-20170522.0637-3d             
          0      -    88K  -                     
tank/documents/<REDACTED>                              
      24.0G   255G  24.0G  /mnt/tank/documents/ra
ja                                               
tank/documents/<REDACTED>@auto-20170516.2300-1m        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170519.1037-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170519.1437-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170519.1837-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170519.2237-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170520.0237-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170520.0637-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170520.1037-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170520.1437-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170520.1837-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170520.2237-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170521.0237-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170521.0637-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170521.1037-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170521.1437-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170521.1837-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170521.2237-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170522.0237-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>@auto-20170522.0637-3d        
          0      -  24.0G  -                     
tank/documents/<REDACTED>                           
      77.5M   279G  77.5M  /mnt/tank/documents/sh
aradh                                            
tank/documents/<REDACTED>@auto-20170516.2300-1m     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170519.1037-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170519.1437-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170519.1837-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170519.2237-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170520.0237-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170520.0637-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170520.1037-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170520.1437-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170520.1837-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170520.2237-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170521.0237-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170521.0637-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170521.1037-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170521.1437-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170521.1837-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170521.2237-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170522.0237-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>@auto-20170522.0637-3d     
          0      -  77.5M  -                     
tank/documents/<REDACTED>                           
        88K   279G    88K  /mnt/tank/documents/sh
ivani                                            
tank/documents/<REDACTED>@auto-20170516.2300-1m     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170519.1037-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170519.1437-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170519.1837-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170519.2237-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170520.0237-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170520.0637-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170520.1037-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170520.1437-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170520.1837-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170520.2237-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170521.0237-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170521.0637-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170521.1037-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170521.1437-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170521.1837-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170521.2237-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170522.0237-3d     
          0      -    88K  -                     
tank/documents/<REDACTED>@auto-20170522.0637-3d     
          0      -    88K  -                     
tank/jails                                       
      2.93G  1.73G   112K  /mnt/tank/jails       
tank/jails/.warden-template-standard             
      1.75G  1.73G  1.68G  /mnt/tank/jails/.warde
n-template-standard                              
tank/jails/.warden-template-standard@clean       
      68.2M      -  1.68G  -                     
tank/jails/ffmpeg                                
      1.18G  1.73G  2.83G  /mnt/tank/jails/ffmpeg
tank/media                                       
      1.29T   630G    88K  /mnt/tank/media       
tank/media/general                               
       674G   350G   674G  /mnt/tank/media/genera
l                                                
tank/media/home                                  
       652G   280G   652G  /mnt/tank/media/home  
[root@freenas] ~#

Code:

[root@freenas] ~# zpool status                   
  pool: freenas-boot                             
 state: ONLINE                                   
  scan: scrub repaired 0 in 0h1m with 0 errors on
 Sat May 20 03:46:22 2017                        
config:                                          
                                                 
        NAME        STATE     READ WRITE CKSUM   
        freenas-boot  ONLINE       0     0     0 
          da0p2     ONLINE       0     0     0   
                                                 
errors: No known data errors                     
                                                 
  pool: tank                                     
 state: ONLINE                                   
  scan: scrub repaired 896K in 3h18m with 0 error
s on Sun May 21 03:18:38 2017                    
config:                                          
                                                 
        NAME                                     
       STATE     READ WRITE CKSUM                
        tank                                     
       ONLINE       0     0     0                
          mirror-0                               
       ONLINE       0     0     0                
            gptid/e4cd9fac-2114-11e7-bdcc-d05099c
12af7  ONLINE       0     0     0                
            gptid/e584e165-2114-11e7-bdcc-d05099c
12af7  ONLINE       0     0     0                
                                                 
errors: No known data errors

FreeNAS version: 9.10.2-U3.

Motherboard: ASRock Rack E3C236D2I

CPU: Intel Pentium G4500 @ 3.5 GHz

RAM: 1 × 16 GB DDR4 ECC 2133 MHz Kingston KVR21E15D8/16 UDIMM

Boot Drive: 1 × SanDisk Cruzer Blade 16 GB USB 2.0 Drive

Storage: 2 × WD Red 3.0 TB SATA drives

Arwen · May 21, 2017

Weird. There could be a setting in the Dataset properties. Though I would have thought it would
give a different error. Unless someone else comes up with something better, (and preferably quite
brilliant), please post the output of zfs get -r all tank/media in code tags.

SRSR333 · May 21, 2017

Arwen said:
please post the output of zfs get -r all tank/media in code tags.

As requested:

Code:

[root@freenas] ~# zfs get -r all tank/media
NAME				PROPERTY				 VALUE					SOURCE
tank/media		  type					 filesystem			   -
tank/media		  creation				 Mon Apr 17 19:04 2017	-
tank/media		  used					 1.29T					-
tank/media		  available				630G					 -
tank/media		  referenced			   88K					  -
tank/media		  compressratio			1.00x					-
tank/media		  mounted				  yes					  -
tank/media		  quota					1.91T					local
tank/media		  reservation			  745G					 local
tank/media		  recordsize			   128K					 default
tank/media		  mountpoint			   /mnt/tank/media		  default
tank/media		  sharenfs				 off					  default
tank/media		  checksum				 on					   default
tank/media		  compression			  lz4					  inherited from tank
tank/media		  atime					off					  inherited from tank
tank/media		  devices				  on					   default
tank/media		  exec					 on					   default
tank/media		  setuid				   on					   default
tank/media		  readonly				 off					  default
tank/media		  jailed				   off					  default
tank/media		  snapdir				  hidden				   default
tank/media		  aclmode				  restricted			   local
tank/media		  aclinherit			   passthrough			  inherited from tank
tank/media		  canmount				 on					   default
tank/media		  xattr					off					  temporary
tank/media		  copies				   1						default
tank/media		  version				  5						-
tank/media		  utf8only				 off					  -
tank/media		  normalization			none					 -
tank/media		  casesensitivity		  sensitive				-
tank/media		  vscan					off					  default
tank/media		  nbmand				   off					  default
tank/media		  sharesmb				 off					  default
tank/media		  refquota				 none					 local
tank/media		  refreservation		   none					 local
tank/media		  primarycache			 all					  default
tank/media		  secondarycache		   all					  default
tank/media		  usedbysnapshots		  0						-
tank/media		  usedbydataset			88K					  -
tank/media		  usedbychildren		   1.29T					-
tank/media		  usedbyrefreservation	 0						-
tank/media		  logbias				  latency				  default
tank/media		  dedup					off					  inherited from tank
tank/media		  mlslabel										  -
tank/media		  sync					 standard				 default
tank/media		  refcompressratio		 1.00x					-
tank/media		  written				  88K					  -
tank/media		  logicalused			  1.30T					-
tank/media		  logicalreferenced		36.5K					-
tank/media		  volmode				  default				  default
tank/media		  filesystem_limit		 none					 default
tank/media		  snapshot_limit		   none					 default
tank/media		  filesystem_count		 none					 default
tank/media		  snapshot_count		   none					 default
tank/media		  redundant_metadata	   all					  default
tank/media		  org.freenas:description						   local
tank/media/general  type					 filesystem			   -
tank/media/general  creation				 Mon Apr 17 19:06 2017	-
tank/media/general  used					 674G					 -
tank/media/general  available				350G					 -
tank/media/general  referenced			   674G					 -
tank/media/general  compressratio			1.00x					-
tank/media/general  mounted				  yes					  -
tank/media/general  quota					1.00T					local
tank/media/general  reservation			  none					 default
tank/media/general  recordsize			   128K					 default
tank/media/general  mountpoint			   /mnt/tank/media/general  default
tank/media/general  sharenfs				 off					  default
tank/media/general  checksum				 on					   default
tank/media/general  compression			  lz4					  inherited from tank
tank/media/general  atime					off					  inherited from tank
tank/media/general  devices				  on					   default
tank/media/general  exec					 on					   default
tank/media/general  setuid				   on					   default
tank/media/general  readonly				 off					  default
tank/media/general  jailed				   off					  default
tank/media/general  snapdir				  hidden				   default
tank/media/general  aclmode				  restricted			   local
tank/media/general  aclinherit			   passthrough			  inherited from tank
tank/media/general  canmount				 on					   default
tank/media/general  xattr					off					  temporary
tank/media/general  copies				   1						default
tank/media/general  version				  5						-
tank/media/general  utf8only				 off					  -
tank/media/general  normalization			none					 -
tank/media/general  casesensitivity		  sensitive				-
tank/media/general  vscan					off					  default
tank/media/general  nbmand				   off					  default
tank/media/general  sharesmb				 off					  default
tank/media/general  refquota				 none					 default
tank/media/general  refreservation		   466G					 local
tank/media/general  primarycache			 all					  default
tank/media/general  secondarycache		   all					  default
tank/media/general  usedbysnapshots		  0						-
tank/media/general  usedbydataset			674G					 -
tank/media/general  usedbychildren		   0						-
tank/media/general  usedbyrefreservation	 0						-
tank/media/general  logbias				  latency				  default
tank/media/general  dedup					off					  inherited from tank
tank/media/general  mlslabel										  -
tank/media/general  sync					 standard				 default
tank/media/general  refcompressratio		 1.00x					-
tank/media/general  written				  674G					 -
tank/media/general  logicalused			  675G					 -
tank/media/general  logicalreferenced		675G					 -
tank/media/general  volmode				  default				  default
tank/media/general  filesystem_limit		 none					 default
tank/media/general  snapshot_limit		   none					 default
tank/media/general  filesystem_count		 none					 default
tank/media/general  snapshot_count		   none					 default
tank/media/general  redundant_metadata	   all					  default
tank/media/general  org.freenas:description						   local
tank/media/home	 type					 filesystem			   -
tank/media/home	 creation				 Mon Apr 17 19:07 2017	-
tank/media/home	 used					 652G					 -
tank/media/home	 available				280G					 -
tank/media/home	 referenced			   652G					 -
tank/media/home	 compressratio			1.00x					-
tank/media/home	 mounted				  yes					  -
tank/media/home	 quota					931G					 local
tank/media/home	 reservation			  none					 default
tank/media/home	 recordsize			   128K					 default
tank/media/home	 mountpoint			   /mnt/tank/media/home	 default
tank/media/home	 sharenfs				 off					  default
tank/media/home	 checksum				 on					   default
tank/media/home	 compression			  lz4					  inherited from tank
tank/media/home	 atime					off					  inherited from tank
tank/media/home	 devices				  on					   default
tank/media/home	 exec					 on					   default
tank/media/home	 setuid				   on					   default
tank/media/home	 readonly				 off					  default
tank/media/home	 jailed				   off					  default
tank/media/home	 snapdir				  hidden				   default
tank/media/home	 aclmode				  restricted			   local
tank/media/home	 aclinherit			   passthrough			  inherited from tank
tank/media/home	 canmount				 on					   default
tank/media/home	 xattr					off					  temporary
tank/media/home	 copies				   1						default
tank/media/home	 version				  5						-
tank/media/home	 utf8only				 off					  -
tank/media/home	 normalization			none					 -
tank/media/home	 casesensitivity		  sensitive				-
tank/media/home	 vscan					off					  default
tank/media/home	 nbmand				   off					  default
tank/media/home	 sharesmb				 off					  default
tank/media/home	 refquota				 none					 default
tank/media/home	 refreservation		   466G					 local
tank/media/home	 primarycache			 all					  default
tank/media/home	 secondarycache		   all					  default
tank/media/home	 usedbysnapshots		  0						-
tank/media/home	 usedbydataset			652G					 -
tank/media/home	 usedbychildren		   0						-
tank/media/home	 usedbyrefreservation	 0						-
tank/media/home	 logbias				  latency				  default
tank/media/home	 dedup					off					  inherited from tank
tank/media/home	 mlslabel										  -
tank/media/home	 sync					 standard				 default
tank/media/home	 refcompressratio		 1.00x					-
tank/media/home	 written				  652G					 -
tank/media/home	 logicalused			  655G					 -
tank/media/home	 logicalreferenced		655G					 -
tank/media/home	 volmode				  default				  default
tank/media/home	 filesystem_limit		 none					 default
tank/media/home	 snapshot_limit		   none					 default
tank/media/home	 filesystem_count		 none					 default
tank/media/home	 snapshot_count		   none					 default
tank/media/home	 redundant_metadata	   all					  default
tank/media/home	 org.freenas:description						   local

Arwen · May 21, 2017

@SRSR333
Well, at least the snapshot_limit is the default.

You have both general and home using refreservation. The manual page has this;

Code:

  If refreservation is set, a snapshot is only allowed if there is enough free  pool  space  outside  of
  this reservation to accommodate the current number of "referenced" bytes in the dataset.

The only thing I can think of, is an interaction of these 2 child dataset's refreservation.

Do you need refreservation?

If not, remove and re-try your snapshots. If it does not help and you still want refreservation, simply restore them.

Good luck.

SRSR333 · May 21, 2017

Arwen said:
The only thing I can think of, is an interaction of these 2 child dataset's refreservation.

Do you need refreservation?

Thank you for that! Removing both datasets' refreservation and attempting a manual recursive snapshot of tank/media worked. I think the periodic snapshot task should work, too; I'll update the post accordingly.

Arwen · May 22, 2017

Great. Glad I could help. (And in a reasonable time frame.)

Please note that I don't truely understand all the details. I just noticed that the refreservation's were not set to the default.
For the more straight forward things like quota and reservation, I understand very clearly.

SRSR333 · May 22, 2017

@Arwen, a quick update—I think my periodic snapshots are working, too, because I haven't received any emails from my system about snapshots failing. Thank you very much for the help; I'll mark this post 'solved'.

Important Announcement for the TrueNAS Community.

SOLVED Why can't I snapshot my dataset?

SRSR333

Dabbler

SweetAndLow

Sweet'NASty

Arwen

MVP

SRSR333

Dabbler

Arwen

MVP

SRSR333

Dabbler

Arwen

MVP

SRSR333

Dabbler

Arwen

MVP

SRSR333

Dabbler

Similar threads