Can not create snapshot: out of space error

mahmud100 · May 11, 2023

I am new to TrueNAS Core, and have spent the last couple of days on the forums and YouTube to try and solve this issue. I do not know what I may be missing and need a little help.

Code:


Error: concurrent.futures.process._RemoteTraceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 1019, in do_create
    ds.properties['freenas:vmsynced'] = libzfs.ZFSUserProperty('Y')
  File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 1016, in do_create
    ds.snapshot(f'{dataset}@{name}', recursive=recursive, fsopts=properties)
  File "libzfs.pyx", line 3798, in libzfs.ZFSDataset.snapshot
libzfs.ZFSException: out of space

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 1024, in do_create
    raise CallError(f'Failed to snapshot {dataset}@{name}: {err}')
middlewared.service_exception.CallError: [EFAULT] Failed to snapshot Storage_Pool/Company_Data/Company_Data@manual-2023-05-11_14-51: out of space
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 139, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1236, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/service.py", line 496, in create
    rv = await self.middleware._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1244, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1250, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1169, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EFAULT] Failed to snapshot Storage_Pool/Company_Data/Company_Data@manual-2023-05-11_14-51: out of space

********************************************************************************************************************************

root@SIVERNAS1P[~]# zfs list
NAME                                                            USED  AVAIL     REFER  MOUNTPOINT
Storage_Pool                                                   4.43T  2.49T      151K  /mnt/Storage_Pool
Storage_Pool/.system                                           52.2M  2.49T      163K  legacy
Storage_Pool/.system/configs-2663265f094743bfbd6dcc41661bf096  12.1M  2.49T     12.1M  legacy
Storage_Pool/.system/cores                                      140K  1024M      140K  legacy
Storage_Pool/.system/rrd-2663265f094743bfbd6dcc41661bf096      38.1M  2.49T     38.1M  legacy
Storage_Pool/.system/samba4                                     442K  2.49T      442K  legacy
Storage_Pool/.system/services                                   140K  2.49T      140K  legacy
Storage_Pool/.system/syslog-2663265f094743bfbd6dcc41661bf096   1.01M  2.49T     1.01M  legacy
Storage_Pool/.system/webui                                      140K  2.49T      140K  legacy
Storage_Pool/Company_Data                                      4.43T  2.49T      140K  /mnt/Storage_Pool/Company_Data
Storage_Pool/Company_Data/Company_Data                         4.43T  4.36T     2.56T  -
Storage_Pool/iocage                                            9.41M  2.49T     8.59M  /mnt/Storage_Pool/iocage
Storage_Pool/iocage/download                                    140K  2.49T      140K  /mnt/Storage_Pool/iocage/download
Storage_Pool/iocage/images                                      140K  2.49T      140K  /mnt/Storage_Pool/iocage/images
Storage_Pool/iocage/jails                                       140K  2.49T      140K  /mnt/Storage_Pool/iocage/jails
Storage_Pool/iocage/log                                         140K  2.49T      140K  /mnt/Storage_Pool/iocage/log
Storage_Pool/iocage/releases                                    140K  2.49T      140K  /mnt/Storage_Pool/iocage/releases
Storage_Pool/iocage/templates                                   140K  2.49T      140K  /mnt/Storage_Pool/iocage/templates
boot-pool                                                      3.88G   142G       24K  none
boot-pool/ROOT                                                 3.87G   142G       24K  none
boot-pool/ROOT/13.0-U3.1                                        183K   142G     1.29G  /
boot-pool/ROOT/13.0-U4                                         3.87G   142G     1.29G  /
boot-pool/ROOT/Initial-Install                                    1K   142G     1.29G  legacy
boot-pool/ROOT/default                                          182K   142G     1.29G  legacy

******************************************************************************************************************************

root@SIVERNAS1P[~]# zfs list -o space,quota
NAME                                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  QUOTA
Storage_Pool                                                   2.49T  4.43T        0B    151K             0B      4.43T   none
Storage_Pool/.system                                           2.49T  52.2M        0B    163K             0B      52.0M   none
Storage_Pool/.system/configs-2663265f094743bfbd6dcc41661bf096  2.49T  12.1M        0B   12.1M             0B         0B   none
Storage_Pool/.system/cores                                     1024M   140K        0B    140K             0B         0B     1G
Storage_Pool/.system/rrd-2663265f094743bfbd6dcc41661bf096      2.49T  38.1M        0B   38.1M             0B         0B   none
Storage_Pool/.system/samba4                                    2.49T   442K        0B    442K             0B         0B   none
Storage_Pool/.system/services                                  2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/.system/syslog-2663265f094743bfbd6dcc41661bf096   2.49T  1.01M        0B   1.01M             0B         0B   none
Storage_Pool/.system/webui                                     2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/Company_Data                                      2.49T  4.43T        0B    140K             0B      4.43T   none
Storage_Pool/Company_Data/Company_Data                         4.36T  4.43T        0B   2.56T          1.87T         0B      -
Storage_Pool/iocage                                            2.49T  9.41M        0B   8.59M             0B       837K   none
Storage_Pool/iocage/download                                   2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/iocage/images                                     2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/iocage/jails                                      2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/iocage/log                                        2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/iocage/releases                                   2.49T   140K        0B    140K             0B         0B   none
Storage_Pool/iocage/templates                                  2.49T   140K        0B    140K             0B         0B   none
boot-pool                                                       142G  3.88G        0B     24K             0B      3.88G   none
boot-pool/ROOT                                                  142G  3.87G        0B     24K             0B      3.87G   none
boot-pool/ROOT/13.0-U3.1                                        142G   183K        0B    183K             0B         0B   none
boot-pool/ROOT/13.0-U4                                          142G  3.87G     2.58G   1.29G             0B         0B   none
boot-pool/ROOT/Initial-Install                                  142G     1K        0B      1K             0B         0B   none
boot-pool/ROOT/default

*******************************************************************************************************************************

root@SIVERNAS1P[~]# zpool status
  pool: Storage_Pool
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        Storage_Pool                                    ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/40114790-ec47-11ed-b577-fcaa1499c7c2  ONLINE       0     0     0
            gptid/404532db-ec47-11ed-b577-fcaa1499c7c2  ONLINE       0     0     0
            gptid/40283f57-ec47-11ed-b577-fcaa1499c7c2  ONLINE       0     0     0
            gptid/403703be-ec47-11ed-b577-fcaa1499c7c2  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:10 with 0 errors on Fri May  5 03:45:10 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada4p2    ONLINE       0     0     0

errors: No known data errors

*******************************************************************************************************************************

root@SIVERNAS1P[~]# zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Storage_Pool  14.5T  5.29T  9.26T        -         -     0%    36%  1.00x    ONLINE  /mnt
boot-pool      151G  3.88G   147G        -         -     0%     2%  1.00x    ONLINE  -
root@SIVERNAS1P[~]#

NugentS · May 11, 2023

Well I would say you have run out of disk space. What makes you think it isn't that?

Look at the dashboard - the Pool widgit.
Or
Storage/Pools and post a screenshot of what you see there

mahmud100 · May 11, 2023

mahmud100 · May 11, 2023

Thanks for your time

NugentS · May 12, 2023

Can you post your hardware please - as per forum rules

mahmud100 · May 16, 2023

Datto S3B2000
Intel(R) Core(TM) i3-4150T CPU @ 3.00GHz
16MB RAM
1 167.68 GiB SSD HDD - System boot
4 4TB WD Red HDD
Storage_Pool RAID2 4.4 TiB
iocage RAID2 10 MiB
TrueNAS-13.0-U4 (Core)
Use Case: Primary File and Media Storage

Whattteva · May 16, 2023

Is Company_Data a zvol or a dataset? That distinction matters when you're taking a snapshot.

mahmud100 · May 16, 2023

Whattteva · May 16, 2023

AHA! That's your problem. You're trying to take a snapshot of a 4 TiB zvol with only 2.49 TiB space left.
Yes I know, snapshots are supposed to be lightweight, but zvols are an exception unfortunately.

Basically, if you do a zfs list Storage_Pool/Company_Data/Company_Data you might get something like this:

Code:

NAME                          USED  AVAIL     REFER  MOUNTPOINT
Storage_Pool/Company_Data/Company_Data  4.43T   <some_amount>    4.04T  -

In order for that zvol to be snapshottable, you must have more space available than what's listed in the REFER section.

mahmud100 · May 19, 2023

Whattteva said:
AHA! That's your problem. You're trying to take a snapshot of a 4 TiB zvol with only 2.49 TiB space left.
Yes I know, snapshots are supposed to be lightweight, but zvols are an exception unfortunately.

Basically, if you do a zfs list Storage_Pool/Company_Data/Company_Data you might get something like this:

Code:
NAME USED AVAIL REFER MOUNTPOINT Storage_Pool/Company_Data/Company_Data 4.43T <some_amount> 4.04T -

In order for that zvol to be snapshottable, you must have more space available than what's listed in the REFER section.

Thanks, I will have to upgrade the disk set.

Important Announcement for the TrueNAS Community.

Can not create snapshot: out of space error

mahmud100

Cadet

Attachments

NugentS

MVP

mahmud100

Cadet

mahmud100

Cadet

NugentS

MVP

mahmud100

Cadet

Whattteva

Wizard

mahmud100

Cadet

Whattteva

Wizard

mahmud100

Cadet

Similar threads