freenas-boot capacity over 80% [solved]

[Pedro22]

Hi all,

This might be trivial, but I'm fairly new to BSD/FreeNAS/ZFS, and I'm having difficulties figuring out this one.

OS Version:
FreeNAS-11.2-U3
(Build Date: Mar 27, 2019 18:24)
Processor:
Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (12 cores)
Memory:
64 GiB

Current alerts:
* The capacity for the volume "freenas-boot" is currently at 96%, while the recommended value is below 80%.

At first, I thought I was keeping too many snapshots

But that didn't seem to be the case, so I double checked

Code:

# zfs list freenas-boot
NAME           USED  AVAIL  REFER  MOUNTPOINT
freenas-boot  53.8G  14.7M    64K  none

# zfs list -r -o space,refer,written -t all freenas-boot
NAME                                           AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  REFER  WRITTEN
freenas-boot                                   14.7M  53.8G         0     64K              0      53.8G    64K      64K
freenas-boot/ROOT                              14.7M  53.7G         0     29K              0      53.7G    29K      29K
freenas-boot/ROOT/11.2-U3                      14.7M  53.7G      847M   52.9G              0          0  52.9G    52.9G
freenas-boot/ROOT/11.2-U3@2018-08-07-06:56:23      -  2.28M         -       -              -          -   837M     837M
freenas-boot/ROOT/11.2-U3@2018-08-22-01:46:58      -  10.4M         -       -              -          -   845M    10.4M
freenas-boot/ROOT/Initial-Install              14.7M     1K         0      1K              0          0   837M       1K
freenas-boot/ROOT/default                      14.7M  8.29M         0   8.29M              0          0   845M    8.29M
freenas-boot/grub                              14.7M  7.02M         0   7.02M              0          0  7.02M    7.02M

Additionally checked /var/log and /data/crash for more possible hints, without success.

Reinstalling FreeNAS and restoring a backup wouldn't be too much of a hassle, but I'm genuinely interested in learning whether this is just me overlooking something obvious or something else.

Thanks in advance

 

[sretalla]

You need to carefully consider the possibility that you're mistakenly storing data on the boot pool.

Anything that isn't stored in /mnt/poolname... is stored in the root filesystem and I suspect you have about 50GB of data in that category somewhere.

 

[Pedro22]

You need to carefully consider the possibility that you're mistakenly storing data on the boot pool.

Anything that isn't stored in /mnt/poolname... is stored in the root filesystem and I suspect you have about 50GB of data in that category somewhere.

Thanks @sretalla,
I already considered your suggestion, actually, that's what I first checked and didn't saw anything fishy.

Code:

# du -hs * | sort -hs
512B    sys
512B    tmp
1.5K    net
1.5K    proc
3.0K    media
4.0K    COPYRIGHT
4.5K    entropy
5.0K    compat
7.0K    dev
 47K    root
107K    libexec
779K    bin
3.5M    sbin
5.9M    rescue
6.3M    lib
 10M    etc
 14M    conf
 23M    data
 75M    boot
161M    var
656M    usr
731G    mnt

So this morning I started checking our custom scripts in cron.
Long story short, the problem was caused by a script trying to send a snapshot to a remote system, which indeed was writing to the root filesystem.
This was not evident at first, because the script was deleting the job on error.
Since deletes happen asynchronously, my guess is that depending when I was checking the system, the email warning was already sent and/or the space was used or about to be reclaimed ... If that makes any sense.

Marking as solved.
Thanks for your help.