SOLVED boot-pool use rapidly increasing, but unable to determine why

I

ivy

Dabbler
Joined
Feb 2, 2021
Messages
12
Today I got the message: Space usage for pool "boot-pool" is 90%. Optimal pool performance requires used space remain below 80%. This has been steadily increasing throughout the week, but obviously I can't ignore it anymore.

I ran some diagnostic commands:
Code:
sudo sudo du /* -hsx --exclude /mnt | sort -h --reverse
du: cannot access '/proc/3809455/task/3809455/fd/4': No such file or directory
du: cannot access '/proc/3809455/task/3809455/fdinfo/4': No such file or directory
du: cannot access '/proc/3809455/fd/3': No such file or directory
du: cannot access '/proc/3809455/fdinfo/3': No such file or directory
2.2G    /usr
110M    /var
60M    /boot
42M    /opt
9.5M    /etc
8.0M    /run
2.5M    /tmp
1.2M    /root
330K    /conf
187K    /data
6.0K    /srv
1.0K    /home
512    /vmlinuz.old
512    /vmlinuz
512    /sbin
512    /media
512    /libx32
512    /lib64
512    /lib32
512    /lib
512    /initrd.img.old
512    /initrd.img
512    /bin
512    /backup
0    /sys
0    /proc
0    /dev


Code:
truenas% sudo zfs list -r boot-pool      
NAME                             USED  AVAIL     REFER  MOUNTPOINT
boot-pool                       86.1G  5.95G       96K  none
boot-pool/ROOT                  86.1G  5.95G       96K  none
boot-pool/ROOT/20.12-ALPHA      2.77G  5.95G     2.75G  legacy
boot-pool/ROOT/21.02-ALPHA.1    83.3G  5.95G     83.3G  legacy
boot-pool/ROOT/Initial-Install     8K  5.95G     2.65G  legacy
boot-pool/grub


Code:
truenas% sudo zfs get all boot-pool/ROOT/21.02-ALPHA.1
NAME                          PROPERTY                VALUE                   SOURCE
boot-pool/ROOT/21.02-ALPHA.1  type                    filesystem              -
boot-pool/ROOT/21.02-ALPHA.1  creation                Thu Feb 18 11:05 2021   -
boot-pool/ROOT/21.02-ALPHA.1  used                    83.3G                   -
boot-pool/ROOT/21.02-ALPHA.1  available               5.95G                   -
boot-pool/ROOT/21.02-ALPHA.1  referenced              83.3G                   -
boot-pool/ROOT/21.02-ALPHA.1  compressratio           1.66x                   -
boot-pool/ROOT/21.02-ALPHA.1  mounted                 yes                     -
boot-pool/ROOT/21.02-ALPHA.1  quota                   none                    default
boot-pool/ROOT/21.02-ALPHA.1  reservation             none                    default
boot-pool/ROOT/21.02-ALPHA.1  recordsize              128K                    default
boot-pool/ROOT/21.02-ALPHA.1  mountpoint              legacy                  local
boot-pool/ROOT/21.02-ALPHA.1  sharenfs                off                     default
boot-pool/ROOT/21.02-ALPHA.1  checksum                on                      default
boot-pool/ROOT/21.02-ALPHA.1  compression             on                      inherited from boot-pool
boot-pool/ROOT/21.02-ALPHA.1  atime                   on                      default
boot-pool/ROOT/21.02-ALPHA.1  devices                 off                     inherited from boot-pool
boot-pool/ROOT/21.02-ALPHA.1  exec                    on                      default
boot-pool/ROOT/21.02-ALPHA.1  setuid                  on                      default
boot-pool/ROOT/21.02-ALPHA.1  readonly                off                     default
boot-pool/ROOT/21.02-ALPHA.1  zoned                   off                     default
boot-pool/ROOT/21.02-ALPHA.1  snapdir                 hidden                  default
boot-pool/ROOT/21.02-ALPHA.1  aclmode                 discard                 default
boot-pool/ROOT/21.02-ALPHA.1  aclinherit              restricted              default
boot-pool/ROOT/21.02-ALPHA.1  createtxg               384634                  -
boot-pool/ROOT/21.02-ALPHA.1  canmount                on                      default
boot-pool/ROOT/21.02-ALPHA.1  xattr                   sa                      inherited from boot-pool
boot-pool/ROOT/21.02-ALPHA.1  copies                  1                       default
boot-pool/ROOT/21.02-ALPHA.1  version                 5                       -
boot-pool/ROOT/21.02-ALPHA.1  utf8only                on                      -
boot-pool/ROOT/21.02-ALPHA.1  normalization           formD                   -
boot-pool/ROOT/21.02-ALPHA.1  casesensitivity         sensitive               -
boot-pool/ROOT/21.02-ALPHA.1  vscan                   off                     default
boot-pool/ROOT/21.02-ALPHA.1  nbmand                  off                     default
boot-pool/ROOT/21.02-ALPHA.1  sharesmb                off                     default
boot-pool/ROOT/21.02-ALPHA.1  refquota                none                    default
boot-pool/ROOT/21.02-ALPHA.1  refreservation          none                    default
boot-pool/ROOT/21.02-ALPHA.1  guid                    12732828197860472709    -
boot-pool/ROOT/21.02-ALPHA.1  primarycache            all                     default
boot-pool/ROOT/21.02-ALPHA.1  secondarycache          all                     default
boot-pool/ROOT/21.02-ALPHA.1  usedbysnapshots         0B                      -
boot-pool/ROOT/21.02-ALPHA.1  usedbydataset           83.3G                   -
boot-pool/ROOT/21.02-ALPHA.1  usedbychildren          0B                      -
boot-pool/ROOT/21.02-ALPHA.1  usedbyrefreservation    0B                      -
boot-pool/ROOT/21.02-ALPHA.1  logbias                 latency                 default
boot-pool/ROOT/21.02-ALPHA.1  objsetid                286                     -
boot-pool/ROOT/21.02-ALPHA.1  dedup                   off                     default
boot-pool/ROOT/21.02-ALPHA.1  mlslabel                none                    default
boot-pool/ROOT/21.02-ALPHA.1  sync                    standard                default
boot-pool/ROOT/21.02-ALPHA.1  dnodesize               legacy                  default
boot-pool/ROOT/21.02-ALPHA.1  refcompressratio        1.66x                   -
boot-pool/ROOT/21.02-ALPHA.1  written                 83.3G                   -
boot-pool/ROOT/21.02-ALPHA.1  logicalused             114G                    -
boot-pool/ROOT/21.02-ALPHA.1  logicalreferenced       114G                    -
boot-pool/ROOT/21.02-ALPHA.1  volmode                 default                 default
boot-pool/ROOT/21.02-ALPHA.1  filesystem_limit        none                    default
boot-pool/ROOT/21.02-ALPHA.1  snapshot_limit          none                    default
boot-pool/ROOT/21.02-ALPHA.1  filesystem_count        none                    default
boot-pool/ROOT/21.02-ALPHA.1  snapshot_count          none                    default
boot-pool/ROOT/21.02-ALPHA.1  snapdev                 hidden                  default
boot-pool/ROOT/21.02-ALPHA.1  acltype                 posix                   inherited from boot-pool
boot-pool/ROOT/21.02-ALPHA.1  context                 none                    default
boot-pool/ROOT/21.02-ALPHA.1  fscontext               none                    default
boot-pool/ROOT/21.02-ALPHA.1  defcontext              none                    default
boot-pool/ROOT/21.02-ALPHA.1  rootcontext             none                    default
boot-pool/ROOT/21.02-ALPHA.1  relatime                on                      inherited from boot-pool
boot-pool/ROOT/21.02-ALPHA.1  redundant_metadata      all                     default
boot-pool/ROOT/21.02-ALPHA.1  overlay                 on                      default
boot-pool/ROOT/21.02-ALPHA.1  encryption              off                     default
boot-pool/ROOT/21.02-ALPHA.1  keylocation             none                    default
boot-pool/ROOT/21.02-ALPHA.1  keyformat               none                    default
boot-pool/ROOT/21.02-ALPHA.1  pbkdf2iters             0                       default
boot-pool/ROOT/21.02-ALPHA.1  special_small_blocks    0                       default
boot-pool/ROOT/21.02-ALPHA.1  zectl:keep              False                   local
boot-pool/ROOT/21.02-ALPHA.1  truenas:kernel_version  5.10.0-1-amd64          local
boot-pool/ROOT/21.02-ALPHA.1  org.zectl:bootloader    grub                    inherited from boot-pool/ROOT


I have created a new boot pool, but upon reboot, it becomes the same size as the old one.

Any help would be appreciated.
 
Last edited:
Arwen

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
You would normally want to look at the snapshots too, (see below). However, since you have supplied the output of "zfs get all ...", it shows you don't have any in snapshots.

Code:
sudo zfs list -t all -r boot-pool


Next, your du command skipped any hidden files or directories. Take a look at;

Code:
sudo du -sk /.[a-zA-Z0-9]* | sort -n


Usually when this happens, it's a case that the user implemented a background download task, but did not get the PATH to the data pool / dataset correct.
 
I

ivy

Dabbler
Joined
Feb 2, 2021
Messages
12
Right. No snapshots. I wish it were that simple.

No obvious hidden use, either:
Code:
sudo du -sk /.[a-zA-Z0-9]* | sort -n
1077    /.kube


Thanks for the help.

Edit: not sure if it's related, but I'm unable to delete the boot pool for the previous alpha of SCALE. No error messages. I can't activate it either.
 
I

ivy

Dabbler
Joined
Feb 2, 2021
Messages
12
Figured it out! One of my ZFS mounts was hiding a bunch of files from a Docker container underneath it. I exported the pool, deleted the files, and everything is good to go. Appreciate the help!
 
Arwen

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
Great.

I had forgotten about the overlay mount problem. Have to try and remember that.
 
I

infinitytec

Dabbler
Joined
Oct 14, 2022
Messages
11
ivy said:
Figured it out! One of my ZFS mounts was hiding a bunch of files from a Docker container underneath it. I exported the pool, deleted the files, and everything is good to go. Appreciate the help!
Click to expand...
I have a similar issue to what you described.

How would I go about checking for this?
 
You must log in or register to reply here.

Similar threads

T
Boot-pool full following automatic scrub [Solved]
Replies
10
Views
4K
artlessknave
artlessknave
C
nscd and rrdcached still constantly writing to boot-pool after system dataset moved
Replies
0
Views
656
calc
C
N
Replication broken due to orphaned system dataset
Replies
3
Views
443
NomsPlease
N
ctag
Migrating to Scale breaks the boot pool
Replies
4
Views
1K
ctag
ctag
R
Replicated Zvol size in the Ui
Replies
0
Views
401
rsriverrat
R
Top