Zpool filling up by the minute

Status
Not open for further replies.
J

jespergc

Cadet
Joined
Jul 18, 2014
Messages
5
Recently upgraded to 9.10-STABLE-201606270534 (dd17351).

This morning i wake up to mails saying The capacity for the volume 'disk1' is currently at 90%, while the recommended value is below 80%. increasing from 90 to 95 % in less than an hour.
I now find out that I have lost 2,5 tb of free space since around 19:00 yesterday.

I cannot seem to find the problem.
Picture of the partition tab from the web UI.
9D9VNRv.png


Heres a "zpool list" "zfs list" and "df -h"

Code:
[root@freenas] /mnt/disk1# zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
disk1         8.12T  7.84T   295G         -    52%    96%  1.00x  ONLINE  /mnt
disk2         8.12T  6.13T  1.99T         -    31%    75%  1.00x  ONLINE  /mnt
freenas-boot  7.19G  2.78G  4.41G         -      -    38%  1.00x  ONLINE  -
[root@freenas] /mnt/disk1# zfs list
NAME                                                         USED  AVAIL  REFER  MOUNTPOINT
disk1                                                       5.22T  23.4G  2.59T  /mnt/disk1
disk1/.system                                               36.7M  23.4G   139K  legacy
disk1/.system/configs-53b355071d334fc1942cfddcf5a8e000      26.2M  23.4G  26.2M  legacy
disk1/.system/cores                                         2.86M  23.4G  2.86M  legacy
disk1/.system/rrd-53b355071d334fc1942cfddcf5a8e000           128K  23.4G   128K  legacy
disk1/.system/rrd-75746dd0efdd4abc9f3dc4c9d12025c7           128K  23.4G   128K  legacy
disk1/.system/samba4                                        4.60M  23.4G  4.60M  legacy
disk1/.system/syslog-53b355071d334fc1942cfddcf5a8e000       2.45M  23.4G  2.45M  legacy
disk1/.system/syslog-75746dd0efdd4abc9f3dc4c9d12025c7        208K  23.4G   208K  legacy
disk1/.warden-template-pluginjail-9.3-x64                    786M  23.4G   781M  /mnt/disk1/.warden-template-pluginjail-9.3-x64
disk1/.warden-template-standard                             1.98G  23.4G  1.98G  /mnt/disk1/.warden-template-standard
disk1/.warden-template-standard-9.3-x64                     1.96G  23.4G  1.90G  /mnt/disk1/.warden-template-standard-9.3-x64
disk1/backup                                                64.7G  23.4G  64.7G  /mnt/disk1/backup
disk1/beet                                                  56.7G  23.4G  58.3G  /mnt/disk1/beet
disk1/couchpotato_1                                          327M  23.4G  1.07G  /mnt/disk1/couchpotato_1
disk1/downloads                                              456G  23.4G   456G  /mnt/disk1/downloads
disk1/film                                                  1.76T  23.4G  1.76T  /mnt/disk1/film
disk1/incomplete                                             170K  23.4G   170K  /mnt/disk1/incomplete
disk1/mineos_1                                               507M  23.4G  1.25G  /mnt/disk1/mineos_1
disk1/musik                                                 83.4G  23.4G  83.4G  /mnt/disk1/musik
disk1/mylar_1                                                186M  23.4G   961M  /mnt/disk1/mylar_1
disk1/owncloud_1                                            6.14G  23.4G  6.89G  /mnt/disk1/owncloud_1
disk1/plexmediaserver_1                                     11.2G  23.4G  11.6G  /mnt/disk1/plexmediaserver_1
disk1/publicFTP                                              128K  23.4G   128K  /mnt/disk1/publicFTP
disk1/seeds                                                  201G  23.4G   201G  /mnt/disk1/seeds
disk1/serier                                                 202K  23.4G   202K  /mnt/disk1/serier
disk1/sickrage_1                                            1.00G  23.4G  1.44G  /mnt/disk1/sickrage_1
disk1/teamspeak                                             1.29G  23.4G  2.24G  /mnt/disk1/teamspeak
disk1/transmission_1                                        1.83G  23.4G  2.19G  /mnt/disk1/transmission_1
disk2                                                       4.08T  1.16T  4.08T  /mnt/disk2
freenas-boot                                                2.77G  4.19G    31K  none
freenas-boot/ROOT                                           2.72G  4.19G    25K  none
freenas-boot/ROOT/FreeNAS-8bc815b059fa92f1c8ba7c7685deacbb  2.66G  4.19G   612M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201505130355           4.39M  4.19G   497M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201508250051           11.5M  4.19G   521M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509282017           13.2M  4.19G   524M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201512121950           12.7M  4.19G   524M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011           16.2M  4.19G   528M  /
freenas-boot/ROOT/Initial-Install                              1K  4.19G   466M  legacy
freenas-boot/ROOT/default                                     36K  4.19G   471M  legacy
freenas-boot/grub                                           47.0M  4.19G  6.33M  legacy
[root@freenas] /mnt/disk1# df -h
Filesystem                                                    Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/FreeNAS-8bc815b059fa92f1c8ba7c7685deacbb    4.8G    612M    4.2G    12%    /
devfs                                                         1.0K    1.0K      0B   100%    /dev
tmpfs                                                          32M    8.1M     24M    25%    /etc
tmpfs                                                         4.0M    8.0K    4.0M     0%    /mnt
tmpfs                                                         5.3G    133M    5.2G     2%    /var
freenas-boot/grub                                             4.2G    6.3M    4.2G     0%    /boot/grub
fdescfs                                                       1.0K    1.0K      0B   100%    /dev/fd
disk1                                                         2.6T    2.6T     23G    99%    /mnt/disk1
disk1/.warden-template-pluginjail-9.3-x64                      24G    781M     23G     3%    /mnt/disk1/.warden-template-pluginjail-9.3-x64
disk1/.warden-template-standard-9.3-x64                        25G    1.9G     23G     8%    /mnt/disk1/.warden-template-standard-9.3-x64
disk1/backup                                                   88G     65G     23G    73%    /mnt/disk1/backup
disk1/beet                                                     82G     58G     23G    71%    /mnt/disk1/beet
disk1/couchpotato_1                                            25G    1.1G     23G     4%    /mnt/disk1/couchpotato_1
disk1/downloads                                               479G    456G     23G    95%    /mnt/disk1/downloads
disk1/film                                                    1.8T    1.8T     23G    99%    /mnt/disk1/film
disk1/incomplete                                               23G    171K     23G     0%    /mnt/disk1/incomplete
disk1/mineos_1                                                 25G    1.3G     23G     5%    /mnt/disk1/mineos_1
disk1/musik                                                   107G     83G     23G    78%    /mnt/disk1/musik
disk1/mylar_1                                                  24G    961M     23G     4%    /mnt/disk1/mylar_1
disk1/owncloud_1                                               30G    6.9G     23G    23%    /mnt/disk1/owncloud_1
disk1/plexmediaserver_1                                        35G     12G     23G    33%    /mnt/disk1/plexmediaserver_1
disk1/publicFTP                                                23G    128K     23G     0%    /mnt/disk1/publicFTP
disk1/seeds                                                   224G    201G     23G    90%    /mnt/disk1/seeds
disk1/serier                                                   23G    202K     23G     0%    /mnt/disk1/serier
disk1/sickrage_1                                               25G    1.4G     23G     6%    /mnt/disk1/sickrage_1
disk1/transmission_1                                           26G    2.2G     23G     9%    /mnt/disk1/transmission_1
disk2                                                         5.2T    4.1T    1.2T    78%    /mnt/disk2
disk1/.system                                                  23G    139K     23G     0%    /var/db/system
disk1/.system/cores                                            23G    2.9M     23G     0%    /var/db/system/cores
disk1/.system/samba4                                           23G    4.6M     23G     0%    /var/db/system/samba4
disk1/.system/syslog-53b355071d334fc1942cfddcf5a8e000          23G    2.4M     23G     0%    /var/db/system/syslog-53b355071d334fc1942cfddcf5a8e000
disk1/.system/rrd-53b355071d334fc1942cfddcf5a8e000             23G    128K     23G     0%    /var/db/system/rrd-53b355071d334fc1942cfddcf5a8e000
disk1/.system/configs-53b355071d334fc1942cfddcf5a8e000         23G     26M     23G     0%    /var/db/system/configs-53b355071d334fc1942cfddcf5a8e000
linprocfs                                                     4.0K    4.0K      0B   100%    /compat/linux/proc
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/beet/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/beet/proc
/mnt/disk1/musik                                              107G     83G     23G    78%    /mnt/disk1/beet/media/music
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/couchpotato_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/couchpotato_1/proc
/mnt/disk1/downloads                                          479G    456G     23G    95%    /mnt/disk1/couchpotato_1/media/downloads
/mnt/disk1/film                                               1.8T    1.8T     23G    99%    /mnt/disk1/couchpotato_1/media/film
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/mineos_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/mineos_1/proc
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/mylar_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/mylar_1/proc
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/owncloud_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/owncloud_1/proc
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/plexmediaserver_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/plexmediaserver_1/proc
/mnt/disk1/film                                               1.8T    1.8T     23G    99%    /mnt/disk1/plexmediaserver_1/media/film
/mnt/disk2/serier                                             5.2T    4.1T    1.2T    78%    /mnt/disk1/plexmediaserver_1/media/serier
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/sickrage_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/sickrage_1/proc
/mnt/disk1/downloads                                          479G    456G     23G    95%    /mnt/disk1/sickrage_1/media/downloads
/mnt/disk1/seeds                                              224G    201G     23G    90%    /mnt/disk1/sickrage_1/media/seeds
/mnt/disk2/serier                                             5.2T    4.1T    1.2T    78%    /mnt/disk1/sickrage_1/media/serier
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/transmission_1/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/transmission_1/proc
/mnt/disk1/downloads                                          479G    456G     23G    95%    /mnt/disk1/transmission_1/media/downloads
/mnt/disk1/film                                               1.8T    1.8T     23G    99%    /mnt/disk1/transmission_1/media/film
/mnt/disk1/incomplete                                          23G    171K     23G     0%    /mnt/disk1/transmission_1/media/incomplete
/mnt/disk1/musik                                              107G     83G     23G    78%    /mnt/disk1/transmission_1/media/musik
/mnt/disk1/seeds                                              224G    201G     23G    90%    /mnt/disk1/transmission_1/media/seeds
/mnt/disk1/serier                                              23G    202K     23G     0%    /mnt/disk1/transmission_1/media/serier
disk1/.warden-template-standard                                25G    2.0G     23G     8%    /mnt/disk1/.warden-template-standard
disk1/teamspeak                                                26G    2.2G     23G     9%    /mnt/disk1/teamspeak
devfs                                                         1.0K    1.0K      0B   100%    /mnt/disk1/teamspeak/dev
procfs                                                        4.0K    4.0K      0B   100%    /mnt/disk1/teamspeak/proc



It shows that disk1 is 8.12 tb of size in zpool list, but in the others it show as 5.22 tb , which for me is pretty wierd.


Can you help me on how to find the problem and get it to stop?
 
J

jespergc

Cadet
Joined
Jul 18, 2014
Messages
5
Hmm, shut down the FTP service because i see one of the usernames under system processes. The activity on the disk has now stopped. Still can't seem to find where all this data comes from or where it has been placed.

System specs are:
FreeNAS-9.10-STABLE-201606270534 (dd17351)
Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
16 gigs of non-ecc ram
2 zpools of 3 x 3tb WD Red harddrives

System processes

Code:
  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
16873    921        4  20    0   204M   103M kqread  2   7:36   1.07% transmission-daemon
91027 root          1  20    0   337M 32860K zio->i  1   2:07   1.07% smbd
15077    816       18  20    0   637M   348M select  3  24:10   0.78% python2.7
5897    816        2  20    0   363M   218M kqread  0   9:50   0.00% python2.7
89753    942       25  52    0   109M 21332K uwait   0   4:27   0.00% ts3server
16882 root          1  20    0 18296K  4908K select  0   3:39   0.00% openvpn
2862 root         12  20    0   235M 25772K nanslp  2   2:11   0.00% collectd
12623    972       16  20    0   333M   146M uwait   1   1:20   0.00% Plex Media Server
2626 root          6  20    0   409M   181M select  2   0:57   0.00% python2.7
13769    972       13  52   15   265M 66744K piperd  1   0:43   0.00% Plex Script Host
13771 root         23  20    0   238M 59620K select  3   0:37   0.00% python2.7
2599 root          1  52    0   227M 64272K select  0   0:24   0.00% python2.7
18998    972       10  52    0   207M 41768K piperd  1   0:14   0.00% Plex Script Host
1388 root          2  20    0 73148K  8760K kqread  3   0:12   0.00% syslog-ng
12144 www           1  20    0   409M 64856K kqread  3   0:06   0.00% httpd
12145 www           1  20    0   417M 64424K lockf   2   0:06   0.00% httpd
1711 root          1  20    0 14468K  2008K select  2   0:06   0.00% powerd
12147 www           1  20    0   421M 60640K lockf   3   0:05   0.00% httpd


EDIT:: Found out it was still filling up disk1 after i turned off the FTP service and actually deleted the user using it. Restart has not helped.

I now have 0 mb free data on disk1.
 
Last edited:
J

jespergc

Cadet
Joined
Jul 18, 2014
Messages
5
I did that and found Ththat a directory called FTP-user was taking up 2,7 TB.
So it must be some problem with my former FTP user.

The problem has changed slightly. My disk now has 0 mb free. So i cannot delete the folder again because of the "No space left on device" messange.

Do you guys have any ideas?
 
C

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Zero out a file with # echo "" > somefilename
 
M

mattbbpl

Patron
Joined
May 30, 2015
Messages
237
cyberjock said:
Zero out a file with # echo "" > somefilename
Click to expand...
Had to do this on an Oracle server the other day because OMCS didn't monitor the thing. It's an odd quirk of the ZFS COW filesystem.
 
J

jespergc

Cadet
Joined
Jul 18, 2014
Messages
5
Thanks for the replies. I have been on vacation and have not been able to look into the solution before now. But I am able to zero out a file and then hopefully remove some afterwards.

I will report back later!

Again, thank you for your help!
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Zpool filling up by the minute"

Similar threads

Z
  • Locked
SOLVED Files missing after upgrade
Replies
7
Views
4K
Zenith
Z
G
  • Locked
Received critical alert emails overnight for freenas-boot volume
Replies
4
Views
2K
cyberjock
C
J
  • Locked
zpool list size changed? (or: does not match zfs list and df-h sizes)
Replies
3
Views
6K
thornae
thornae
vom
  • Locked
Safe to remove old plugin jail volumes ? Which ones ?
Replies
1
Views
2K
dlavigne
D
alexhore
  • Locked
How Can I re-import this volume
Replies
6
Views
2K
alexhore
alexhore
Top