Lost around 2TiB worth of space overnight

[cgcmgr]

Hi everyone,

I really hope you can help me. On August 29th, my Pool showed 8.16TiB used and 2.25TiB available. see attachment. I did add a few movies to my NAS , but nothing more than 100Gb worth of data. I also configured WinSCP to SSH into my NAS so that I can copy files between my Windows machine and my NAS. I tested by copying a few files, but then deleted them. The next morning my NAS was running very slow and I could not access my Emby server. I also could barely open my media drives via Windows. They would sit and open after about a minute. The same went for WinSCP. All of a sudden it wouldn't open until after a minute or two. I checked my NAS and noticed it was showing only 5.34MiB of space left, see attachment. Where did all of my space go? If you look at both of the Pool attachments, you can see iocage, Media and Personal are all pretty much that same in both captures. So where is all this extra data?

I did read a few other threads about lost space overnight and ran a zfs list and a zfs list -t snapshot and attached them. The output was too large to grab screen shots., so I copied the output into a Word file and created a PDF. Although there seems to be a lot of Snapshots, I do not have Snapshots turned on, see attachment.

Is anyone able to determine what happened overnight? Is there a log file I can check to see what was written overnight and where to?

I'm wondering if at this point it just makes sense to start over with TrueNAS 13, I'm on 12. I do have a few Config Files based on my 12 setup. Would I be able to start clean with 13 and import the 12 Config? Would that work or would I really have to start all over from scratch?

If anyone can help, I'd greatly appreciate it.

Thanks,
Chris

 

[cgcmgr]

Bumping this thread with another question. I did some digging and looking at my storage structure in the TrueNAS GUI, I see my NAS (ChrisNAS) with iocage, Media and Personal beneath it as it should be and as you see in the attached pictures above. Using WinSCP, I looked at my storage structure. I see the name of my NAS (ChrisNAS) and beneath it is iocage, Media, Personal and root. Root only contains a folder named emby-server. I don't think this is my emby-server as I have my two Emby servers within iocage. Looking at the size of the root folder, it is 2,289GB which pretty much equates to the missing 2.23TiB. I'm wondering if this somehow got copied there.

Two questions:
1. Does anyone else have the root folder in this location? I'm wondering f I can just delete it.
2. If I do delete it and something doesn't work, can I just copy it back and whatever didn't work, will now work?

Just trying to get the space back.

Thanks,
Chris

 

[winnielinnie]

It makes it more difficult for others to help when you attach a bunch of screenshots and then PDF documents that need to be downloaded to view the contents.

You can use the [code][/code] and [spoiler][/spoiler] tags to share text in the post itself.

For example, using only [code][/code]:

Code:

truenas% ls -l /usr/share
total 39
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 atf
drwxr-xr-x    9 root  wheel   20 Jul 15 12:49 bsdconfig
drwxr-xr-x   12 root  wheel   12 Jul 15 12:49 calendar
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 certs
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 dict
drwxr-xr-x   10 root  wheel   10 Jul 15 12:49 doc
drwxr-xr-x    2 root  wheel   14 Jul 15 12:49 dtrace
drwxr-xr-x   41 root  wheel   41 Jul 15 12:49 examples
drwxr-xr-x    2 root  wheel    3 Jul 15 12:49 firmware
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 flua
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 games
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 i18n
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 keys
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 kyua
drwxr-xr-x  195 root  wheel  195 Jul 15 12:49 locale
drwxr-xr-x   12 root  wheel   13 Jul 15 12:49 man
drwxr-xr-x    3 root  wheel   27 Jul 15 12:49 misc
drwxr-xr-x    2 root  wheel   58 Jul 15 12:49 mk
drwxr-xr-x  148 root  wheel  148 Jul 15 12:49 nls
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 openssl
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 security
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 sendmail
drwxr-xr-x    2 root  wheel    9 Jul 15 12:49 skel
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 snmp
drwxr-xr-x    5 root  wheel    5 Jul 15 12:49 syscons
drwxr-xr-x    2 root  wheel   22 Jul 15 12:49 tabset
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 vi
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 vt
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 zfs
drwxr-xr-x   18 root  wheel   66 Jul 15 12:49 zoneinfo

Another example, using [code][/code] within [spoiler][/spoiler]:

Show : Directory listing of /usr/share

Code:

truenas% ls -l /usr/share
total 39
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 atf
drwxr-xr-x    9 root  wheel   20 Jul 15 12:49 bsdconfig
drwxr-xr-x   12 root  wheel   12 Jul 15 12:49 calendar
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 certs
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 dict
drwxr-xr-x   10 root  wheel   10 Jul 15 12:49 doc
drwxr-xr-x    2 root  wheel   14 Jul 15 12:49 dtrace
drwxr-xr-x   41 root  wheel   41 Jul 15 12:49 examples
drwxr-xr-x    2 root  wheel    3 Jul 15 12:49 firmware
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 flua
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 games
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 i18n
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 keys
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 kyua
drwxr-xr-x  195 root  wheel  195 Jul 15 12:49 locale
drwxr-xr-x   12 root  wheel   13 Jul 15 12:49 man
drwxr-xr-x    3 root  wheel   27 Jul 15 12:49 misc
drwxr-xr-x    2 root  wheel   58 Jul 15 12:49 mk
drwxr-xr-x  148 root  wheel  148 Jul 15 12:49 nls
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 openssl
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 security
drwxr-xr-x    2 root  wheel    2 Jul 15 12:49 sendmail
drwxr-xr-x    2 root  wheel    9 Jul 15 12:49 skel
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 snmp
drwxr-xr-x    5 root  wheel    5 Jul 15 12:49 syscons
drwxr-xr-x    2 root  wheel   22 Jul 15 12:49 tabset
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 vi
drwxr-xr-x    4 root  wheel    4 Jul 15 12:49 vt
drwxr-xr-x    3 root  wheel    3 Jul 15 12:49 zfs
drwxr-xr-x   18 root  wheel   66 Jul 15 12:49 zoneinfo

 

[winnielinnie]

I tried viewing the PDFs, but the broken formatting makes my eyes cross.

Can you post the output directly in here?

Code:

zfs list -t filesystem -r -o space ChrisNAS

My hunch is that you (or "something") is saving data directly inside the top-level root dataset. (Perhaps a rogue logfile? A misconfigured share?)

 

[cgcmgr]

I get ya with the formatting. I'm caught up in work right now, but will repost later today. I do appreciate the response. What I did notice last night is that within TrueNAS when looking at Storage, I see my NAS name (ChrisNAS) followed by iocage, Media and Personal. Iocage contains my two Emby servers, Media contains all of my movies and pictures and music. etc. and Personal is just personal stuff. When using WinSCP to SSH into my NAS, I navigate to ChrisNAS and see iocage, Media, Personal and root. Within root is a lone folder named emby-server. I find this odd since my Emby servers are located in iocage. The size of root is 2,289GB which almost equates to the missing 2.25TiB. I'm wondering if that is the cause of the lost space and if can delete the root folder or the emby-server within it without issue. I can't really recall if the root folder was here all along and if it was, if it contained the emby-server folder or not. The root folder does not contain anything other than the emby-server folder that I an see, unless there are hidden files and folders.

 

[winnielinnie]

My hunch is that you (or "something") is saving data directly inside the top-level root dataset. (Perhaps a rogue logfile? A misconfigured share?)

When using WinSCP to SSH into my NAS, I navigate to ChrisNAS and see iocage, Media, Personal and root. Within root is a lone folder named emby-server. I find this odd since my Emby servers are located in iocage. The size of root is 2,289GB which almost equates to the missing 2.25TiB.

You misconfigured something at one point. Perhaps by accident, or perhaps an oversight in your jail's mountpoints. You in fact have data living in the top-level root dataset. This is a big "no no". I believe TrueNAS Core 13 prevents you from changing the permissions on the root dataset. But you said you're still on Core 12.

 

[cgcmgr]

Thanks for answering again. If I misconfigured something, I must have done so from the very beginning in 2018 when I first installed FreeNAS 11. As I was all new to this I followed some tutorials to set everything up. How would you recommend setting it up?

Please educate me as I'd like to get this right and prevent any future damage.. You say I have data living in the top level root dataset. When you refer to data, are you referring to my movies, pictures and the like? All that stuff is in Media in the pic I attached in the original post. Are you saying Media and Personal are in the top level dataset? I thought Media and Personal were their own separate datasets. If you are referring to the emby-server folder in iocage/root, I have no clue what that is and was actually thinking about deleting it. I have included a pic of my mountpoints.

As I said, I had this running since FreeNAS 11 and only experienced this issue a few days ago. The only data I save is to the folders I have created within Media so I don't think it was me saving anything. A rogue log file could very well be possible, but why only now?

Below is the output of zfs list -t filesystem -r -o space ChrisNAS. Hopefully I got this thing right.

Code:

NAME                                                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
ChrisNAS                                                   4.88G  10.4T        0B   2.23T             0B      8.17T
ChrisNAS/.system                                           4.88G   212M        0B    151K             0B       212M
ChrisNAS/.system/configs-229b43ea8de442e68f6896e2baff0028  4.88G   148M        0B    148M             0B         0B
ChrisNAS/.system/cores                                     1024M   128K        0B    128K             0B         0B
ChrisNAS/.system/rrd-229b43ea8de442e68f6896e2baff0028      4.88G  40.5M        0B   40.5M             0B         0B
ChrisNAS/.system/samba4                                    4.88G  11.4M     10.8M    680K             0B         0B
ChrisNAS/.system/services                                  4.88G   140K        0B    140K             0B         0B
ChrisNAS/.system/syslog-229b43ea8de442e68f6896e2baff0028   4.88G  11.6M        0B   11.6M             0B         0B
ChrisNAS/.system/webui                                     4.88G   128K        0B    128K             0B         0B
ChrisNAS/Media                                             4.88G  8.14T        0B   8.14T             0B         0B
ChrisNAS/Personal                                          4.88G  3.08G        0B   3.08G             0B         0B
ChrisNAS/iocage                                            4.88G  30.4G        0B   16.3M             0B      30.4G
ChrisNAS/iocage/download                                   4.88G  1.50G        0B    128K             0B      1.50G
ChrisNAS/iocage/download/11.2-RELEASE                      4.88G   272M        0B    272M             0B         0B
ChrisNAS/iocage/download/12.0-RELEASE                      4.88G   206M        0B    206M             0B         0B
ChrisNAS/iocage/download/12.2-RELEASE                      4.88G   402M        0B    402M             0B         0B
ChrisNAS/iocage/download/12.3-RELEASE                      4.88G   403M        0B    403M             0B         0B
ChrisNAS/iocage/download/12.4-RELEASE                      4.88G   252M        0B    252M             0B         0B
ChrisNAS/iocage/images                                     4.88G   128K        0B    128K             0B         0B
ChrisNAS/iocage/jails                                      4.88G  22.7G        0B    128K             0B      22.7G
ChrisNAS/iocage/jails/emby                                 4.88G  17.5G     3.14M    448K             0B      17.5G
ChrisNAS/iocage/jails/emby-server                          4.88G  5.21G      703K    482K             0B      5.21G
ChrisNAS/iocage/jails/emby-server/root                     4.88G  5.21G      546M   4.68G             0B         0B
ChrisNAS/iocage/jails/emby/root                            4.88G  17.5G     11.8G   5.68G             0B         0B
ChrisNAS/iocage/log                                        4.88G   169K        0B    169K             0B         0B
ChrisNAS/iocage/releases                                   4.88G  6.17G        0B    128K             0B      6.17G
ChrisNAS/iocage/releases/11.2-RELEASE                      4.88G  1.25G        0B    128K             0B      1.25G
ChrisNAS/iocage/releases/11.2-RELEASE/root                 4.88G  1.25G     17.5M   1.23G             0B         0B
ChrisNAS/iocage/releases/12.0-RELEASE                      4.88G   737M        0B    140K             0B       737M
ChrisNAS/iocage/releases/12.0-RELEASE/root                 4.88G   737M        0B    737M             0B         0B
ChrisNAS/iocage/releases/12.2-RELEASE                      4.88G  1.68G        0B    140K             0B      1.68G
ChrisNAS/iocage/releases/12.2-RELEASE/root                 4.88G  1.68G        0B   1.68G             0B         0B
ChrisNAS/iocage/releases/12.3-RELEASE                      4.88G  1.78G        0B    140K             0B      1.78G
ChrisNAS/iocage/releases/12.3-RELEASE/root                 4.88G  1.78G        0B   1.78G

I tried viewing the PDFs, but the broken formatting makes my eyes cross.

Can you post the output directly in here?

Code:

zfs list -t filesystem -r -o space ChrisNAS

My hunch is that you (or "something") is saving data directly inside the top-level root dataset. (Perhaps a rogue logfile? A misconfigured share?)

 

[winnielinnie]

When you refer to data, are you referring to my movies, pictures and the like?

I'm not sure. You'll have to browse it (command-line or file browser) to determine yourself.

Are you saying Media and Personal are in the top level dataset?

No. "ChrisNAS" is the top-level root dataset. The root dataset of any pool shares the same name as the pool itself. (This can cause confusion for new users of ZFS.) Nothing should live directly inside this dataset. It's only purpose (in TrueNAS) is a glorified placeholder for properties that children can inherit. You don't mess with its permissions or ACL. You don't store files inside it.

Below is the output of zfs list -t filesystem -r -o space ChrisNAS. Hopefully I got this thing right.

This clearly shows that your root dataset ("ChrisNAS") is using 2.23 TiB. The column "USEDDS" lists how much space is being consumed by the dataset itself, which is distinct from children and snapshots.

I also notice you have two jails named "emby" and "emby-server". The jail's dataset for "emby/root" is using up an unusual amount of space from its snapshots + the dataset itself. Were you saving media files in the jail's filesystem itself? This is also a "no no". You're meant to create mountpoints for the jail, which point to datasets that will hold the media, such as "Media", "Personal", etc.

 

[cgcmgr]

Regarding the 2.23 TiB. That's what I was trying to explain. There is a folder named root in ChrisNAS that I don't ever recall seeing, see the attachments. It contains a folder called emby-server and the total size of the root folder is 2,289GB. I never wrote to it. I'm not sure what happened, but I have a feeling that is the missing 2.25TiB. Maybe I'll just delete the emby-server folder within root and see what happens.

Yes, I have 2 Emby Jails and they are named emby and emby-server. I used to only have just emby, but I was never being prompted to update. After posting in the Emby Forum, I was told to run a command to to update it. All this did was create a new Jail named emby-server. I figured I'd leave it and configure it with my movies, etc. and use it a "backup" in case anything happens to the 1st Emby jail, and something did.

emby/root is actually the same size as emby-server/root.. However, there is an issue with emby. Out of the blue it is running extremely slowly. As a matter of fact, I cannot even watch a movie off of it any longer. When I access Emby, the movie pictures don't even all load anymore. I noticed this when I tried to update emby from 4.7.13.0 to 4.7.14.0. It would give me an error saying "Am error occurred while fetching package. Impossible to get the value from Last-Modified HTTP header." After that, the server started to have issues. I did try to update the jail to 12.4 from 12.3 multiple times and it continually failed. emby-server works just fine.

So based on the screen shot I included. You recommend that my mount points do not point to mnt/ChrisNAs/iocage/jails/emby/root/media? Should they point to themselves? mnt/ChrisNas/Media

I know it may look like I'm saving to the filesystem based on the used space, but the only access I had to my NAS prior to the 29th was:
1. Via WinSCP where I had it set as FTP and I could only read from the NAS, not write to it
2. The two mapped drives I have on my Windows machine that point to Media and Personal. This is how I add my movies and pictures, etc. to my NAS.

Not until the 29th did I set up WinCP to SSH into the NAS and that's when I started to have this space issue. I know you can't answer this question without more info, but in your opinion, does it look like the emby-server folder in the root folder of ChrisNAS is the cause of the lost space, especially since I have not purposefully saved anything to the root of ChrisNAS? Do you think I can delete it? After all, as you can see, my emby jails are in iocage not ChrisNAS/root. Maybe back it up and restore it if anything goes wrong after the deletion?

I have also included a capture of the size of the ChrisNAS/root folder.

 

[winnielinnie]

Maybe I'll just delete the emby-server folder within root and see what happens.

2.23 TiB smells like actual media, not simply a jail's base filesystem. (Otherwise, it would only be a handful of GiB's.) You could inadvertently destroy important media files that you accidentally copied to this folder (perhaps through the jail or share, somehow) if you just outright delete this folder.

I used to only have just emby, but I was never being prompted to update.

Sounds like you were using a "Plugin" and not a manually-created jail. Unfortunately, "Plugins" are no longer supported.

emby/root is actually the same size as emby-server/root..

emby/root total size is 18 GiB (USEDDS + USEDSNAP). It's odd (though not beyond plausibility) that this is due to jail upgrades. It's still large enough to warrant a safety check to make sure nothing important lives there.

So based on the screen shot I included. You recommend that my mount points do not point to mnt/ChrisNAs/iocage/jails/emby/root/media? Should they point to themselves?

No, the mountpoint appears correct. But something drastic happened on August 29, and there's a chance that important files (2+ TiB worth) live on your root dataset.


(I also sent you a DM.)

 

[cgcmgr]

@winnielinne - I really want to thank you for all the assistance with this, greatly appreciated.