Massive Files (TB) that have small size on disk

[soapee01]

I'm not sure if this is a FreeNAS (11.2-U8) issue or a QuickBooks issue, but I've got some files that show up as 3.25TB in windows (and ls on freenas) but size on disk shows 8MB in windows. ZFS does not show that much storage space used.

I'm using rsync to copy this across to another backup location, and it tries to copy the full 3.25TB.

Has anyone seen this before and know what's going on?

 

[winnielinnie]

COMPRESSION, BABY! That's what I'm talking about! YES! @soapee01 over here breaking world records!

Actually, to be honest, that is a bit insane. No, really insane.

First of all, are QuickBooks files known to balloon to such ridiculous sizes? I'd expect them to be megabytes in size.

Secondly, you confirmed using SSH / Shell within your FreeNAS server that this file is really 3.25 TB in size?

Finally, are you using rsync over the SMB share? or directly from FreeNAS itself?

 

[soapee01]

Code:

ls -alh "/mnt/pool1/foo.qbw"
-rwxrwxr-x+ 1 contoso\user contoso\domain users   3.3T Mar 17 13:45 /mnt/pool1/foo.qbw

du -h "/mnt/pool1/foo.qbw"
7.7M    /mnt/pool1/foo.qbw

and yeah, was copying via ssh with rysnc from linux. I had to adjust my script to exclude max-file-size=10g to get this thing to finish.

really it's rsnapshot on the linux box (rsync under the hood) using ssh to freenas to copy the files.

 

[winnielinnie]

ls -alh "/mnt/pool1/foo.qbw" -rwxrwxr-x+ 1 contoso\user contoso\domain users 3.3T Mar 17 13:45 /mnt/pool1/foo.qbw

This is the part that blows my mind. That is indeed a 3.3 TERABYTE file.

I'm not sure if this is a FreeNAS bug, let alone a ZFS issue, as perhaps it could have happened on any filesystem.

I'm curious to know if it's a normal QuickBooks file (several of megabytes in size) that somehow got padded with zeros for the remaining 3.3 TB?

If you do a hex dump of the last chunk of the file, maybe the last 4 KB (and maybe even a random section somewhere in the middle of the 3.3 TB madness), does it reveal it's nothing but zeros?

For example,

Code:

tail -c 4096 /mnt/pool1/foo.qbw | hexdump -C

EDIT: You don't have to paste the output in here! It'll be very obvious to you if the output shows nothing but zeros.

 

[Samuel Tai]

Copying large files with Rsync, and some misconceptions - Fedora Magazine

There is a notion that a lot of people working in the IT industry often copy and paste from internet howtos. We all do it, and the copy-and-paste itself is not a problem. The problem is when we run things without understanding them. Some years ago, a friend who used to work on my team […]

fedoramagazine.org

Try modifying the script to use rsync --sparse.

 

[winnielinnie]

Copying large files with Rsync, and some misconceptions - Fedora Magazine

There is a notion that a lot of people working in the IT industry often copy and paste from internet howtos. We all do it, and the copy-and-paste itself is not a problem. The problem is when we run things without understanding them. Some years ago, a friend who used to work on my team […]

fedoramagazine.org

Try modifying the script to use rsync --sparse.

@Samuel Tai but the real concern is that there is an application (supposedly QuickBooks) that is writing a single 3.3 TB file. Or perhaps malware trying to crush disk usage with infinite "zeros", but is thwarted due to ZFS's inline compression? No application nor malware would be privy to the fact that the folder/share (as is seen in Windows) rests atop a ZFS dataset.

In other words, if @soapee01 was not using ZFS, and simply saved their QuickBooks file locally (without inline compression), then they would indeed be stuck with a massive 3.3 TB .qbw file on their disk.

 

[soapee01]

The complaint I've been getting is explorer.exe crashing on user computers, sometimes while saving qb files, and other times saving pdf files to the samba shares.

I'm getting another backup in place to upgrade in hopes that it would fix the crashing issues, but this may actually explain it. It's odd to be sure.

And yeah, the hexdump is all 0's.

 

[winnielinnie]

And yeah, the hexdump is all 0's.

I had a hunch. Something is causing an incessant stream of zeros to write to the file ad infinitum. ZFS's inline compression prevented your pool from filling up rapidly.

I'm getting a backup in place to upgrade in hopes that it would fix the crashing issues, but this may actually explain it. It's odd to be sure.

Yeah no kidding.

But wonder why the clients accessing the SMB share experienced explorer.exe crashing? Different computers, same crash?

 

[soapee01]

I forgot to add QB files are usually in 10's to 100s of MBs, but I have seen plenty in multiple GBs when people add scanned invoices, etc and attache them to payments. Now I wonder how many of these files there may be, I may need to just use find on the system and see what pops up, but I'll wait for the rsync to finish up.

 

[soapee01]

But wonder why the clients accessing the SMB share experienced explorer.exe crashing? Different computers, same crash?

Yeah, different computers, different files, different times/days etc. I don't get good reports as to when it actually happens, or even which files so I can try and correlate anything to logs unfortunately.

 

[winnielinnie]

I forgot to add QB files are usually in 10's to 100s of MBs, but I have seen plenty in multiple GBs when people add scanned invoices, etc and attache them to payments.

For sure, but... 3.3 terabytes?

 

[soapee01]

For sure, but... 3.3 terabytes?

Yeah, nope.

I'll add, I'm pretty sure this isn't the first time I've seen this happen. What's new is explorer.exe crashing. I seem to have run across some TB sized qbw files at the same client a couple of years back (same freenas), and just thought, that's odd and restored from snapshot.

It could just as easily be trying to OPEN a 3TB file that is crashing explorer. At least I've got a place to look now. I'll have to do some testing tomorrow.

 

[sretalla]

Your compression ratio must be large on that pool....

 

[winnielinnie]

Your compression ratio must be large on that pool....

Perhaps this is an elaborate scheme by @soapee01 to pitch a new late-night televised ZFS commercial? We know what you're up to!

SCRIPT Rev. 1.0a​

(depressing black-and-white video of a person covering their tears sitting at a computer desk)

NARRATOR: "Are you tired of always running out of disk space because of those pesky terabyte QuickBooks files?"

(person slams their head on the desk)

NARRATOR: "Annoyed at all those trips to the store to buy more external drives to save your terabyte Excel spreadsheets?"

(person gets tangled up in numerous USB cables)

NARRATOR: "Hate it when your Word documents require several harddrives to fully save?"

(person throws their laptop out the window)

NARRATOR: "Well those days are over..."

(video transitions from black-and-white into vibrant colors)

NARRATOR: "...thanks to the breakthrough technology of Zee Eff Ess!"