iSCSI target full w/ errors on scan drive

[Robert Marley]

config details
iSCSI target is FreeNas 3 phys 1TB drives, raidz into 1.95TB, mapped to a windows 8.1 client.
the client can see the disk, but clearly it's having issues with writes and access overall.
the crashplan archive I store on it won't progress in it's compact cycle and doesn't see it as being there most of the time.

From Win8.1 NTFS perspective the drive is 945GB full. 50% of total capacity, you'd think.
from the FreeNas side, log into scary alerts
CRITICAL: The capacity for the volume 'Raidz' is currently at 98%, while the recommended value is below 80%.
storage manager confirms Zvol is exactly 1.95TB and there are 0kb available. So FreeNas thinks it's really full, but status is 'healthy'

How do I clean up this mess without losing my data?
and what did I configure wrong?

 

[jgreco]

The problem here is that you've filled your disk to 98% capacity. The recommended value is not only "below 80%", but for iSCSI it's probably "below 60%", because block storage is one of the hardest things for ZFS to handle, and you need a significant amount of free space in order for ZFS to not suffer horrible fragmentation.

NTFS may be telling you that it is "50%" full, but the problem is that if you write blocks 1-100 with "file1", then blocks 101-200 with "file2", and then remove "file1", NTFS merely updates the directory and free block list to indicate that the space is free - but the data is still out there on the disk. ZFS has no way to know that that space is unused by NTFS. So all of blocks 1-200 still appear to have data. It is merely NTFS that thinks those blocks are "empty" because it has noted in its own little format that they are free.

Modern systems might use TRIM or other methods to advise the storage system that the space is actually free.

So, first, you probably want to turn on compression on the ZFS pool. Compression will probably help you sort out the problem if you do it smartly.

Next, you want to create, on your Windows box local drive, a 1MB file of zeroes - highly compressible data. See if you can copy that onto your iSCSI target. If so, this is probably salvageable.

If so, then create a much larger (1GB?) file of zeroes and do it again. You're overwriting blocks that used to have stale data with zeroes, which compresses on the ZFS side and frees space.

Keep going until you're healthier.

 

[Robert Marley]

With LZ4 already turned on, I had this idea. I had reduced the NTFS partition in Computer Management to only 1.4TB. Then in the remaining 400GB I attempted a Long (not quick) format of a new NTFS partition, hoping this would write zeroes, but for some reason it didn't work. I'll try copying zeroes out files instead and see if that fixes it. Win8.1 has full trim support, is it maybe just not turned on for iSCSI drives?

 

[jgreco]

Well, right sort of idea, but if it didn't work, it didn't work. If the files fail, try a disk erasing utility (eraser.heidi.ie, etc) being suuuuuuper careful of course.

 

[Robert Marley]

writing txt files full of zeroes is working, but there's nothing to prevent this from happening over and over again. Presumably every time crashplan does a heavy write cycle to this volume while pruning or compacting it's going to keep filling up. What could I do to fix it permanently from this point, now that my data is probably recoverable.

 

[jgreco]

Well, it's not that difficult.

You cannot use the entire space on a ZFS pool. In normal usage, you shouldn't exceed 80%. But for iSCSI, that number is lower. Maybe 60%. For a heavy write environment, though, actually lower. Maybe a lot lower. Fragmentation effects hit hard by about 50% full. You won't run into a catastrophe such as what you experienced, but it will be SLOW. Not right away, but over time, as countless write cycles scatter disk blocks around, trying valiantly to find free space. The more blocks of contiguous free space you can give it, the better it works. And the only reliable way to do that on an ongoing basis is to give it massive amounts of free space to work with.

Take a look at figure 4 in that link. A pool that is 10% full will be about 15-20x faster than a pool that's 80% full. That's substantial. Remember, that isn't true right away ... for the first write cycle, that pool is just as fast at 10% as it is at 80%. But over time, with dozens or hundreds of write cycles, it'll degrade.

So you make sure your pool has lots of free space. Replace those 1TB drives with 4TB drives, making an 8TB usable space pool, and continue to only use 2TB of it. The thing'll fly and you won't have this sort of issue again.

But also be aware that we don't like RAIDZ1, as a single failure removes redundancy from your pool. You could get 4 x 4TB drives in RAIDZ2, which is a pretty good configuration that balances practicality and availability. I realize that the price to accomplish this makes for a very expensive 2TB of storage, and I don't actually expect you to do it, but that's a fix that'd truly fix your issue AND keep performance at a good level.

 

[Robert Marley]

what if performance doesn't matter at all, since Crashplan's crappy Java platform will always be the bottleneck? this device will only ever receive writes at 100mbit/s tops. Is there any kind of scrub or maintenance cycle that will force a cleanup, either from the Windows client side, from the FreeNas side? Should I use something other than ZFS? other than FreeNas? I can always move this iSCSI service back to my Synology, which never had these problems.

 

[jgreco]

If performance really doesn't matter, you can probably dare it with 80% utilization. ZFS is a complex and featureful system, but if you do not need the features, then it is kind of like picking up an 18-wheeler to do your grocery shopping... it'll work but may not be ideal.

ZFS lacks any means to defragment, because it isn't even clear (from the ZFS PoV) what that means, given snapshots and variable blocksizes and whatnot.

The usual fix is to give ZFS what it needs, which is typically resources and legroom, and then it is totally awesome. ZFS is usually poor at using every last bit of its resources at 100% efficiency, though, which is what you basically were expecting out of the setup you had.

No one here will claim (I hope) that ZFS is right for every use case.

 

[mav@]

There were reports about Windows not using UNMAP for volumes upgraded from pre-9.3 FreeNAS, that had no UNMAP support. Offloading the data and recreating volume/filesystem from scratch on FreeNAS 9.3 in that case made Windows to use UNMAP, freeing pool space on file deletion.

 

[Robert Marley]

weird, this has only ever been FreeNas 9.3. I've never upgraded. The whole system was only configured a month or so ago.
is there any other way I can check on this, or jump start things?

 

[mav@]

I don't know way to check what Windows really thinks about UNMAP support, but you may check what Windows's defrag tool thinks about it. It has special parameter key (IICR it is /L) to forcefully TRIM/UNMAP all unused space.

Just as possibility: some advanced storage features in Windows, such as UNMAP and ODX, may be blocked by third-party drivers/tools installed, such as anti-virus, virtualization drivers, etc.