iSCSI using twice the space...

[mattg889]

My system spec's are as follows:
Hp Proliant dl385 g7 server
64gigs of ram
24 amd cpu cores at 2.1ghz
32gb lexar flash drive for FreeNAS os
LSI hba controller for the drives
8-1tb drives set up in raid-z2
Smart data says the drives are all good
4 gigabit NICs setup in a load balance aggregation
FreeNAS 9.3

I want to set up a iCSI drive to hold my steam library. I set up a 1TiB iscsi drive through the wizard and when I was done setting it up the initial size of the zvol was about 300KiB, So I figured it would dynamically grow as I added files. So I started copying the 900GB of data to the iscsi drive. But it stopped near the end with a write error. I go to FreeNAS to check and FreeNAS says my volume is over 96% full. I had 2.3TiB of space available when I started all this. I should have enough room to make a 1TiB drive. I checked the zvol usage and it was using 1.7TiB, It doubled the size of the data....

So I deleted it and started over. I made another iSCSI drive manually following the FreeNAS guide. Now the initial zvol size was 1TiB. I then made a 768GB partition on that 1TB drive so that the drive will never be more than 75% full. (now I am referring to the iSCSI drive in windows, not the zfs volume). So I re-downloaded my steam library and windows says it is using 511GB of space on the drive. But now that initial 1TiB is at 1.1TiB in FreeNAS. Again it doubled the space used and it is growing beyond the 1TiB size.
I also made a 512GiB iSCSI drive for other games, and that one is having the same issue. Windows says the other games are using 335GB whereas FreeNAS is saying it is using 720GiB
I know GB and GiB are not the same, but 1GiB is not double 1GB..... What is going on. Doubling the space is not at all efficient.

Before everyone chimes in about how I shouldn't go over 80% usage on my zfs volume, I know, I have read it in over a dozen threads. If FreeNAS wouldn't be doubling the space that my data is using it wouldn't be going over 80% in the first place. The problem is not that the volume is over 80% full, the problem is the space the data is using is being doubled.
Even so, pure performance isn't my goal here, so if I am hovering around 80% usage it's not the end of the world. Again though, over 80% usage is not the problem.
Anyone have any idea why my data's space usage is being doubled?
I posted some screenshots if they help.

 

[mav@]

For the first, FreeNAS 9.3 is extremely ancient, update to something newer, like 11.1-U1 released not so long ago.

For the second, check ZVOL block size you are using to back your iSCSI disk. Your 8-wide RAIDZ2 can be space-efficient only when storing large blocks of data (at least 32KB or more), and your old FreeNAS 9.3 may not be clever enough to increase that block size automatically. Trying to store on that RAIDZ2 pool ZVOLs with default 8KB can quite likely result in double space usage.

 

[SweetAndLow]

1. Update to the latest version
2. Why bother with iSCSI, SMB probably will be better for your use case.
3. Zvol should never really be more than 60% of your pool size. Performance will decrease.
4. Double check your block size it night affect things.

 

[mattg889]

See this thread for the reason why I'm using freenas 9.3:
https://forums.freenas.org/index.php?threads/freenas-not-compressing-and-connection-issues.60330/

I want to use iSCSI because many games complain if you have your games stored on a network share, but with iSCSI the games think they are just on a local drive. It works perfect for running the games. I already tried hosting them on a network share, and I ran into many problems.
Again, the pool % used is not the problem, the 1tib iSCSI zvol is way less than 60% of the pool.

I left the block sizes at the defaults, what size should I specify then? 32k? larger?

 

[jgreco]

You shouldn't be using RAIDZ with block storage at all, because RAIDZ uses a variable amount of parity to protect your data depending on the block size, and this can cause excessive waste. It is totally possible to engineer a situation where your parity uses several times as much space as the data it is protecting. Performance will slowly degrade over time if your pool is more than 50% full, as well.

https://extranet.www.sol.net/files/freenas/fragmentation/delphix-small.png

Because you've got an 8-wide Z2, your optimal block size is six drives wide.

https://extranet.www.sol.net/files/freenas/fragmentation/RAIDZ-small.png

The only case where allocation happens optimally is where you have a block size of 6 * your sector size, or some multiple thereof. So if your ashift=9, then that's a 3072 byte block, or if ashift=12, then 24576 bytes. These are not natural sizes. You only get the natural sizes if you follow the old power rules for the number of disks in a pool (RAIDZ2 = 6 or 10 disks, etc).

So then as an example of how this goes awry, if your ashift=9 and you have a zvol volblocksize of 4K, you need to allocate an (optimal) 3072 byte block on eight sectors, but then you have a leftover of two 512-byte sectors, which needs to be protected by parity, so that's three sectors, but ZFS won't allocate odd numbers, so that's four sectors, so you're using twelve total sectors to write that 4K block, or 50% overhead. If your ashift=9 and a zvol of 8K, that's two optimal blocks on sixteen sectors with a leftover of four 512-byte sectors, plus parity/pad, six more sectors, so that's twenty two sectors to write that 8K block, or 38% overhead.

But it all goes to total heck with ashift=12 (4K sectors). The most obvious one is where you have a 4K zvol volblocksize and you store a 4K block. You need a 4K parity to protect that, so you're instantly at 100% overhead. If you do that on RAIDZ3, then you actually need 3 * 4K parity to protect it, so 300% overhead. So you go to larger volblocksize. So the 8K zvol volblocksize seems like it might work out better, but in reality you need two sectors for data, one for parity, and then one for pad, so you're still at the 100% overhead. You actually have to get up into larger volblocksizes and design them in concert with your pool in order to have a chance at a more optimal configuration. As you raise the volblocksize, you will approach a more optimal situation, but not actually reach it.

But even then, this is sort of screwed up because with the introduction of ZFS compression, things don't always work the way you might think. You can actually get a block compressed down by quite a bit, and that throws all this off. You're more likely to get compression with a larger block. So the generalized answer to your query is to use a larger volblocksize and turn on compression.

This isn't the trite problem that you'd like to think it is. And I've probably made some mistakes because I'm only at two coffees this morning.

 

[jgreco]

* Or to stop using RAIDZ which in my experience is a horrible choice for almost any significant iSCSI usage.

 

[mattg889]

OMG thank you jgreco, that has got to be the single most informative post I have seen so far on the FreeNAS forum. Now I know that I have FreeNAS configured completely wrong, lol. Sucks though because my server has 16 drive bays and all that means there is no way to properly use them all with raidz2. At any rate, looks like I will have to redo the raidz2 volume at some point with 6 drives. I do plan to add another volume in the future too.
One thing I don't quite understand though, my other server has 8 drives in a raid6 array on a dedicated controller and it hasn't had any problems at all for over 4 years now. It's running Ubuntu though, but still, the dedicated raid controller seems to handle 8 drives fine.

As for the iSCSI, for now I put a couple 1tb drives in the FreeNAS server and set them up striped. I don't need redundancy for the iSCSI drive. Although I would prefer it. Is there no way to properly put a iSCSI drive on a redundant array, other than a mirror...? Would a iSCSI drive work properly if I redo my raidz2 volume with 6 drives? Or is a mirror really the only way to make a iSCSI drive with redundancy? (I would do a mirror, but I don't have any bigger drives on hand)

 

[jgreco]

Well, it's totally possible to "figure it out" and get it to line up perfectly, or at least better, and that's what a lot of people did before compression. The problem got worse on an ashift=12 pools. Before that it was probably a pretty tepid affair to do a RAIDZ2 with 10 disks on ashift=9 so that your optimal block size worked out for 4K or 8K writes. But now there's compression and 4K sectors, and iSCSI favors 512 byte sectors, so there's a LOT of variables that play into how well this works out. With today's modern disks and compression, this is pretty much beyond mere mortals to figure out on their own, so while @mav@'s comment was technically a "sufficient" answer, it wasn't really that helpful as a solution.

The way it works with mirrors is that each block is mirrored on another disk, resulting in a reliable 100% overhead. This sucks too.

So I'm not sure what the solution for you is. For the intended use case, I'm tempted to say that I'd actually call "extremely light duty use" to be an exception to lots of aspects of this, so especially if your activities are mostly-read occasional-write, you can probably twiddle around with large zvol volblocksizes to see if one of them seems to get you a happier result. But the other thing I'll add, and you might groan a bit, is that the better fix is just to add some bigger drives, or replace them all with bigger drives, because the performance of ZFS tends to get better with more space free. You already have a monster base system there.

I apologize for the lack of happier magical fixes.