ZFS RAIDZ2 capacity question

[uid3003]

Hi All!

Have a basic capacity question. I've done a little research, and it just isn't adding up for me. Please go easy on me if this has been covered before.

I've got a cabinet with 12*2TB disks. I created a RAIDZ2 volume, which reports as having 21.7TiB available in the volume listing in the WebUI.

When I go to create a zVol, I am only able to use ~16TiB worth. Why? I know the math for ZAIDZ2 works out to something like 18.6TB after drive loss and formatting, so where's my extra 2.6TB?

12 * 1863GB = 22356GB - (1863GB *2) = 18630GB what did I miss?

Screenshot attached just for the hell of it!

Thanks everyone!

 

[uid3003]

Hi all,

Also, just to add a little info, when creating the volume, it estimates the capacity as 18.17TiB, which by my calculations should be ~20TB.

Thanks!

 

[Bidule0hm]

In fact it's 18.19 TiB for the data (and 3.638 TiB for parity) but this doesn't include the ZFS overheads (metadata, blocks, etc...).

 

[uid3003]

Hi, Thanks for responding!

So are you saying ZFS overhead is chewing up 2.6TB out of a 20TB raid set? That's ALOT of overhead. I totally understand the parity usage, but I'm not understanding where the rest has gone.

Thanks!

 

[Bidule0hm]

Yep, I'm a bit surprised too. Not sure it's 100 % overhead, maybe there's something else going on. But as it's a new volume I don't think you have things like snapshots and co that can take some space.

 

[SweetAndLow]

Random suggestion: you shouldn't make your zvol more than ~60% of your pool size. You will have very poor performance if you fill things too much.

 

[uid3003]

I thought that was only when using iSCSI? (Which I am, but regardless...)

Is it only performance I will lose by using more than 50-60%, or could I actually face data loss?

The more and more I read about this, the less practical FreeNAS and ZFS seems as a production data store. I can apparently only utilize less than half of the available space. What good is this? It's a fantastic waste of resources, especially when we were looking at FreeNAS to save money. I'm going to end up needing more RAM and disk space to get FreeNAS to work well than if I just deploy Linux or even a Windows storage server for that matter. Even with the licensing cost of Windows, I'd lose less resources, time and money to deploy a fully functional data store. I'm thinking of bailing for plain old Linux with LVM, unless someone can pursued me to stay for some reason. There just doesn't seem to be much benefit to FreeNAS after all research. It seemed promising at first, but just has far too many draw backs. I sure hope TrueNAS doesn't face these same restrictions.

 

[SweetAndLow]

I thought that was only when using iSCSI? (Which I am, but regardless...)

Is it only performance I will lose by using more than 50-60%, or could I actually face data loss?

The more and more I read about this, the less practical FreeNAS and ZFS seems as a production data store. I can apparently only utilize less than half of the available space. What good is this? It's a fantastic waste of resources, especially when we were looking at FreeNAS to save money. I'm going to end up needing more RAM and disk space to get FreeNAS to work well than if I just deploy Linux or even a Windows storage server for that matter. Even with the licensing cost of Windows, I'd lose less resources, time and money to deploy a fully functional data store. I'm thinking of bailing for plain old Linux with LVM, unless someone can pursued me to stay for some reason. There just doesn't seem to be much benefit to FreeNAS after all research. It seemed promising at first, but just has far too many draw backs. I sure hope TrueNAS doesn't face these same restrictions.

Just depends on what you find valuable in the product. Your Linux deployment won't have a GUI to manage your storage and share and it also won't have the self correcting filesystem. FreeNas never says it a cheap solution people just assume it is, it's basically the same thing as truenas but truenas sells the hardware bundled with the software.

If I was you I just wouldn't use iscsi unless you have to. It's a dying technology and has a limited use case. If you want to use it and don't care about performed you can make it up to 80 or 90% of your pool but don't expect 100MB/s throughout. If you go with those other solutions I have a feeling the same thing will happen with your performance. You still need more memory and you still need more disk to prevent fragmentation.

Good luck with your choice and post back with your findings if you poc something else.

 

[mjws00]

I think you may have wrote zvol and meant pool.

With iscsi the 60% fragmentation/performance issue pertains to all CoW filesystems. It is a design trade off and simply the nature of the beast. Every OS suffers exactly the same.

 

[uid3003]

Well put SweetAndLow. Not sure why I or anyone else assume it will be a cheaper solution, perhaps because it is free and open source, that is the illusion. While Linux won't have a GUI, for my purpose, this isn't a turn off because once I deploy these data stores and set the snapshot schedule, I expect to never touch it again unless I need to recall a snapshot, which I don't mind getting dirty in the console to do.

Curious though, why do you say iSCSI is a dying technology? I'm a data center engineer, and I believe quite the contrary. It is a fantastic technology that seems to be really taking off lately, and has so many benefits over NFS, Fiber Channel or even SAS. It's really the basis for many, if not most OpenStack and CloudStack implementations on the storage side. I don't think the use case in my situation (medium-large data centers) is as limited as you may think.

Now to dev test FreeNAS vs Debian w/ LVM vs CentOS w/ LVM!

 

[pschatz100]

I'm not an expert on large disk arrays, but much of the information I read suggests that the optimum number of data disks in an array would be a power of two - which in your case would mean 8 data disks. With RaidZ2, that would equate to a 10 disk array (8 data + 2 parity) and with RaidZ3 that would equate to an 11 disk array (8 data + 3 parity.) I've also read that using more than 11 disks in a single ZFS array is not recommended.

Could this have a bearing on your problem?

 

[gpsguy]

For iSCSI, striped mirrors are preferred and you'll want plenty of RAM.

I thought that was only when using iSCSI? (Which I am, but regardless...)

Compare the total cost with some of the commercial vendors. Especially, since you can run FreeNAS on commodity hardware.

... especially when we were looking at FreeNAS to save money