Greater than 80% volume capacity: OK for large static storage?

[jgreco]

The top one is the pool itself.

 

[Robert Trevellyan]

But I am confused as to why the main ZFS volume is listed twice in the GUI as what appears to be a subset of itself, with two vastly different values for Used/Available space.

The top one is the volume, the one immediately inside is the all inclusive dataset that recent FreeNAS versions now display. The capacity numbers on the volume are raw storage before overhead, and the capacity numbers on the all inclusive dataset take overhead into account.

 

[RedBear]

The top one is the pool itself.

The top one is the volume, the one immediately inside is the all inclusive dataset that recent FreeNAS versions now display. The capacity numbers on the volume are raw storage before overhead, and the capacity numbers on the all inclusive dataset take overhead into account.

Thanks for trying, but nothing about the way this information is presented is making any sense to me so far. The screenshot in the FreeNAS guide makes even less sense. I have no idea how you fit 22.8GiB inside of 2.2GiB and why the "all inclusive dataset" (just like on my screenshots) has less than half the apparent available capacity of the parent volume:

But all of this is off-topic in this thread and I have other questions about why both the total "size" being reported and the apparent available space are wildly different from the estimates I got from an online RAID calculator for a RAID-Z3 array of 8x120GB disks, so I should probably start a new thread specifically dedicated to getting a better handle on all this.

 

[Robert Trevellyan]

In your earlier screenshots, the numbers basically make sense. Raw capacity for 8 120GB disks is naively 960GB, which is close enough to 390+482 when you realize that GiB != GB.

Then the pool capacity basically makes sense when you realize you've given up 3 drives-worth of space to RAIDZ3 overhead, leaving 600GB or about 585GiB, less the additional filesystem overhead.

But like you, I can't make sense of the latest screenshot you posted.

 

[RedBear]

In your earlier screenshots, the numbers basically make sense. Raw capacity for 8 120GB disks is naively 960GB, which is close enough to 390+482 when you realize that GiB != GB.

Then the pool capacity basically makes sense when you realize you've given up 3 drives-worth of space to RAIDZ3 overhead, leaving 600GB or about 585GiB, less the additional filesystem overhead.

But like you, I can't make sense of the latest screenshot you posted.

OHHHHH, so when it says the "size" of my pool is "872G", that's actually gibibytes, not gigabytes? How incredibly obnoxious. But at least it makes much more sense that I'm only losing 22GiB (894 minus 872) to filesystem overhead to begin with, rather than 88GB. That's only like 2.5% instead of 10% capacity loss for formatting. I'm cool with that.

But seriously, that's extremely obnoxious, that only the letters "T", "G", "M" or "K" are being used in place of proper units. So I guess I have to assume that the "M" is mebibytes and the "K" is kibibytes also, and when I do go to larger disks and see a "T" it will stand for tebibytes. 'Cause if there were any unit mixing going on I'd hafta choke a bee-yotch.

I can't possibly be the only uninformed user that automatically assumed that if anyone were to shorten units down to a single letter they would "obviously" be referring to the binary units that have been in use for decades and are still the primary units used to sell storage media, not these newfangled base-10 units that no one outside the geek world has ever heard of. I am very much not in favor of shortening units down to the point where they become ambiguous. Not cool. I don't get how anyone finds ambiguous units acceptable.

But on the bright side, after a lot of fumbling around with Spotlight unit conversions and a calculator, you've totally transformed my understanding of what I'm seeing. Except 234GiB + 175GiB = 409GiB = 440GB which still falls pretty far short of the 559GB that the ServeTheHome RAID calculator says should be the "usable storage" on this array. Yes, I'm giving up three out of eight disks for parity, I get that, and there will always be a bit more loss for the actual filesystem formatting, but I seem to be losing more than 50% of the raw storage capacity in the end. There's another 160GB (150GiB) missing, that's more than an entire additional drive of space lost. Something about that doesn't jibe. If there's really that much additional "filesystem overhead" with RAID-Z3 I'll definitely have to go to an 11-disk array or this whole venture will simply be too expensive per gig^H^H^H gibibtye. Or, I'll have to live with 8 disks and RAID-Z2.

Thanks for the big push on the road to ZFS enlightenment. Still a long way to go.

 

[Robert Trevellyan]

But seriously, that's extremely obnoxious, that only the letters "T", "G", "M" or "K" are being used in place of proper units.

Where are you seeing this?
EDIT: you mean with the command-line ZFS tools?

I can't possibly be the only uninformed user that automatically assumed that if anyone were to shorten units down to a single letter they would "obviously" be referring to the binary units that have been in use for decades and are still the primary units used to sell storage media, not these newfangled base-10 units that no one outside the geek world has ever heard of. I am very much not in favor of shortening units down to the point where they become ambiguous. Not cool. I don't get how anyone finds ambiguous units acceptable.

Strictly speaking, drive manufacturers have always used base-10. Some others have habitually used GB incorrectly to mean base-2, which is why disk sizes always look lower in Windows than they do in hard drive sales literature. Of course, we all tend to use GB incorrectly when referring to RAM, which has always been sized in base-2, and so should really be counted in GiB. In some languages (e.g. French) there's no confusion because they use clearly different abbreviations. At least, that's my understanding of all this. FreeNAS only recently started using the binary units.

As for all the space you're 'losing', I think you would probably find that the percentage overhead would be lower with a larger pool.

 

[Ericloewe]

OHHHHH, so when it says the "size" of my pool is "872G", that's actually gibibytes, not gigabytes? How incredibly obnoxious. But at least it makes much more sense that I'm only losing 22GiB (894 minus 872) to filesystem overhead to begin with, rather than 88GB. That's only like 2.5% instead of 10% capacity loss for formatting. I'm cool with that.

But seriously, that's extremely obnoxious, that only the letters "T", "G", "M" or "K" are being used in place of proper units. So I guess I have to assume that the "M" is mebibytes and the "K" is kibibytes also, and when I do go to larger disks and see a "T" it will stand for tebibytes. 'Cause if there were any unit mixing going on I'd hafta choke a bee-yotch.

I can't possibly be the only uninformed user that automatically assumed that if anyone were to shorten units down to a single letter they would "obviously" be referring to the binary units that have been in use for decades and are still the primary units used to sell storage media, not these newfangled base-10 units that no one outside the geek world has ever heard of. I am very much not in favor of shortening units down to the point where they become ambiguous. Not cool. I don't get how anyone finds ambiguous units acceptable.

But on the bright side, after a lot of fumbling around with Spotlight unit conversions and a calculator, you've totally transformed my understanding of what I'm seeing. Except 234GiB + 175GiB = 409GiB = 440GB which still falls pretty far short of the 559GB that the ServeTheHome RAID calculator says should be the "usable storage" on this array. Yes, I'm giving up three out of eight disks for parity, I get that, and there will always be a bit more loss for the actual filesystem formatting, but I seem to be losing more than 50% of the raw storage capacity in the end. There's another 160GB (150GiB) missing, that's more than an entire additional drive of space lost. Something about that doesn't jibe. If there's really that much additional "filesystem overhead" with RAID-Z3 I'll definitely have to go to an 11-disk array or this whole venture will simply be too expensive per gig^H^H^H gibibtye. Or, I'll have to live with 8 disks and RAID-Z2.

Thanks for the big push on the road to ZFS enlightenment. Still a long way to go.

FreeNAS 10 will let you select your favorite units.

 

[DifferentStrokes]

FreeNAS 10 will let you select your favorite units.

Awesome!

 

[jgreco]

FreeNAS 10 will let you select your favorite units.

Wow, I can get my pool size reported in kiloquads?!?!? Awesome!!!!

 

[rogerh]

As for all the space you're 'losing', I think you would probably find that the percentage overhead would be lower with a larger pool.

With 120GB disks the 2GB swap partitions are significant, whatever kind of GB the latter are.

 

[Ericloewe]

Wow, I can get my pool size reported in kiloquads?!?!? Awesome!!!!

You'll have to convince jkh to allow for arbitrary units.

Is a quad 32 bits or a nibble?

 

[jgreco]

You'll have to convince jkh to allow for arbitrary units.

Is a quad 32 bits or a nibble?

It's a great question, because it could also be a single digit. ISTR there's talk in ST:TOS of trinary computing systems, so some people have suggested that "kiloquad" is an analog to "kilobit", except using a four value digital signal.

The problem comes in when you identify that an isolinear optical chip is spec'ced as having a total memory capacity of 2.15 kiloquads per chip, but they talk about stuff like "single-axis optical crystal layering to achieve subwavelength switching distances," so also when you consider that a contemporary SD card can be had in 512GB variants, it seems like an isolinear chip is supposed to hold a lot of data. Also, the Enterprise computer core is rated to access modular arrays of 144 isolinear chips at a speed of 4,600 kiloquads/sec. Since a contemporary computing system can manage tens of GB/sec to its memory, we have to assume a kiloquad must be a very large unit of storage...

 

[Robert Trevellyan]

With 120GB disks the 2GB swap partitions are significant, whatever kind of GB the latter are.

Yes, but for 8 disks, still only about 1/10th of what RedBear sees as 'missing'.

 

[RedBear]

Where are you seeing this?
EDIT: you mean with the command-line ZFS tools?

Strictly speaking, drive manufacturers have always used base-10. Some others have habitually used GB incorrectly to mean base-2, which is why disk sizes always look lower in Windows than they do in hard drive sales literature. Of course, we all tend to use GB incorrectly when referring to RAM, which has always been sized in base-2, and so should really be counted in GiB. In some languages (e.g. French) there's no confusion because they use clearly different abbreviations. At least, that's my understanding of all this. FreeNAS only recently started using the binary units.

As for all the space you're 'losing', I think you would probably find that the percentage overhead would be lower with a larger pool.

Yes, with the command-line tools. Nothing but T/G/M/K. How is the user supposed to know what actual units those letters are referring to? The "G" could stand for Gumboferbsniddles for all I could tell before this. With larger drives I'd have Turboferbsniddles!

OK, now this is hilarious, or possibly very sad. I was going to tell a cute little funny story earlier about how, when I first heard of tebi/gibi/mebi/kibi many years ago, I was "confused" into thinking that these new units were supposed to be referring to base-2 because they contained the "bi" from "binary", and the regular tera/giga/mega/kilo would be redefined as base-10 units. Turns out I was somewhat less confused about this issue those many years ago than I am now! I guess nobody noticed that I was referring to tera/giga/mega/kilo as the base-2 units, and tebi/gibi/mebi/kibi as base-10. Derp.

[* insert a kibibyte of facepalm meme images here *]

 

[jgreco]

Well, I'll just once again offer a hearty F U to the HDD marketing arses who switched units but kept using the same labels mid-stream.

 

[RedBear]

Yes, but for 8 disks, still only about 1/10th of what RedBear sees as 'missing'.

Mystery solved, I think. You're all going to find this quite amusing, I'm sure. I kept looking around trying to figure out where one goes to verify the Z-level of a pool, then remembered I'd seen raidz-something in the output of zpool status:

Code:

$ zpool status
  pool: TestVolume1
state: ONLINE
  scan: scrub repaired 0 in 0h49m with 0 errors on Mon Jun  1 04:34:27 2015
config:

    NAME                                                STATE     READ WRITE CKSUM
    TestVolume1                                         ONLINE       0     0     0
      raidz2-0                                          ONLINE       0     0     0
        gptid/6fa5f02c-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
        gptid/70dacffd-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
        gptid/7229b9aa-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
        gptid/7360c7f6-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
      raidz2-1                                          ONLINE       0     0     0
        gptid/75bf9b8e-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
        gptid/770560dc-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
        gptid/785205cf-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0
        gptid/798929be-0032-11e5-a439-6c0b8409b367.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Jun  2 04:47:44 2015
config:

    NAME                                          STATE     READ WRITE CKSUM
    freenas-boot                                  ONLINE       0     0     0
      gptid/8c5d2aa0-fb71-11e4-a61f-001517b066a2  ONLINE       0     0     0

errors: No known data errors

All you ZFS experts should immediately notice that I do not in fact have a RAID-Z3 array of eight disks. Instead what I seem to have here is a striped set composed of two separate RAID-Z2 arrays of four disks each. It now makes perfect sense that I'm "losing" more than four entire drives-worth of space from the total. Needless to say I got very confused by the Volume Manager web GUI when doing my initial test setup. It's a good thing I'm playing in a disposable sandbox of tiny, cheap drives, because I am getting poop EVERYWHERE so far.

I destroyed the pool and figured out what I had done wrong the first time*, then created an actual 8-disk RAID-Z3 and wound up with 482GiB of space. That plus the 2 (GiB?) of swap per disk gets a lot closer to the 559GB of theoretical usable space I get from the RAID calculator.

That Volume Manager GUI really needs some work. The manual version of the dialog is actually much easier to understand.

* If anyone's interested, what happened was that I was trying to "add" the drives so I clicked the big plus button in the middle of the dialog, which changed this:

... into this:

... and made the RAID-Z3 option disappear. Which is why I wound up with two RAID-Z2 arrays, not RAID-Z3. The resulting volume layout shown here actually made perfect logical sense to me at the time because I was thinking of the physical layout of the drives. They're installed in two separate 4-disk backplanes and connected to two separate SFF-8087 ports on the M1015 card. I now realize that it doesn't actually make any sense that FreeNAS would know anything about that or be even attempting to show it graphically.

I remain confused about the exact purpose of the big plus button and why I could only click it once before it grayed out even though there are 8 drives, but at least I finally figured out how to set up and identify an 8-disk array correctly. Now if someone were to be gracious enough as to congratulate me, totally undeservedly, for figuring all this out on my own, I could feel slightly less dumb. Wink.

 

[jgreco]

No, seriously, congratulations.

The devs have struggled for a long time with how to make an understandable and accessible GUI for FreeNAS, and I think it is inherently very difficult to do something like this. It also shows the wisdom of taking some time to experiment and understand what you're doing before you just blindly start using the thing, which is, sadly, not exactly the idea behind an appliance device.

 

[Robert Trevellyan]

It's a good thing I'm playing in a disposable sandbox of tiny, cheap drives, because I am getting poop EVERYWHERE so far.

:D

I remain confused about the exact purpose of the big plus button and why I could only click it once before it grayed out even though there are 8 drives, but at least I finally figured out how to set up and identify an 8-disk array correctly.

If you had, say, 3 120GB drives and 5 250GB drives, there would be 2 plus signs. One would invite you to begin the process of creating a 3-disk RAIDZ1 volume and the other would invite you to begin the process of creating a 5-disk RAIDZ2 volume. And if you clicked both, you would get a volume made of the 2 vdevs I just described. The idea would be that since ZFS behave as though all the drives in a vdev are the size, you would prefer to use them that way for efficiency.

 

[RedBear]

:D

If you had, say, 3 120GB drives and 5 250GB drives, there would be 2 plus signs. One would invite you to begin the process of creating a 3-disk RAIDZ1 volume and the other would invite you to begin the process of creating a 5-disk RAIDZ2 volume. And if you clicked both, you would get a volume made of the 2 vdevs I just described. The idea would be that since ZFS behave as though all the drives in a vdev are the size, you would prefer to use them that way for efficiency.

You confirm the wording in the guide that seems to imply the GUI volume manager is designed to try to help you create an "optimal" pool. Yet even though my eight drives are identical sizes, and listed together next to a single plus button, the GUI splits them into two (vdevs?) of four drives each. And once you click the plus button there is no way to undo what it does except to cancel out of the dialog.

Meanwhile if I just grab the slider and drag right I end up creating an 8-disk array.

 

[RedBear]

No, seriously, congratulations. :)

The devs have struggled for a long time with how to make an understandable and accessible GUI for FreeNAS, and I think it is inherently very difficult to do something like this. It also shows the wisdom of taking some time to experiment and understand what you're doing before you just blindly start using the thing, which is, sadly, not exactly the idea behind an appliance device.

Thanks, I knew the resident grinch would have a kind word.

It is definitely a monumental task to make such a powerful tool both easy and safe to use.