SOLVED Probably dumb 'available space' question

[ChrisRM]

This is based on a 4x4Tb Z2 array that I'm using. I've just finished expanding it from a 4x1Tb array by slowly replacing each drive with a bigger one, and the available space looks odd. Resilvering is complete, and it looks like this:

I can account for the actual space used, no worries about that but... How come the space goes from 16 to 12.1(explicable), from 12.1 to 5.6(inexplicable)? I can account for the jails and Time Machine backups, but where's the rest gone?

puzzled...

 

[Alecmascot]

This has come up at least 3 times in the last few months.
It is also in the users guide as I recall.........

 

[ChrisRM]

I used
'8.1.11. Replacing Drives to Grow a ZFS Pool' to guide me in upgrading the array but there's nothing that I can find that talks about using only a given percentage for any particular pool or device...

 

[danb35]

Really makes you wonder why they bother writing the manual:

Note that in this example, there are two datasets named volume1. The first represents the ZFS pool and its Used and Available entries reflect the total size of the pool, including disk parity. The second represents the implicit or root dataset and its Used and Available entries indicate the amount of disk space available for storage.

 

[ChrisRM]

Really makes you wonder why they bother writing the manual:

Agreed. Married to a tech author, I know just that feeling. That's why I put in the post that the 16Tb to 12Tb reduction is explicable, and yet the 50% reduction *after* indexing overheads is inexplicable. The manual, for whatever reason, doesn't answer the 'why' question...

 

[danb35]

The manual, for whatever reason, doesn't answer the 'why' question...

The manual answers precisely that question, in the section I quoted.

Edit: And your system's been doing this since you created the pool--it didn't start when you expanded the pool by replacing the disks.

 

[ChrisRM]

The manual answers precisely that question, in the section I quoted.

I think my problem here is the way I phrased the question. In an array with 16Tb to start with, I then get 12.1Tb available, presumably after the Z2 filesystem parity is taken off. I then have 2.5Tb 'used' out of that. Where does the root dataset get reduced from (12.1 - 2.5) to 5.6?

The way the manual excerpt is written ('The first represents the ZFS pool and its Used and Available entries reflect the total size of the pool, including disk parity.') suggests that parity has already been taken off the available space, explaining the drop from 16 to 12. The second line doesn't explain that one can lose >50% of the pool space for <something>; what's that <something>? There's no quotas set (set to '0' throughout), there's no reserved space set (again, '0'), and I can't find in the 11.1 reference manual or Fester's guide anything that explains this drop. It's always possible I've *really* misunderstood the language used but, if so, I'd be grateful for the correction...

 

[danb35]

In an array with 16Tb to start with, I then get 12.1Tb available, presumably after the Z2 filesystem parity is taken off.

No. In an array of 16 TB (14.6 TiB) total, you have 2.5 TiB + 12.1 TiB = 14.6 TiB in total capacity, as shown on the first line. "Available" means "free" in this case. The second line shows the space net of parity. RAIDZ2 uses two disks' worth of parity, so your 4 x 4 TB disks in RAIDZ2 leaves 2 x 4 TB, or 8 TB, of capacity net of parity. That's 7.2 TiB. Your pool is showing 1.2 TiB used, 5.6 TiB available, for a total of 6.8 TiB. The remainder is filesystem overhead, reserved space, etc.

 

[cobrakiller58]

@danb35 you beat me to the punch.

 

[ChrisRM]

Bingo! That's the missing stuff, and many thanks for your forbearance. :)

<thinks> Maybe the N54L won't be big enough...

Chris M

 

[cobrakiller58]

Probably should have grabbed some 10TB or 12TB drives instead of the 4's, I upgraded my case when I saw a decent price to allow for future expansion.