Total usable space

[lmaFNS]

Dear members,
This is my first post here and excuse me for any mistakes.
We purchased a 96TB (16*6TB) storage server and configured it as RAIDZ-2 with two volumes of 48TB (8 drives in each vol). The usable space the server shows is 60TiB (30TiB in each volume) which is 66 Terabytes. From the calculations I made I should be getting a 72TB usable space with RAIDZ2 config. I am loosing 6TB (one drive space) somewhere.

The only reason I can think of is if the OS is installed in a drive and that drive is now totally taken up by FreeNAS. First, how do I check where the OS is installed? I didn't configure the system. I don't think the overheads will be 6TB... Could any one help me out what's happening here?

Thanks and appreciate your time!
L

 

[BigDave]

The only reason I can think of is if the OS is installed in a drive and that drive is now totally taken up by FreeNAS.

Use this calculator.
From the freenas gui, go to storage, then click view disks,
all the disks in your pool will show there.
Your disk with the OS on it should not be listed there
if you are using 9.3 like I am...
If you're one short, then your hunch is correct.

 

[cyberjock]

Or post a debug file from your server. That will tell me what is going on.

My 10x6TB pool that is a single RAIDZ2 shows 54.5TB total space. If you subtract about 12TB from that to give you about 40TB, so something is not right.

 

[BigDave]

RAID Mode: RAIDz2
Disk Size: 6 TB
Quantity of Disks: 8
*Raw Storage: 48.0 TB / 48000.0 GB
*Usable Storage: 32.7 TB / 33527.6 GB
Multiply X2 = 65.4 TB total usable space

 

[lmaFNS]

RAID Mode: RAIDz2
Disk Size: 6 TB
Quantity of Disks: 8
*Raw Storage: 48.0 TB / 48000.0 GB
*Usable Storage: 32.7 TB / 33527.6 GB
Multiply X2 = 65.4 TB total usable space

Thanks for the reply Dave.
The link you provided gives the right numbers I see but other calculators give different numbers. For ex: http://wintelguy.com/raidcalc.pl
I understand that two disks would be used for double parity out of 8 disks per volume. That should leave a usable space of 6*(8-2)=36TB per volume. From the link I provided here, it seems ZFS uses ~1.5% space for metadata which should be about half a TB per vol. I still should see the remaining ~2.5TB per vol as free space.

Also, yes I see all the drives (attachment) on the GUI. So, the OS must be installed in a pen drive and all 6TB hard disks should be usable for data storage.
Best,
L

 

[lmaFNS]

Or post a debug file from your server. That will tell me what is going on.

My 10x6TB pool that is a single RAIDZ2 shows 54.5TB total space. If you subtract about 12TB from that to give you about 40TB, so something is not right.

I don't seem to understand the logic Cyberjock. How did you end up with 40TB with RAIDZ2 config on your 10*6TB drives? What debug file section would help and where do I find it?

 

[Starpulkka]

I think that intel calculator calculates marketin GB not a real GB if you think like that an actual kilo (1000) actually needs 1024 so as an example 6000GB dont give you 6000GB it gives you 5450GB (or somethin) and if you put that 5450GB on that intel calculator it gives real numbers. Markettin is funni 4GB is really 3.5GB in nvidia company. As for servehome forum its real deal theres an asterix in below *Usable storage is the actual post-format amount
Edit: This is just how i have managed to explain easily to consumers something, please dont tell, how SI- systems and bits and bytes actually go.. normal consumers just seem to not understand.
Edit2: Oh sorry i totally missed that TiB i only looked

which is 66 Terabytes.
L

And thought its that decimal system.(even you did say bytes)
You should get 32.19TiB +32.19TiB totals 64.38TiB if you look in gui (raidz2)
What does zpool get size poolname shows?
Do you really do have on one hdd like Capacity: 6 001 175 126 016 bytes (this is what my wd red 6TB shows in smartctl)

 

[Scharbag]

Good ole decimal/binary math. Found this from another site:

We seem to generally talk in decimal, and that's how capacity is reported by hard drive manufacturers, but the underlying computer system (well, Windows) reports in binary, hence the difference.

1 byte = 10^0 in decimal, 1 byte = 2^0 in binary ( i.e. in both cases, it equals 1)
1 KB = 10^3 in decimal, 1 KiB = 2^10 in binary (1,024 bytes)
1 MB = 10^6 in decimal, 1 MiB = 2^20 in binary (1,048,576 bytes)
1 GB = 10^9 in decimal, 1 GiB = 2^30 in binary (1,073,741,824 bytes)
1 TB = 10^12 in decimal, 1 TiB = 2^40 in binary (1,099,511,627,776 bytes)

To convert from decimal to binary, you need to do some math, so for 1TB, it would be:

(10^12)*(10^12)/(2^40)= 909,494,701,772 bytes (909.49 GiB)

So for a 6TB drive, there should be 5.456 TiB (5,587 GiB) of useable space (according to a Google converter).

Using 6000 GB for the drive size in an online calculator, the result is 72,000 useable GB of space in the 8x2x6 RaidZ2 that you are configuring which agrees with your assumption. Converting 72 TB to TiB, we get 65.48 TiB.

For my system of 6x2x3 Raid Z2, using the above numbers, I get a calculated total useable space number of 21.82 TiB. When I check my system's actual numbers:

Code:

% zpool list bigtank
NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
bigtank  32.5T  17.0T  15.5T         -    18%    52%  1.00x  ONLINE  /mnt
% zfs list bigtank
NAME      USED  AVAIL  REFER  MOUNTPOINT
bigtank  11.3T  9.66T   368K  /mnt/bigtank

From the above system output, I get 20.96 as the total size of the zfs system for bigtank. There is less than 1 TiB of difference between calculated and actual, which fits with the ZFS overhead I think. That or drive specifications not being exactly 3TB in my system.

So, in your system, a total space number of ~66 TiB seems to be what you should expect. So yeah, either the 6TB drives you bought are not really 6TB or something is freaky.

Not sure if this helped but I learned about Tebibytes!!

Cheers,

 

[lmaFNS]

I think that intel calculator calculates marketin GB not a real GB if you think like that an actual kilo (1000) actually needs 1024 so as an example 6000GB dont give you 6000GB it gives you 5450GB (or somethin) and if you put that 5450GB on that intel calculator it gives real numbers. Markettin is funni 4GB is really 3.5GB in nvidia company. As for servehome forum its real deal theres an asterix in below *Usable storage is the actual post-format amount
Edit: This is just how i have managed to explain easily to consumers something, please dont tell, how SI- systems and bits and bytes actually go.. normal consumers just seem to not understand.
Edit2: Oh sorry i totally missed that TiB i only looked
And thought its that decimal system.(even you did say bytes)
You should get 32.19TiB +32.19TiB totals 64.38TiB if you look in gui (raidz2)
What does zpool get size poolname shows?
Do you really do have on one hdd like Capacity: 6 001 175 126 016 bytes (this is what my wd red 6TB shows in smartctl)

Yes, I have been clear with the difference in TiB and TB.
The zpool get size command shows 43.5T (TiB) which is right. But the usable space is only 30TiB per volume. And all drives are on and being used.
The calculator provided by Dave above gives 30TiB as usable space but some others shows more (~32.5TiB) and am not sure where the difference is coming from
L

 

[Starpulkka]

Yes, I have been clear with the difference in TiB and TB.
The zpool get size command shows 43.5T (TiB) which is right.

Ok if it shows on console and is saying 43.5T you lack lots of usable space, starting to think that you dont have two raidz2.
Edit:aha you have two on those so you do have all space.
I used Freenas 9.2.1.9 so im not sure if you use 9.3 is it gives differend numbers on console, but you sayed 43.5T so i assume you dont mean TiB.
In 9.2.1.9 gui and console gives different numbers, as in console it should give higher number T than in gui TiB (9.2.1.9).
Why dont you just post debug file what cyberjock asked days ago. =) I cant compare more numbers now, i currently have delicate FS mergin process goin on my freenas and linux servers 22TiB and linux server crashed one time already.
Edit:
As for why gui dont show real usable space 32.19TiB, you say its only show 30TiB, i dont have a clue, maby its glitch in 9.3 gui version, or i miscalculated it. As you sayed you see in console 43.5T+43.5T it is right, witch you already sayed. There has been some bugs in size calculations in different versions in past..

 

[lmaFNS]

Ok if it shows on console and is saying 43.5T you lack lots of usable space, starting to think that you dont have two raidz2.
I used Freenas 9.2.1.9 so im not sure if you use 9.3 is it gives differend numbers on console, but you sayed 43.5T so i assume you dont mean TiB.
In 9.2.1.9 gui and console gives different numbers, as in console it should give higher number T than in gui TiB (9.2.1.9).
Why dont you just post debug file what cyberjock asked days ago. =) I cant compare more numbers now, i currently have delicate FS mergin process goin on my freenas and linux servers 22TiB and linux server crashed one time already.

I use 9.3 and when I say 43.5T, I meant TiB as mentioned in the message. I checked the configuration and it's RAIDZ2-0 for both volumes. Attaching here the debug file obtained with the command "freenas-debug -z" (dump ZFS configuration).
Hopefully this will be useful to everyone on this list to debug this issue...

 

[Kljajoni]

Hello,

I have exactly the same problem as ImaFNS. Today I created RAIDZ2 volume from 8 WD 6TB Red Pro drives. Total space 43.5, usable space 30 TiB. I tried to copy some data on it, but usable space is still little lower than expected (32.5TiB)

I am using Freenas 9.3.0. I cannot upgrade (yet) to 9.3.1 (LSI 9211-8i, P16/P20 driver mismatch) to try to see what would happen with new version.

Does anyone have an idea what could be the problem and how can I troubleshoot it ?

 

[pirateghost]

Hello,

I have exactly the same problem as ImaFNS. Today I created RAIDZ2 volume from 8 WD 6TB Red Pro drives. Total space 43.5, usable space 30 TiB. I tried to copy some data on it, but usable space is still little lower than expected (32.5TiB)

View attachment 9712

I am using Freenas 9.3.0. I cannot upgrade (yet) to 9.3.1 (LSI 9211-8i, P16/P20 driver mismatch) to try to see what would happen with new version.

Does anyone have an idea what could be the problem and how can I troubleshoot it ?

There isn't a problem. Do the math.

 

[Kljajoni]

I admin that I do not understand in detail how is usable space calculated, but can someone explain to me why is the Estimated capacity below (32.74 TiB) different from the capacity that I am getting (30 TiB) ? Something similar was discussed in this thread , but I did not found (or did not understood) the right answer (still haven't figured out where's more than 2 TiB gone)

Tried to create couple of different RAIDZ2 configurations with 6TB drives , got this results (in TiB)

Total Usable

4 drives - 21.7 vs 10.2 ~ 47%
5 drives - 27.2 vs 15.6 ~ 57%
6 drives - 32.5 vs 21 ~ 64.6%
7 drives - 38 vs 24.5 ~ 64.4%
8 drives - 43.5 vs 30 ~ 68.9%

If this is normal behaviour for ZFS - I am fine with that, just need confirmation.
And definitively will do my homework to understand this in more detail.

 

[depasseg]

So with raidz2, you are losing 2 drives worth of usable capacity. So a 4 drive array had 2 drives available for usable data minus the overhead.

@Bidule0hm has a great disk space calculator that factors in all these things.

 

[AMiGAmann]

Check SirMaster's explanation in this post:
https://forums.freenas.org/index.php?threads/total-usable-space.38778/page-3#post-248071

You don't need to live with that "missing" space! Just create a dataset with 1MiB recordsize (if you are storing big files). You can then effectively store more than the displayed free space.