less free space with 8-disks raidz2 than a pair of 4-disks raidz1 ?!?

[djoole]

I had a zpool with 2 4-disks raidz1 vdev inside.
It made me 10.7 TiB free space.

I realized this layout was completely insane (loosing one raidz1 would crash the entire pool), so i've just mad a 8-disks raidz2.

Now, free space is 10.1 TiB

I lost .6 TiB; how come a raidz2 takes more space than 2 raidz1??

 

[Milhouse]

Were all the disks exactly the same capacity? What are they, all 1.5TB? All from the same manufacturer, and the exact same number of sectors or slightly different?

Finally, maybe RAIDZ2 requires slightly more meta-data storage than RAIDZ1.

 

[djoole]

Actually, all the disks are 2 TB, not the same manufacturer, but seem to be the same in capacity.

Here is the detailed info :

Code:

ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada7: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

And recovering this info, i have a doubt : i created the zpool with "force 4K sectors" option.
Looking at the above info and this :

Code:

Sector Sizes:     512 bytes logical, 4096 bytes physical

, it seems the drives are formatted in 512b sectors.

What the..?

Is there any way to check if creation in 4K is in place or not?


Edit : here is some more info if it can help :

Code:

[root@nas] /mnt/zepool/apps# zpool history zepool
History for 'zepool':
2011-10-15.20:32:31 zpool create -o cachefile=/data/zfs/zpool.cache -O aclmode=passthrough -O aclinherit=passthrough -f -m /mnt/zepool -o altroot=/mnt zepool raidz2 /dev/ada0p2.nop /dev/ada1p2.nop /dev/ada2p2.nop /dev/ada3p2.nop /dev/ada4p2.nop /dev/ada5p2.nop /dev/ada6p2.nop /dev/ada7p2.nop
2011-10-15.20:32:33 zfs inherit mountpoint zepool
2011-10-15.20:32:33 zpool export zepool
2011-10-15.20:32:36 zpool import -R /mnt zepool
2011-10-15.20:32:36 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-15.20:57:12 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-15.20:57:12 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-15.21:45:20 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-15.21:45:20 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-15.21:52:20 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-15.21:52:21 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-15.21:59:21 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-15.21:59:22 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-15.22:28:38 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-15.22:28:38 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-16.00:26:09 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-16.00:26:10 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-16.11:07:44 zpool import -o cachefile=none -R /mnt -f zepool
2011-10-16.11:07:45 zpool set cachefile=/data/zfs/zpool.cache zepool
2011-10-16.11:09:12 zpool clear zepool

And before the creation, in the messages logs, there was something like "gnop create -S 4096 /dev/ada*"

(i made the creation with the web ui)


By the way, is there any way not to loose the messages logs after reboot?

 

[Milhouse]

Is there any way to check if creation in 4K is in place or not?

"zdb|grep ashift" should return "ashift=12" for 4K formatted vdevs, and "ashift=9" for 512 byte.

When you created the single 8-disk vdev, did you remember to use the same amount of swap (default is 2GB per disk).

Other than that, not sure how you've got your numbers, either with the first two-vdev attempt or the second single vdev - assuming 1.8-1.9TB per disk, you should have seen over 11TB of usable storage capacity with two RAIDZ1 vdevs...

 

[djoole]

ashift is 12, great :)
Thanks!

As for the number, i got them on the web ui (size column = 10.1 TiB)

I've just done a "df -H" and it says 11 T size

Windows, from the CIFS share, says 10.1 TB

Theorically, a single disk has 1.819 TB, so a raidz2 with 8 disks should have 10.914.
If i remove 16 GB of swap, there is 10.9 TB left

I miss .8 TB, i assume FreeNAS system uses them for something, but i don't know what...

 

[Milhouse]

Use the output from "zfs list", as this is probably the only one figure that can be trusted as df (IMHO) gets confused by ZFS datasets etc. As for Windows... ;-)

Yes, using the lower figure of 1.819TB, you'd be looking at about 10.9TB of actual storage for RAIDZ2 (and in theory, similar for two RAIDZ1).

If you're still missing 0.8TB, the only thought that comes to mind is "reserved space" which some filesystems implement (eg. ext2, a default figure of about 8% that can be changed with tunefs/tune2fs), but I don't think this applies to ZFS.

 

[djoole]

zfs list shows 10.1TB (used + available)

So it seems that ZFS use 800GB as reserved space.

Is there a command to show how the disks are physically partitionned? Maybe i could see that mysterious missing area..

I have another question completely off topic, but as i have the attention of a "senior"... ;)
I understood that if i wanteds to write something elsewhere than /mnt/myzpools, i have to send the commande "mount -uw /", write, then "mount -ur /"
Are the modified files saved on the system disk? (SD card in my case)
Isn't there like a copy in RAM that will erase all my modifications?
Because i tried to modifier the /etc/ssl/freenas/httpd.pem file, and after a reboot, it comes back to original...

 

[Milhouse]

Are the modified files saved on the system disk? (SD card in my case)
Isn't there like a copy in RAM that will erase all my modifications?
Because i tried to modifier the /etc/ssl/freenas/httpd.pem file, and after a reboot, it comes back to original...

It looks like httpd.pem is being generated from scratch during the boot, using the key details entered into Settings -> SSL, so I would suggest trying to add your key via the GUI.

Normally though, if you want a system file change to survive a reboot you make the change below /conf/base/.

 

[djoole]

The SSL settings GUI is bugged (http://forums.freenas.org/showthrea...y-Generated-SSL-Certificate-Can-t-be-replaced), that's why i wanted to do it manually in the first place ;)
Well i guess i'll have to live with it, it's not that important, it's just that my browser show a warning when i log to the Web UI.