FreeNAS 8.0.2 - ZFS RAIDZ Size issue

[Lucas Rey]

Hello guys,
I have 3 x 1TB HDD configured in ZFS RAIDZ on FreeNAS 8.0.2. As the following screenshot, there is something wrong with total size. Why it reports only 1.3 TB?

Code:

[root@nas] ~# df -h
Filesystem             Size    Used   Avail Capacity  Mounted on
/dev/ufs/FreeNASs1a    927M    429M    424M    50%    /
devfs                  1.0K    1.0K      0B   100%    /dev
/dev/md0               4.3M    3.6M    384K    90%    /etc
/dev/md1               732K     28K    648K     4%    /mnt
/dev/md2                75M     32M     37M    47%    /var
/dev/ufs/FreeNASs4      20M    1.3M     17M     7%    /data
tank                   1.3T    883G    401G    69%    /mnt/tank
tank/tvix              942G    541G    401G    57%    /mnt/tank/tvix

Code:

[root@nas] ~# zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          raidz1                                        ONLINE       0     0     0
            gptid/9f4c1cef-24e4-11e1-8a28-50e549677b47  ONLINE       0     0     0
            gptid/9fab2a8a-24e4-11e1-8a28-50e549677b47  ONLINE       0     0     0
            gptid/a00aec67-24e4-11e1-8a28-50e549677b47  ONLINE       0     0     0

errors: No known data errors

I replicated the scenario using VMWare with 3 x 1TB Virtual disk, and the following picture show the result. The correct size is 1.9 TB:

Could someone help me to understand why my "real" NAS has only 1,3 TB ???

 

[jgreco]

The VMware configuration probably uses base2 sizes, so it's likely going to be larger than a correctly-configured system with three 1TB HDD's. If your hardware based system was reporting 1.7 or 1.8TB, I'd say it was probably just fine. There's something amiss if it is only reporting 1.3, but you haven't really given us anything to work with. Post the contents of the relevant lines from /var/run/dmesg.boot as a starting point; let's see what your system *thinks* it has for hardware.

 

[Lucas Rey]

This is all dmsg boot:

Code:

[root@nas] ~# cat /var/run/dmesg.boot
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-RELEASE-p3 #7: Fri Sep 30 12:51:49 PDT 2011
    jpaetzel@servant.iXsystems.com:/b/sf_freenas_build/obj.amd64/b/sf_freenas_build/FreeBSD/src/sys/FREENAS.amd64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Atom(TM) CPU D525   @ 1.80GHz (1799.97-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106ca  Family = 6  Model = 1c  Stepping = 10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 4294967296 (4096 MB)
avail memory = 4090466304 (3900 MB)
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP/HT): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP/HT): APIC ID:  3
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
netsmb_dev: loaded
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bfbe0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xff00-0xff07 mem 0xfdf00000-0xfdf7ffff,0xd0000000-0xdfffffff,0xfdd00000-0xfddfffff irq 16 at device 2.0 on pci0
agp0: <Intel Pineview SVGA controller> on vgapci0
agp0: detected 1020k stolen memory
agp0: aperture size is 256M
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xee00-0xeeff mem 0xfdcff000-0xfdcfffff,0xfdcf8000-0xfdcfbfff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 50:e5:49:67:7b:47
re0: [ITHREAD]
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci0: <JMicron JMB363 AHCI SATA controller> mem 0xfdefe000-0xfdefffff irq 17 at device 0.0 on pci2
ahci0: [ITHREAD]
ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
atapci0: <JMicron JMB363 UDMA133 controller> port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 18 at device 0.1 on pci2
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0xfe00-0xfe1f irq 23 at device 29.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801G (ICH7) USB controller USB-A> on uhci0
uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0xfd00-0xfd1f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801G (ICH7) USB controller USB-B> on uhci1
uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0xfc00-0xfc1f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801G (ICH7) USB controller USB-C> on uhci2
uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0xfb00-0xfb1f irq 16 at device 29.3 on pci0
uhci3: [ITHREAD]
usbus3: <Intel 82801G (ICH7) USB controller USB-D> on uhci3
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xfdfff000-0xfdfff3ff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
pcib3: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci3: <ACPI PCI bus> on pcib3
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel ICH7 AHCI SATA controller> port 0xfa00-0xfa07,0xf900-0xf903,0xf800-0xf807,0xf700-0xf703,0xf600-0xf60f mem 0xfdffe000-0xfdffe3ff irq 19 at device 31.2 on pci0
ahci1: [ITHREAD]
ahci1: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich5: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
orm0: <ISA Option ROM> at iomem 0xd0000-0xd2fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
coretemp0: Can not get Tj(target) from your CPU, using 100C.
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
coretemp1: Can not get Tj(target) from your CPU, using 100C.
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
coretemp2: Can not get Tj(target) from your CPU, using 100C.
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
coretemp3: Can not get Tj(target) from your CPU, using 100C.
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <KINGSTON SS100S28G D100309a> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 7641MB (15649200 512 byte sectors: 16H 63S/T 15525C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD10JPVT-00A1YT0 01.01A01> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <WDC WD10JPVT-00A1YT0 01.01A01> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <WDC WD10JPVT-00A1YT0 01.01A01> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 4
ZFS storage pool version 15

 

[jgreco]

So you've got three "1TB" (== 950GB) drives, so in theory it seems like you should wind up around 1800-1900GB usable. I'm not convinced from these numbers that you don't. The real problem with ZFS is that it kind of throws conventional disk utilization concepts out the window; it's *confusing* to know what the numbers mean.

What's the output of "zpool list"? That's really the best place I've found to identify actual pool numbers from ZFS; it'll give you the pool size (expect yours to be in the neighborhood of 2800GB) and if it does say that, then this really comes down to an exercise in "understanding ZFS disk space reporting," which I can't help you with too much, since I hate it and try not to understand it too much, except for the bits I deem important.

I can tell you that ZFS reports disk space very differently than conventional FFS/UFS. For example, if you have a disk configured with something like "copies=" or compression, the space allocated (and reported) includes blocks allocated under those policies. My head goes numb trying to differentiate some of the numbers reported.

 

[Lucas Rey]

Well, now I'm confused. This is the output from zpool list:

Code:

[root@nas] ~# zpool list
NAME   SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
tank  2.72T  2.09T   645G    76%  ONLINE  /mnt

The size is ok, but the used space is totally wrong!!! Plus, I don't use any compression for my pool....
At this point, I think I'll backup all the data to an external storage disk and then destroy/recreate the pool....

 

[jgreco]

The size is indeed OK. Used space is a complicated value: just like "size" is inclusive of space not actually available to you as a user, because you're using RAIDZ, so also is the USED value "off" a bit... that count includes RAIDZ-required data blocks, I believe. So my guess is more like 1.4T "used". Now this is always where I start scratching my head and trying to remember the right things to summarize, but I do notice from your original

tank 1.3T 883G 401G 69% /mnt/tank
tank/tvix 942G 541G 401G 57% /mnt/tank/tvix

that 883G + 541G = 1424G, which is right in the ballpark of expectations, and 1424G + 401G = 1825G which is in the ballpark of your expected capacity, so if I'm reading this right, there's nothing wrong.

 

[Lucas Rey]

Nope! tank/tvix is a simply dataset where I point for NFS mount.

I just backup all my data into 1TB external HDD. The whole space used, for all data, is: 883 GB (948.421.463.581 byte)
This value include also data from /mnt/tank/tvix => space used ONLY for this directory: 541 GB

So I believe your assumption (883G + 541G = 1424G) is not correct! and the total space used for main pool is 883GB
tank 1.3T 883G 401G 69% /mnt/tank
tank/tvix 942G 541G 401G 57% /mnt/tank/tvix

What is not still clear, is why the total space reported is 1.3Tb... I'm performing another backup to another external HDD (just for security reason), after finish it I'll destroy the pool, and recreate it from scratch.

 

[jgreco]

Okay, so something's wrong, because your ZFS mountpoints tell a different story.

Specifically, if the space needed for all your data is 883GB, for all data contained on both "tank" and "tank/tvix", then you've done something odd like maybe created shadow data. The "Used" value for "tank" all on its own is 883GB; that leaves about zero for tank/tvix. ZFS does not include the space used by a child dataset in the parent's size.

So now my question becomes this: think very carefully about how you went about creating tank/tvix. Was there already data in the directory? Did you create that child dataset, then wonder "where the heck did my data go? oh well, I'll copy it on there again."

Because if you did, then you created your own little problem here. You put two copies of some of your data on the system, because you made some of it inaccessible via the filesystem when you created tank/tvix. That explains pretty much everything we're seeing and you're claiming, and it's an easy mistake to make.

 

[Lucas Rey]

Well, well, well you most probably hits the gol! I had completely forgotten these, but reading what you wrote I remember that I created the ZFS pool, then copied all the data from backup disk (even tvix directory), then, since my Dvico Tvix reader need to point to a specific NFS mount, I created a dataset to tank/tvix. The data under tvix was gone, so I copied again the tvix contents.... But only now (after some weeks) I notice the free space issue.....

Now the question is simple, is there a way to restore the original situation without destroy pool and sure taking care of all tank contents??

 

[jgreco]

Yup. I originally started out with instructions on how to remove the extra dataset, but it's really a good idea to have that, so let's assume you want to keep it.

Do the following steps.

# umount -f /mnt/tank/tvix

Forcibly unmounts tvix dataset. Just in case you had something open in there, otherwise no -f needed.

# mv /mnt/tank/tvix /mnt/tank/oldtvix

Rescue the stranded data by relocating it. You then get to figure out what to do with it, later.

# mkdir /mnt/tank/tvix

Make a mountpoint. I think this may not be required for ZFS, but it's still fine to do it.

# reboot

at which point ZFS will remount everything, NFS will be refreshed, and your stranded data is available at /mnt/tank/oldtvix for your administrative consideration. This also has the advantage that all the steps are easily understood and have minimal lasting side effects.

I had already submitted a bug report regarding this. It's technically inadvertent user error, but it should probably be detected and either forbidden or at least whined-about by the system. The number of truly useful scenarios where one might legitimately want to do this on a NAS product is relatively small, and could probably be restricted to needing to do it by hand at the CLI.

 

[Lucas Rey]

jgreco, many many thanks, it works!!! ;)

Now the situation is the following:

The "tank" size remains 1,3, but the free space should be now correct: 939.8 GiB, and is available for both tank and tvix!

There is still something strange on the pool size. Why it reports 1,3 and 1,4 Tb ?? Maybe there is something on dataset mechanism still not clear for me, or probably it's only a cosmetic issue, whereas I don't setup any quota for this dataset! BTW, this is something I can learn on the net! ;) The most important thigs is the restored available space!
Used Space: 535 + 350 + Available space: 940 = 1825 Gb

Thanks again!

 

[Saeed A Siddiki]

hey guys - i am a newbeee to all of this. My problem is a little diffrent and would hope you can help me out here --


10 DRIVES RECOGNIZED BUT THE total TB's don't add up: (the hardware)

Corsair Obsidian Series Black 650D Mid Tower Computer Case (CC650DW-1)
MSI Z68A-G43 (G3)- LGA1155 Intel Z68 Express Chipset ATX Desktop Motherboard Core i7/i5/i3 SATA300/600 DDR3 USB3.0 CrossFireX 7.1CH Audio
Syba SD-SATA-4P Serial ATA150 4x Ports RAID Controller Card with SIL3114 Chipset - Retail x 2
Crucial Ballistix Sport 16GB kit (8GBx2) DDR3-1600 1.5V 240-Pin UDIMM BLS2CP8G3D1609DS1S00
Cooler Master Silent Pro Gold (SPG) 1200 Watts Modular Power Supply
Syba SY-CAB40007 Molex 4 Pin to 2x 15 Pin SATA Power Cable (5.5 Inches) x 4
SilverStone CFP52B 5.25-Inch to 3.5-Inch Bay Converter with Hot-Swappable Function (Black)
Silverstone Tek Four-in-One SATA Power Connectors with Power Stabilizing Capacitors (CP06)
Intel Core i3-2120 Dual-Core Processor 3.3 GHz 3 MB Cache LGA 1155 - BX80623I32120


The hard drives are the regular run of the mill in these capacities - green drives and one WD Blue Drive:

4 X 1TB * = *4
3X 2TB*= *6
2X 3TB *= *6
1X * 750GB = *750 GB *

TOTAL TO = 16TB physical*
-3tb overhead (in theory)

i should end up at least with at least 12 TB OR 10TB ---

BUT WHATS SHOWING under
RAIDZ IS 5.6TB*and
under RAIDZ2 5.3TB (i still don't have REAL DATA stored on this volume yet)

Based on what you guys suggested - what would be this predicament which is different --- why am i only getting 5.6 TB? or is it even really being reported correctly? how can i report the most accurate capacity numbers? under APPLS FINDER i am getting 5.6TB availability as a total? shall i attempt different VDEV for the varying capacities of drives i have here?

Would appreciate anyone who can further my fruitful exploration before i actually start moving my DATA files on this
bad boy …. cheers

 

[donairb]

If you have all of those drives in one vdev, the smallest drive determines the size of the vdev.

So, 10 drives in RAIDZ, where the smallest drive is 750GB = 9 x 750GB (~6.7 TB)

same 10 drives in RAIDZ2 = 8 x 750GB (~6.0TB)

A better idea would be to go with all drives the same size or split up into different vdevs, each containing same-sized drives.

You could get ~10TB by using three vdevs:
4 x 1TB RAIDZ = 3TB
3 x 2TB RAIDZ = 4TB
2 x 3TB RAID1 (mirror) = 3TB

Brian