pool size, autoexpand and what am I missing?

[Xcapee]

I have a 4 drive RAIDZ zpool, which now has 4x3TB WD Green drives. It started with 2x3TB and 2x2TB drives because that was what I had on hand. I have since replaced the 2x2TB drives with 3TB drives (resilvering with each drive). autoexpand was true prior to the replacements.

Each WD Green Drive reports 3.00TB

bosnas# smartctl -i /dev/ada3 | grep Capacity
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
bosnas# smartctl -i /dev/ada2 | grep Capacity
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
bosnas# smartctl -i /dev/ada1 | grep Capacity
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
bosnas# smartctl -i /dev/ada0 | grep Capacity
User Capacity: 3,000,592,982,016 bytes [3.00 TB]

So I would expect the size of the zpool to be 9TB or 8.18TiB but

bosnas# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 7.25T 4.17T 3.08T 57% 1.00x ONLINE /mnt

I am assuming the size reported is in TiB, but that still leaves about 1TB that appears to be AWOL.

7.25 TiB is pretty close to 8TB so I am guessing zfs is still looking at the vdev as being built on 2TB drives. So assuming it was the autoexpand that failed, I exported the drives and imported them as per Enabling ZFS Pool Expansion After Drive Replacement. No change.

What am I missing?

 

[Dusan]

Try to run "zpool online -e tank <device>" for every component of the pool (the devices you see in zpool status).

 

[Xcapee]

Thank you. That fixed it. The man page is very sparse on zpool online. What does the -e flag do? Expand?

bosnas# zpool online -e tank ada0
bosnas# zpool online -e tank ada1
bosnas# zpool online -e tank ada2
bosnas# zpool online -e tank ada3

bosnas# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT

tank 10.9T 4.17T 6.70T 38% 1.00x ONLINE /mnt

 

[Dusan]

http://www.freebsd.org/cgi/man.cgi?query=zpool:

-e Expand the device to use all available space. If the device is part of a mirror or raidz then all devices must be expanded before the new space will become available to the pool.

 

[Xcapee]

Ha ha - thanks. I just realised I've been reading the FreeBSD Release 8.1 man pages (which as I say, had very sparse info on "online").

So, any theories why autoexpand may not have worked?

 

[Dusan]

Is the autoexpansion enabled? What does "zpool get autoexpand tank" say?

 

[Xcapee]

bosnas# zpool get autoexpand tank
NAME PROPERTY VALUE SOURCE

tank autoexpand on local

 

[Xcapee]

bosnas# zpool get autoexpand tank
NAME PROPERTY VALUE SOURCE

tank autoexpand on local
I had set it from the command line, prior to any of the drives being replaced (I think).

 

[Dusan]

Sorry, no idea at the moment. I would have to check the code.

 

[Xcapee]

I am guessing I might have turned autoexpand on after I replaced one of the failed 3TB drives. Even though it wouldn't have expanded at this point, since I wasn't at 4x3TB, it may have missed the opportunity since each drive needed to be told to expand with the online -e. It may have been that only one device was blocking the expansion as a whole.

Thanks again for your help :)