[FreeNAS-9.1.0-RELEASE-x64 (dff7d13)]Replace ZFS pool disks to grader disks

[Hugo]

Thanks for your quick response and advice. I understand you will have to rebuild the pool in order to get the 4k sector size. But it seems like I'm better of using raidz2 pools anyway (which also require a rebuild of the zpool).
I will probably move the data to another zpool to be able to setup new pools using 4k sectors and raidz2. As this is an early stage production system it's probably best to do this right now.

 

[cyberjock]

Thanks for your quick response and advice. I understand you will have to rebuild the pool in order to get the 4k sector size. But it seems like I'm better of using raidz2 pools anyway (which also require a rebuild of the zpool).
I will probably move the data to another zpool to be able to setup new pools using 4k sectors and raidz2. As this is an early stage production system it's probably best to do this right now.

That's exactly what I would do if I were in your shoes. :)

Edit: Actually, I might try for a 6 disk RAIDZ2, so that it's the "optimal" configuration for RAIDZ2.

 

[Hugo]

Thanks for the superb support! :)

 

[djseto]

FreeNAS 9 just use 4K sector by default unless you instruct it not to do so (with sysctl). You can check ashift after creation by doing 'zdb -C | grep ashift', where 12 means 4k and 9 means 512.

This definitely isn't true for me. See my post below your original one. Can someone please explain why a newly created pool still shows 512 and not 4k?

 

[vvizard]

I'm pretty sure I specified 4K sectors when I created my pool while back on Freenas-7.x (later upgraded to 8.x and 9.x). But now I just ran into this issue with my first HDD failing, and having to replace it. Turns out my pool is ashift=9 after all, with 8x3TB drives (which are all 4K capable). Kind of sucks, as it's to expensive to buy new drives to back up all the data to recreate a pool.

I used sysctl (had to use both of them) to replace the drive. What's in it in the long run? Can I continue to run this, and keep upgrading FreeNAS/zfs at the regular releases, or is there some precautions I now will have to take with these sysctls in place?

 

[c0y0te]

Hi,

We are aware of this issue and the alert system will be extended to help assist in this case.

For now, please try setting this in your sysctl variable, as described at http://doc.freenas.org/index.php/Sysctls :

Code:

vfs.zfs.vdev.larger_ashift_minimal = 0

Fine print: setting this to 0 would make FreeNAS use ashift=9 if disk is not "AF" ("Advanced Format" or 4K sector), which allows using non-AF disks to replace existing disk in non-AF ZFS pools, but will make it hard for future
And see if this would solve your problem. If not, you will have to additionally set:

Code:

vfs.zfs.vdev.larger_ashift_disable = 1

Fine print: setting this to 1 would make FreeNAS ignore the fact that you are using AF disk. This would negatively affect performance but will make it possible to use an AF disk in a non-AF pool.
Note that we would recommend backing up all of your data and recreate the pool without these tunables, as that would give you a more future-proof pool.

Same error for me replacing a bad drive, although I was using an exact matching replacement disk (Wd RE4-GP Enterprise 2TB was under warranty). In order to replace the disk, I had to set the sysctl variable vfs.zfs.vdev.larger_ashift_minimal = 0. I then was able to replace and resilver. My question, can I now safely remove the sysctl variable or should I leave it in there?

 

[John Doe]

Well, this sucks. I'm using FreeNAS 8.3.1 and I told it to force 4K when I set it up. I use full disk encryption as well and the documentation seems to indicate that 4K is required for that. So, I assumed that it's been doing that for the last year and a half. But I just did that zdb check and it says ashift=9! These are 4K disks. I just tried to replace one with the exact same make and model and got the "devices have different sector alignment" error. :( Since FreeNAS forced these 4K disks to 512 in the first place despite my wishes, I'm not sure why it won't do the same with the new one.

When I try to use those sysctl values it says "unknown oid", so I'm guessing those are 9.x specific. :(

Any ideas on how I can make this work? I seem to remember having to do a format when I created the volume initially, perhaps something to do with that? Can I reformat the drive as though I were adding it? I feel like the replace should do that. Actually, I feel like by buying the exact same make and model drive, this should've just worked with no problems. Sigh, nothing ever does any more.

There's about 5TiB of data here, kind of hard to backup. Really don't want to re-rip the thousands of DVDs (the Buffy box set alone has 30 discs! and Angel adds 20 more! so several hundred boxes turn into thousands of discs and lots of time, especially the ones that require special handling due to extra "protection").

 

[solarisguy]

@John Doe, did you upgrade since creating the pool?!

 

[solarisguy]

And what does <code>smartctl</code> exactly say about the old disks and the new disk?

 

[John Doe]

Wow, that was a fast response. Thanks...

I upgraded from 8.3.0, it was entirely too much work so I never did it again. :p Actually, once I get things the way I like, I rarely upgrade. I created the pool, but by the time I actually started working on it 8.3.1 was out. So, yeah, I upgraded 0.0.1 versions. I wish I had known that U can't upgrade FreeNAS without removing the ability to replace disks.

The old disk was failing SMART, was failing to even read values on occasion. The new one passes with no errors. I'm not sure what info U want, but all the drives are this model: http://www.newegg.com/Product/Product.aspx?Item=N82E16822148844 I'm not a big believer in mixing-and-matching.

 

[solarisguy]

Looking for something like

Code:

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD10JPVX-22JC3T0
Firmware Version: 01.01A01
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jun  4 17:22:03 2014 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Black
Device Model:     WDC WD1502FAEX-007BA0
Firmware Version: 05.01D05
User Capacity:    1,500,301,910,016 bytes [1.50 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Oct 22 21:32:07 2014 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Is there anything different in the first section of smartctl output but the Serial Number and LU WWN Device Id between the new drive and the failed one?

Or execute camcontrol identify adaX once with adaX being identity of the new drive and then being one of the old ones. Again, any differences in the first two sections?

What is your zpool version?

 

[John Doe]

Yeah, there are a couple of differences: device model on the old ones is ST3000DM001-1CH166 but on the new ones it's ST3000DM001-1ER166, the model is the same, but the part moved from CH to ER, I wonder if that's some sort of origin code (China to Eritrea‽), same firmware though; and looks like the ATA standard / protocol got an upgrade from "ATA-8-ACS revision 4" / "ATA/ATAPI-8 SATA 3.x" to "ACS-2" / "ATA/ATAPI-9 SATA 3.x".

smartctl -i (old, new)

Code:

Model Family:     Seagate Barracuda (SATA 3Gb/s, 4K Sectors)
Device Model:     ST3000DM001-1CH166
Serial Number:    Z1F0ZQLF
LU WWN Device Id: 5 000c50 04e0a5fca
Firmware Version: CC43
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:

Model Family:     Seagate Barracuda (SATA 3Gb/s, 4K Sectors)
Device Model:     ST3000DM001-1ER166
Serial Number:    Z500B7WM
LU WWN Device Id: 5 000c50 0792bb678
Firmware Version: CC43
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ACS-2 (unknown minor revision code: 0x001f)
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

camcontrol identify (old, new)

Code:

protocol              ATA/ATAPI-8 SATA 3.x
device model          ST3000DM001-1CH166
firmware revision     CC43
serial number         Z1F0ZQLF
WWN                   5000c5004e0a5fca
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       5860533168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6
media RPM             7200

Code:

protocol              ATA/ATAPI-9 SATA 3.x
device model          ST3000DM001-1ER166
firmware revision     CC43
serial number         Z500B7WM
WWN                   5000c500792bb678
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       5860533168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6
media RPM             7200

Do U think it could be something protocol related? It's weird that camcontrol says the new drive is version 9 yet smartctl says it's 8, but gets confused reading the standard. That's sounding a bit finicky to me. I've been using Unix-a-likes a very long time, but my favorite part was when they started detecting hardware on their own and I never had to do any of this low level admin type stuff again; I'm a little out of shape. :p

So, same make, model, physical characteristics, and firmware. The only difference is origin and protocol. So, either FreeNAS is racist and will work with Chinese drives, but not African; or it's in need of C-3PO. ;) Sorry, I didn't sleep too well and I'm a little loopy for this level of analysis. :D

 

[solarisguy]

Let me guess:
* created volume/pool using 8.3.0
* upgraded to 8.3.1
* upgraded ZFS pool version to version 30 (check with zpool get version)
* encrypted the volume/pool (not available before version 30)

Then you hit a bug possibly due to different ways disk geometry is seen by 8.3.0 and 8.3.1. And since GELI was introduced with 8.3.1 that is quite likely.

The above sequence hinges on your words upgraded from 8.3.0

Workarounds might be possible, however I am not familiar with ZFS or FreeBSD internals. Did you try partitioning the new disk exactly the same way the old disk is/was? Under 8.3.0?

 

[cyberjock]

Umm.. you are confused @solarisguy

If you upgraded to v30 you are not using anything open-source. ZFS went from v28 to v5000 because Oracle close-sourced zfs with v29.

Also, you cannot encrypt a volume/pool except during creation. There is no option to encrypt after the fact. ;)

Put down the beer!

 

[solarisguy]

You are right :), there were events very recently... ;) No beer or anything like that though

And I should not talk about encryption, since I am not using it...

I took 30 straight from http://www.freenas.org/images/resou...8.3.1_guide.html#__RefHeading__22587_54666437

I started with 8.3.1, but my hardware has substantially changed, if I have 8.3.1 up and running I open my mouth again.

 

[John Doe]

Darn it, I thought I answered the version question, but I see that I did not. It's 28. I don't know how FreeNAS handles the encryption, I just remember being happy I didn't have to figure out how to do it myself. But maybe I should've built it myself, then at least I might know a little of what was going on. :( The only thing I ever upgraded was FreeNAS; if during that process it asked me to do something else, I probably did, but only because I foolishly thought such a small version increase couldn't possibly break anything; especially since I hadn't done anything with the system yet. Like I said, once I start changing things I'm loathe to upgrade; but if I haven't done anything yet, an upgrade shouldn't break anything.

No, I didn't partition the disk. There doesn't seem to be a way to do that in FreeNAS, and as I said, I long ago stopped dealing with hardware at that level from the command line (relearning fdisk doesn't sound appealing). I tried doing a quick wipe, but that didn't help. I even tried creating a new volume with the disk, thinking it might force a format on it; that didn't help either.

Hard to believe that a 0.0.1 version increase really does mean that I can no longer replace the disks. There have been many things I've disliked about FreeNAS, but this takes the cake. It took away all the work of managing a NAS with a nice web interface, so the problems were worth it; little did I know it would also take away the only point of having RAID: replacing disks without losing data. Sigh.
hmm... I just got your last post while I was writing this and according to that link 8.3.1 introduced the encryption support. I thought I created the pool and then upgraded, but if that's true then I had to have created the pool after upgrading. Which means I'm on the exact same system under which the pool was created and yet it can't replace the disk.

It's sounding more and more like I better find a way to back this stuff up if I don't want to re-create it from source material. WD makes a 6TB My Book; I have 5.4TB (5.0TiB) of data; I hate that HDD manufacturers--in collusion with file-system developers--make it unclear if that will fit once the drive is formatted. Then I'll be faced with a decision: I really don't want to build a fully encrypted RAID NAS system with e-mail monitoring and scheduled scrubs and NFS by hand, but I'm hesitant to use FreeNAS again and I wonder if any other NAS-specific OS/distro would be any better (my guess is they'd be even worse). FML.