Incorect sectors on WD20EARS disks

[Daisuke]

Hi guys,

I created a ticket for developers, but I'm asking also here hoping I will find a solution to my problem. While creating a ZFS array on WD20EARS disks, forcing the 4K sectors will do absolutely nothing. Well it does, partially. The main da0 disk is still formatted with 63 sectors making the hole array misaligned.

Anyone knows how to create properly an aligned array? I don't even know how to properly destroy the data, by default the array stays there, even if I destroy it in GUI. Running:
dd if=/dev/zero of=/dev/da0 bs=1m count=1

results on Device busy, so I cannot remove the existing partitions.

 

[Digidoc]

You shouldn't have to force 4K sectors on those drives since they're already 4K sector by default (unless you set the jumper on them to not be). It sounds like you may have a bad drive.

Try using a gparted live CD to wipe the drives:

http://gparted.sourceforge.net/

then try to rebuild the ZFS array in FreeNAS. If you still can't format that one drive, pull it and try formatting it in Windows (and force 4K sectors on it in the format utility) and see if it lets you do it.

Something tells me that you're still going to have a problem with that drive even after you go through all of this.

BTW, two months ago I bought 7 brand new WD20EARS drives. One of them failed less than 24 hours into the build. It was causing all sorts of weirdness with FreeNAS until I figured out the drive was bad.

 

[Daisuke]

The WD20EARS are known for reporting the wrong sectors to OS. This was fixed on WD20EARX, WD don't care/won't fix the EARS.
Do you know what steps I need to perform with Gparted, once I put it on a USB stick? I'm new to FreeBSD, using CentOS since forever.

 

[ProtoSD]

default the array stays there, even if I destroy it in GUI. Running:
dd if=/dev/zero of=/dev/da0 bs=1m count=1

results on Device busy, so I cannot remove the existing partitions.

Someone just reported a bug in the bugs thread about this the other day. The disks still think they're part of a GEOM device, but just clicking the DESTROY in the gui doesn't wipe the disks, you need to click that checkbox.

This happens because what we call GEOM is holding the disks from some reason, it is because they are currently in use by _something_.
Something could be a geom mirror/stripe/raid3, zpool, whatever...

I suggest you first to try use Auto Import to import anything the GUI can detect, and then Destroy it, be clicking in "Export Volume" and then check the box to erase the disks...
If that doesnt work:
# sysctl kern.geom.debugflags=16

and only then "dd" the disks and reboot

 

[Digidoc]

This is the first I've heard about that error with the EARS drives (I've heard of the infant mortality problem and experienced it myself though).

Gparted is pretty straight forward to use. It's mouse driven and you basically click on the partitions/drives you want to manage.

Try using Protosd's method first though before restoring to gparted.

The WD20EARS are known for reporting the wrong sectors to OS. This was fixed on WD20EARX, WD don't care/won't fix the EARS.
Do you know what steps I need to perform with Gparted, once I put it on a USB stick? I'm new to FreeBSD, using CentOS since forever.

 

[Daisuke]

Someone just reported a bug in the bugs thread about this the other day. The disks still think they're part of a GEOM device, but just clicking the DESTROY in the gui doesn't wipe the disks, you need to click that checkbox.

Sorry, I don't understand what option to check. I destroyed already all data, no volumes found.
But they are still present, if I run:
# gpart show

How do I wipe all disks completely? I see a bunch of files into /dev, where I should see none related to disks.

This is the first I've heard about that error with the EARS drives (I've heard of the infant mortality problem and experienced it myself though).

Gparted is pretty straight forward to use. It's mouse driven and you basically click on the partitions/drives you want to manage.

I think there are 1 million threads reporting the WD20EARS craziness regarding the bad sectors info. Try it on your system also, they report 512, instead of 4096. Does Gparted has a method to properly align the sectors to 4096? Because if it does not, then I might as well wait for instructions how to properly create a ZFS array through CLI and then auto-import it into GUI.

This is what I'm looking at, with clean disks:

# dd if=/dev/zero of=/dev/da0 bs=1m count=1
(do all this for all disks, da1, da2, etc.)

# gnop create -S 4096 /dev/da0
# zpool create nas raidz1 da0.nop da1 da2 da3 da4 da5
# zpool export nas
# gnop destroy /dev/da0.nop
# zpool import nas

I don't even know if this is the proper method to create the RaidZ1 array, going with the FreeBSD documentation out on a limb. I would appreciate if anyone can actually post the proper steps.

 

[ProtoSD]

I see a bunch of files into /dev, where I should see none related to disks.

Don't make the same mistake I made many many years ago ;-) files in /dev are devices needed to run your system, like drivers in a way, don't delete them, even when a disk has no data, there should always be a file for it in /dev.

Did you try what I quoted in post #4 above?

 

[Daisuke]

OK, ran the commands:

Code:

[root@pluto] ~# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16
[root@pluto] ~# dd if=/dev/zero of=/dev/da0 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.064293 secs (16309357 bytes/sec)
[root@pluto] ~# dd if=/dev/zero of=/dev/da1 bs=1m count=1
dd: /dev/da1: Operation not supported
[root@pluto] ~# dd if=/dev/zero of=/dev/ada0 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.006544 secs (160231948 bytes/sec)
[root@pluto] ~# dd if=/dev/zero of=/dev/ada1 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.006565 secs (159725677 bytes/sec)
[root@pluto] ~# dd if=/dev/zero of=/dev/ada2 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.432328 secs (2425418 bytes/sec)
[root@pluto] ~# dd if=/dev/zero of=/dev/ada3 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.007119 secs (147288899 bytes/sec)
[root@pluto] ~# dd if=/dev/zero of=/dev/ada4 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.006484 secs (161722615 bytes/sec)
[root@pluto] ~# dd if=/dev/zero of=/dev/ada5 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.006562 secs (159795317 bytes/sec)

That wiped my USB disk also, heh. Preparing a new USB disk and will report back.

 

[Digidoc]

How are you checking the sector size (within the GUI or from the command line)? I'm assuming you're doing everything from an SSH prompt, correct?

 

[Daisuke]

Ya, through a CentOS terminal, I connect with ssh.

Edit: Writing the 8.0.2 image to USB stick, as we speak. Before I boot it, is there any steps I need to do in order to make sure the /dev/da0 that holds the software is properly mapped to 4k sectors?

 

[Daisuke]

OK, everything looks much better now:

Code:

# gpart show
=>     63  7831467  da0  MBR  (3.7G)
       63  1930257    1  freebsd  [active]  (943M)
  1930320       63       - free -  (32K)
  1930383  1930257    2  freebsd  (943M)
  3860640     3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  3926538       - free -  (1.9G)

=>      0  1930257  da0s1  BSD  (943M)
        0       16         - free -  (8.0K)
       16  1930241      1  !0  (943M)

# diskinfo -v /dev/da0
/dev/da0
        512             # sectorsize
        4009754624      # mediasize in bytes (3.7G)
        7831552         # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        487             # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.

How do I make the /dev/da0 move to 64 sectors, instead of 63? That is the major problem. Once this is done, everything will be properly aligned. I'm thinking out loud:
# gpart add -b 8 /dev/da0

 

[ProtoSD]

How do I make the /dev/da0 move to 64 sectors, instead of 63? That is the major problem. Once this is done, everything will be properly aligned. I'm thinking out loud:
# gpart add -b 8 /dev/da0

da0 is your flash drive with FreeNAS, so there's nothing to worry about, you're done!

 

[Daisuke]

I wish. Here what the console reports, when I create a ZFS array with forced 4k sectors:

Code:

Nov  6 23:44:32 pluto freenas[1805]: Executing: dd if=/dev/zero of=/dev/ada0 bs=1m count=1
Nov  6 23:44:32 pluto freenas: 1+0 records in
Nov  6 23:44:32 pluto freenas: 1+0 records out
Nov  6 23:44:32 pluto freenas: 1048576 bytes transferred in 0.006478 secs (161865464 bytes/sec)
Nov  6 23:44:32 pluto freenas[1805]: Executing: dd if=/dev/zero of=/dev/ada0 bs=1m oseek=`diskinfo ada0 | awk '{print int($3 / (1024*1024)) - 4;}'`
Nov  6 23:44:32 pluto freenas: dd: /dev/ada0: short write on character device
Nov  6 23:44:32 pluto freenas: dd: /dev/ada0: end of device
Nov  6 23:44:32 pluto freenas: 5+0 records in
Nov  6 23:44:32 pluto freenas: 4+1 records out
Nov  6 23:44:32 pluto freenas: 4284416 bytes transferred in 0.078611 secs (54501388 bytes/sec)
Nov  6 23:44:32 pluto freenas[1805]: Popen()ing: gpart create -s gpt /dev/ada0
Nov  6 23:44:33 pluto freenas[1805]: Popen()ing: gpart add -b 128 -t freebsd-swap -s 4194304 ada0
Nov  6 23:44:33 pluto freenas[1805]: Popen()ing: gpart add -t freebsd-zfs ada0
Nov  6 23:44:33 pluto freenas[1805]: Popen()ing: gpart bootcode -b /boot/pmbr-datadisk /dev/ada0

That is totally wrong, it should be (or something like that):
# gnop create -S 4096 /dev/ada0
# gpart create -s gpt /dev/ada0 ...

 

[ProtoSD]

Ok, well I give up. What happened since post #11 when everything was fine except your confusion about the flash drive not being aligned?

 

[Daisuke]

Nothing was fine, since the beginning. The forced 4k sector option is a partial illusion. Here it is what ada0 reports (it should be a 4k sector, since I force it):

Code:

# gpart show ada0
=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834703     2  freebsd-zfs  (1.8T)

# diskinfo -v ada0
ada0
        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        3876021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.

That will result in poor performance as the OS sees the sector size as 512bytes, while in fact they are 4096. Yes, the actual pool is created with:

Code:

# gnop create -S 4096 /dev/ada0p2
# gnop create -S 4096 /dev/ada1p2
# gnop create -S 4096 /dev/ada2p2
# gnop create -S 4096 /dev/ada3p2
# gnop create -S 4096 /dev/ada4p2
# gnop create -S 4096 /dev/ada5p2

# zpool create -o cachefile=/data/zfs/zpool.cache -O aclmode=passthrough -O aclinherit=passthrough -f -m /mnt/nas -o altroot=/mnt nas  raidz /dev/ada0p2.nop /dev/ada1p2.nop /dev/ada2p2.nop /dev/ada3p2.nop /dev/ada4p2.nop /dev/ada5p2.nop
# zfs inherit mountpoint nas
# zpool export nas

but the rest of partition on each drive is stored on the old 512 bytes format, creating performance issues with the cache, swap, etc.

Anyone knows how to correct this issue with a manual build of the array? I wish I would know more about FreeBSD.

 

[Digidoc]

So I just checked my disks to see if I'm getting the same thing. I'm seeing something similar, but with one thing that's different:

Code:

ada0
        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        3876021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.

All except for one of the drives in my FreeNAS system is a 4K sector drive by default. Essentially my drives are doing the same thing as your drives are doing, except I have a #stripesize of 4096 (I created my RAIDZ2 stripe from within the GUI).

I don't think this is an issue with FreeNAS directly though. If you notice what it says above, the cylinders, heads and sectors reported are according to firmware.

Honestly? I've been running this system for three months now and I haven't seen any real degradation in performance that can be attributed to the sector size being mis-aligned. If anything it's been "stupid fast"; I get on average 90MB/sec read/write speeds from it (sometimes bursting to 120MB/sec) over a gigabit network using a CIFS share. It's fast enough that yesterday I tested editing photos I take directly from the CIFS share (through Lightroom 3 and Photoshop CS5). The photo was a Canon raw file taken with a Canon 1DsMk3, so it was a 21MB file. The resultant .PSD file ended up being 528MB when I was done with it. While it wasn't as fast as a locally connected SATAII HDD, it wasn't until the file started to get up into the 500MB range that I could see it slowing down. Prior to that it was taking less than 10 seconds to save the file over the wire (on average)

 

[William Grzybowski]

Dude,

There is nothing wrong here.

Yes, as you said the disks will report 512 sector size to the OS but that does not matter at all since the zpool will be created with 4096, you can check that by zdb| grep ashift

It must be 12, once its created it will always use 4096 (2^12)... Also all the partitions created by freenas are aligned in 4096 by default

#gnop create -S 4096 /dev/ada0
This will just create a dev /dev/ada0.nop that will emulate temporarily 4096 sector size:
diskinfo -v /dev/ada0.nop

And thats it, the GUI will create the pool with /dev/ada0.nop so the ashift can be 12 (instead of 9) and then remove it...

 

[Digidoc]

Thank you William! That's pretty much what I was thinking myself; the actual stripe is 4096 bytes.

If you look at his data above though, the stripe size doesn't show that it's at 4096 (where mine reports that it is). I'm wondering if that's because he created his stripe using the command line and I created mine through the GUI instead?

BTW, I tried the "zdb| grep ashift" command, and just as you said, it reports "12".

Dude,

There is nothing wrong here.

Yes, as you said the disks will report 512 sector size to the OS but that does not matter at all since the zpool will be created with 4096, you can check that by zdb| grep ashift

It must be 12, once its created it will always use 4096 (2^12)... Also all the partitions create by freenas are aligned in 4096 by default

#gnop create -S 4096 /dev/ada0
This will just create a dev /dev/ada0.nop that will emulate temporarily 4096 sector size:
diskinfo -v /dev/ada0.nop

And thats it, the GUI will create the pool with /dev/ada0.nop so the ashift can be 12 (instead of 9) and then remove it...

 

[William Grzybowski]

Thank you William! That's pretty much what I was thinking myself; the actual stripe is 4096 bytes.

If you look at his data above though, the stripe size doesn't show that it's at 4096 (where mine is though). I'm wondering if that's because he created his stripe using the command line and I created mine through the GUI instead?

Your does because the FreeBSD succeeds to detect the disk as a 4096 sector size, in some cases the OS can retrieve that info correctly, in some it doesn't (depending on the disk, firmware version, achi mode, etc)

For example, in my WD15EARS it shows as 512 sector size, still it works pretty ok with the 4k sector size checked...

 

[Daisuke]

Dude,

There is nothing wrong here.

Yes, as you said the disks will report 512 sector size to the OS but that does not matter at all since the zpool will be created with 4096, you can check that by zdb| grep ashift

It must be 12, once its created it will always use 4096 (2^12)... Also all the partitions created by freenas are aligned in 4096 by default

#gnop create -S 4096 /dev/ada0
This will just create a dev /dev/ada0.nop that will emulate temporarily 4096 sector size:
diskinfo -v /dev/ada0.nop

And thats it, the GUI will create the pool with /dev/ada0.nop so the ashift can be 12 (instead of 9) and then remove it...

Thanks for the info. What about the cache and swap files? They are used on the "slow" partitions created originally by FreeNAS, correct?