ZFS and 4K

[cyberjock]

Well, the zpool is blank and I can change the ashift. I guess I'll let it finish :P I was expecting it to take a few hours at the most. I did follow the directions and have everything set correctly. We'll see how this all goes I guess.

 

[tingo]

Another, also with pool in use.
FreeNAS 8.2.0-RELEASE-p1-x64, Asus E35M1-I, 8 GB RAM, 6 x Samsung HN-M101MBB, RAIDZ1

values

Code:

[root@kg-f3] /mnt/zstore/home-tingo/test# sysctl hw.physmem
hw.physmem: 8153223168
[root@kg-f3] /mnt/zstore/home-tingo/test# sysctl kstat.zfs.misc.arcstats.c
kstat.zfs.misc.arcstats.c: 5846991325

test

Code:

[root@kg-f3] /mnt/zstore/home-tingo/test# zdb -C zstore | grep ashift
                ashift=9

    Command line used: iozone -r 4k -r 8k -r 16k -r 32k -r 64k -r 128k -s 6g -i 0 -i 1 -i 2
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                  
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         6291456       4  104553   46250   175951   149704     918     639                                                         
         6291456       8  126172   47481   169062   120174    1899    1294                                                         
         6291456      16  129753   47980   204112   132605    3604    2528                                                         
         6291456      32  158847   53023   229718   188100    6081    4600                                                         
         6291456      64  198143   61299   194177   170953    9644   10262                                                         
         6291456     128  225659  205506   156392   162202   14490  243319                                                         

iozone test complete.

The test took forever on this box.
HTH

 

[paleoN]

Did you miss jgreco's [post=46188]earlier post[/post]? Come to think of it you likely already had the test running. Unfortunately, the results as is aren't useful. You should re-read the [post=43638]first post[/post].

To use the supplied command you need to create a tunable hw.physmem with a value of 4G or 4294967296 (which is 4G in bytes), disable any & all kmem / arc_max type tunings and reboot.

That being said the FreeNAS devs want to see the performance difference between ashift=9 & ashift=12 pools on the same hardware. Which is difficult to do with a pool in use unless you want to backup & restore the whole thing.

 

[paleoN]

So how long should these tests take?

I take it you missed [post=43712]post #5[/post].

It took two or three days to run on my old Barracuda 400GB pool (it's okay, laugh).

One full test run on my drives took about 14? hours, give or take a few hours.

The vast majority of that time is spent on the reclen=4 random read/write tests. The rest goes quite a bit faster, relatively speaking.

 

[cyberjock]

I take it you missed [post=43712]post #5[/post].

Nope. I saw that post. But he was referring to only a small part of the test. I was curious about the entire test. :)

Edit: The 4k test seemed to take the longest.. by far. It took more than 6 hours(went to bed so not sure when it finished) for the 4k test. But after 9 hours of sleep the whole test was complete. Switched to ashift 9 and I am rerunning the test.

 

[cyberjock]

Well, I just got a smack in the face:

18xWD20EARS (4k sector size drives)

Did a whole test with a 4k sector size. Recreated the zpool on the same drives without checking the "force 4k sector size". Ran the test. After the test was over I looked at the 2 results and I was a little puzzled. So I thought I'd check my sector size. FreeNAS apparently uses a 4k sector size if it detects 4k sector drives. I'm not sure how I could force a 512byte sector or if I could would the results be helpful at all. If the results will be helpful I can redo the test but I'll need to know how to force a 512byte sector size.

So yeah.. I guess I'll have to wait for the other zpool to be empty. It has 512 byte sectors for sure and I can provide both sides of the spectrum.

 

[paleoN]

I'm not sure how I could force a 512byte sector or if I could would the results be helpful at all. If the results will be helpful I can redo the test but I'll need to know how to force a 512byte sector size.

Code:

zpool destroy tank

zpool create tank raidz2 da0p2 da1p2 da2p2 da3p2 da4p2 da5p2 raidz2 da6p2 da7p2 da8p2 da9p2 da10p2 da11p2 raidz2 da12p2 da13p2 da14p2 da15p2 da16p2 da17p2

Or however the pool was configured.

 

[William Grzybowski]

To _force_ 512 bytes you need "gnop create -S 512 daXp2", otherwise it will use the detected by the OS (diskinfo -v daX).
Forcing 512 does not seem smart.

 

[cyberjock]

Yeah, I'm going to give up trying to test this zpool. I don't think anyone cares about 512-byte sectors on 4k drives rather than 4k sectors on 512-byte sector drives. So I'll provide results from the other zpool in a week or so when I get the spare drives together.

 

[Stephens]

Yeah, I'm going to give up trying to test this zpool. I don't think anyone cares about 512-byte sectors on 4k drives rather than 4k sectors on 512-byte sector drives. So I'll provide results from the other zpool in a week or so when I get the spare drives together.

It's useful information, just not for the purposes of this thread. I have a pool with 5 4K drives and 1 512-byte drive. FreeNAS defaulted to 512-byte, which means the majority of the pool is underperforming. How much? I don't know. For networking related traffic, the pool is still faster than my gigabit network. I suspect things like scrubs could go faster with optimal sizes. But for this thread, the big question is how much defaulting to 4K would impact legacy (512-byte) drives/pools.

 

[cyberjock]

I'm not sure how useful it would be. All of my drives are 4k drives, which is why FreeNAS defaulted to the 4k sector size. Honestly, I'm a little scared to think how long the test would take if I forced a 512-byte sector size of my zpool. As it was the whole series of tests took something around 15-20 hours.

It's kind of a mixed match. I'd expect anyone buying even one 4k sector drive will have enough performance from their zpool to exceed gigabit network speeds. I dunno. I'll think about it. Somehow I think the test would be painful to wait on. The thought of the test taking 30+ hours makes me shiver.

 

[Stephens]

I wasn't suggesting you should run it.

 

[cyberjock]

I wasn't suggesting you should run it.

I didn't think you were suggesting I should.

I've decided not to run the test with my 4k zpool. As soon as I get my hands on a soon-to-be-retired FreeNAS zpool I'll run the tests again.

It just sucks that I spent so much time on the tests to figure out I had no actual useful comparison. :(

 

[jgreco]

So hypothetically, if it already detects 4K drives and uses 4K for them, and uses 512 for 512 drives, that's already pretty optimal from what I can tell, right? The only ambiguity is if there's a mix, because then it seems to default to 512. Right?

The only problem with that is that I don't recall seeing it detect this automatically in the manner described. Maybe the drives I was doing it with weren't sufficiently new-and-sophisticated though.

 

[cyberjock]

I remember when 4k was brand new some 4k drives would still report themselves as being 512bytes so that some OSes wouldn't freak out. Many additionally had a jumper for Windows XP installations to shift the sectors to align default XP installations to align to the sector boundaries.

 

[Stephens]

@noobsauce80, It's the "512e" I mentioned earlier, which Windows had to account for as well.

@jgreco, I think the wording of the option "Force..." isn't coincidental and you describe it correctly. FreeNAS does try to detect, but it gives you the option of forcing it. Since I *DO* have a system with mixed 4k/512-byte drives, I should have used that option when I created my pool (but somehow didn't). But I still look forward to people with 512-byte drives doing the tests both ways because I still say 4k should be the default (even in a mixed drive setup) and that selecting 4k for a 512-byte drive has minimal, if any, negative impact. It also positions the pool well going forward (replacing drives, etc.).

 

[jgreco]

@noobsauce80, It's the "512e" I mentioned earlier, which Windows had to account for as well.

@jgreco, I think the wording of the option "Force..." isn't coincidental and you describe it correctly. FreeNAS does try to detect, but it gives you the option of forcing it. Since I *DO* have a system with mixed 4k/512-byte drives, I should have used that option when I created my pool (but somehow didn't). But I still look forward to people with 512-byte drives doing the tests both ways because I still say 4k should be the default (even in a mixed drive setup) and that selecting 4k for a 512-byte drive has minimal, if any, negative impact. It also positions the pool well going forward (replacing drives, etc.).

I agree with your conclusion, but the tests I did for 4k on 512-byte drives did seem to indicate some significant degradation, and as a pool that is built out of 512 drives is already probably an older/slower pool, heaping something that slows it down further ... requires a bit of consideration.

If, going forward, the pool will be expanded through replace/resilver, then 4k is definitely going to be a win - but only eventually.

For a pool that will never be expanded, though, and is already at a disadvantage because it's older technology, the 512 seems better.

 

[Suiname]

How can I check what block configuration my zpool is using? I couldn't see it through the GUI and don't remember it giving me a choice when I created it.

 

[sabreofsd]

How can I check what block configuration my zpool is using? I couldn't see it through the GUI and don't remember it giving me a choice when I created it.

SSH to the box and run:
zdb -C BackupMainVol1 | grep ashift

12 means that its 4k :)

 

[tingo]

Did you miss jgreco's [post=46188]earlier post[/post]? Come to think of it you likely already had the test running. Unfortunately, the results as is aren't useful. You should re-read the [post=43638]first post[/post].

Nope, I didn't miss that post. As I stated in mine, I provided the results I could (with reasonable effort), for what they are worth.
In an ideal world I would have no limits on money, space (for the equipment) and time.
In the real world, I provide what I can with what I have, and hope it will be at least somewhat useful to others.