Pool performance with 3 mirrored vdevs

[Alex]

Hi,

I'm running V8 beta4 with 3 mirrored vdevs
Comparing to the performance that other people are getting Performance-Testing-with-DD mine seems slow.

Here is a dd test for write and read:

Code:

fs# dd if=/dev/zero of=/mnt/tank/prj/file bs=2M count=10000
20971520000 bytes transferred in 102.514991 secs (204570276 bytes/sec)

fs# dd of=/dev/null if=/mnt/tank/prj/file bs=2M count=10000
20971520000 bytes transferred in 60.366584 secs (347402795 bytes/sec)

System:
OS Version: FreeBSD 8.2-RELEASE-p2
Platform: AMD Opteron(tm) Processor 4170 HE (6 cores)
Memory: 12271MB
Load Average: 0.02, 0.16, 0.23
FreeNAS Build: FreeNAS-8.0.1-BETA4-amd64

MB: SUPERMICRO MBD-H8SCM-F
6 onboard SATA in AHCI mode
2x Intel 82574L Onboard LAN

Pool:
6 x WD20EARS, aligned to 4k, sleep timer set to 5 min.

Code:

fs# zpool status
 pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          mirror                                        ONLINE       0     0     0
            gptid/8b3a9287-bd6a-11e0-ad3a-00259021fb4e  ONLINE       0     0     0
            gptid/8c032841-bd6a-11e0-ad3a-00259021fb4e  ONLINE       0     0     0
          mirror                                        ONLINE       0     0     0
            gptid/9e616f00-bd6a-11e0-ad3a-00259021fb4e  ONLINE       0     0     0
            gptid/9f749373-bd6a-11e0-ad3a-00259021fb4e  ONLINE       0     0     0
          mirror                                        ONLINE       0     0     0
            gptid/b4142bf6-bd6a-11e0-ad3a-00259021fb4e  ONLINE       0     0     0
            gptid/b59c34ba-bd6a-11e0-ad3a-00259021fb4e  ONLINE       0     0     0

errors: No known data errors


fs# zdb | grep ashift
                ashift=12
                ashift=12
                ashift=12

Any thoughts?
Is there anything I can try to tweak?

Thanks,
Alex

 

[matthewowen01]

that looks about right. writing to 3 drives at 1.6 gb/s and reading at 2.8 gb/s.

if you don't have data yet, try a raidz2 instead.

 

[Alex]

at the link above guys are getting almost twice of that:

Code:

test-mirror-raidz                               ONLINE       0     0     0
          mirror                                        ONLINE       0     0     0
            gptid/f561fd45-951a-11e0-a1a7-001b218b578e  ONLINE       0     0     0
            gptid/f5dcb8f8-951a-11e0-a1a7-001b218b578e  ONLINE       0     0     0
          mirror                                        ONLINE       0     0     0
            ada2p2.nop                                  ONLINE       0     0     0
            ada3p2.nop                                  ONLINE       0     0     0
          mirror                                        ONLINE       0     0     0
            ada4p2.nop                                  ONLINE       0     0     0
            ada5p2.nop                                  ONLINE       0     0     0

dd if=/dev/zero of=/mnt/test-mirror-raidz/ddfile bs=2048k count=10000
20971520000 bytes transferred in 50.566710 secs (414729770 bytes/sec)

dd of=/dev/null if=/mnt/test-mirror-raidz/ddfile bs=2048k count=10000
20971520000 bytes transferred in 36.187830 secs (579518584 bytes/sec)

My understanding is that at least while reading zfs should be able to use all 6 drives to fetch different parts of the file/data, at least when copying single big file.

And even reading from only 3 drives should be faster I think.
Am I missing something?

Thanks,
Alex

 

[matthewowen01]

well, ZFS does not guarantee where it places it data. when you write data, it will be written to no less than one vdev. so in their setup, they are guaranteed to have their data split across every disk, you are not. however, you'll get better performance for simultaneous transfers as you can read and write to different vdevs at the same time.

 

[Alex]

It seems that these last results they've got on the same pool as mine, 3 mirrored vdevs.
The pool was named "test-mirror-raidz" but it's not raidz.

Thanks,
Alex

 

[matthewowen01]

i see you have 12 GB of ram, 2 4's and 2 2's i guess, are they installed right so they are running in dual channel?

the only other thing else that stands out is you're running beta 4 while he was running beta 2.

 

[Alex]

yes, memory installed like 2 4 2 4.
Good point about beta versions, I guess I'll wait till the next release, but
from playing around with this box for last 3 month I think it's something else.
I was trying different pool configs with Freenas8 RC and Release. Never was able to get to the results that I think would make sense.
Is it possible that there is some kinda problem with FreeBSD 8 and this motherboard's sata controller?

Thanks,
Alex

 

[Alex]

Another 2 things that are obviously different:

1 - I'm running AMD build, is there anyway this can make a difference?
Do you think it's worth trying to run these drives in the same config on an intel cpu/mb?

2 - WD greens vs Seagate greens.


Thanks,
Alex

 

[matthewowen01]

the 2 versions are not actually intel vs amd, it's 32 bit vs 64 bit. intel owns the 32 bit spec and amd owns the 64 bit spec. i think the 32 bit version should be dropped, but that's me.

are you sure that you did the 4k thing right? can you recreate it without enabling them?

 

[Alex]

I enabled the 4k switch when creating the pool and adding vdevs,
and from what I've read, this result -

Code:

fs# zdb | grep ashift
                ashift=12
                ashift=12
                ashift=12

means that it's aligned proper.

Do you mean try to create the pool without the 4k switch?

Alex

 

[matthewowen01]

i unfortunately don't have any real experience with 4k drives and I'm out of ideas at the present.

sorry, hopefully someone else has some ideas.