What the ....? Big system SLOWER than small system

[davegibson]

Hello all,

Just a quick shot-in-the-dark question: can anyone explain why this may be happening?

I have a smaller system, single little i3 CPU, 12GB RAM, and 8x3TB SATA disks on an LSI SAS9211-8i. These disks are in one big RAIDZ pool (7 usable space). This system, under iSCSI load, is known to have latency issues on the array.

Thus, I built a bigger system: ASUS Z8NA-D6 mobo w/dual Xeon E5504 and 48GB RAM, and 16x3TB SATA disks on two LSI SAS9211-8i cards. This system has what I would consider a better-optimized array: 3 pools of 5 drives each in their own raidz pool, all in the same pool for a total of 15 drives with 12 drives usable space. This should get 12 drives of striping (vs. 7 drives on smaller system) and 3 pools worth of IOPS increase.

But when I run a dd to test speeds, the smaller system is faster...!

dd speed examples (using dd if=/dev/zero of=/mnt/Volume/testfile bs=1024 count=50000, etc.):
---------------------------
Small system, write speed:
51200000 bytes transferred in 0.305716 secs (167475676 bytes/sec)
Small system, read speed:
51200000 bytes transferred in 0.130692 secs (391760766 bytes/sec)
---------------------------
Large system, write speed:
51200000 bytes transferred in 0.466420 secs (109772267 bytes/sec)
Large system, read speed:
51200000 bytes transferred in 0.198514 secs (257916338 bytes/sec)
---------------------------
SO:
Any ideas why this larger system isn't kicking the crap out of the smaller one? The larger system is showing no signs of any trouble; clean messages file, etc.

I'm a little stumped...

Thanks!
-Dave

 

[survive]

Hi davegibson,

All you are doing in you "dd" tests is seeing how fast you can write to memory (assuming my math is correct). You need to write out a much bigger file if you want to see how fast your disks are!

So you are essentially benchmarking your RAM so it's not surprising the i3 is faster.....odds are you are using faster memory in it. I would expect you are using (possibly slower) registered memory in the bigger system and with such a tiny test the "cost" of the register is more pronounced.

Try upping the count and or blocksize so you are writing out a file larger than the installed memory.

-Will

 

[davegibson]

Thanks Will. I just tried running a 100GB file (dd if=/dev/zero of=testfile bs=1024 count 100000000) and got similar results. Here they are:

Small system:
write speed 797.658005 secs (128375819 bytes/sec)
Large system:
write speed 1115.687705 secs (91781956 bytes/sec)

I should also point out that the small system is under load (has 75 VMs currently running on it) and the large system is idle.

This really seems wrong to me.

Another thought: In the larger system, Seagate 3TB 7200rpm drives are in use, but the smaller system has Hitachi 3TB 7200 drives. Not certain of the model numbers, but I may try experimenting with Hitachis later. I don't believe either are low-power drives... but uncertain at this time.

Thanks for any thoughts and replies!
-Dave

 

[peterh]

is seems a little slow overall, however differing measure methods might be misleading

I use bonnie with a large file ( larger then mem > and is experiences > 200MB/s write speed on 8G machines
( zfs varous configs some get substantionally more then 200GB/s ) All with sata controllers ( no hardware raids !)

 

[survive]

Hi davegibson,

What version are you running?

-Will

 

[davegibson]

Thanks guys. The larger system is running 8.0.4; the smaller (but strangely faster) one is running 8.0.2-release.

-Dave

 

[survive]

Hi davegibson,

8.0.4 release? There was some tuning done on the 8.0.4 betas that impacted performance that were removed in the release.

-Will

 

[davegibson]

Yep, release bits, grabbed yesterday. Confirmed 8.0.4-RELEASE.

I see my LSI adapter is still in IR mode; I'm changing that now, and retesting.

 

[davegibson]

Okay, so I've set the HBAs to IT mode. I still get the same speeds; write speed about 110MBps. I've tried making 3 5disk RAIDZ pools, two 8disk pools, and one huge 16disk pool. All get roughly the same speeds using the dd speedtest at /mnt/MainPool.

So back to the bottom line: two different systems, using same power supplies/cases/SATA backplane/LSI 9211 8i, but different in motherboard/cpu/ram. Both nice Asus motherboads. And the dual quadcore cpu system with 48GBs is quite slower than the single dualcore cpu with 12GB RAM.

I'm stuck. Any ideas?

Thanks,
Dave

 

[b1ghen]

Did you try with a larger block size using dd? I think 1k is very small to get some good numbers.

I have a 6 Seagate 2TB Green drives in a RAIDZ2 and increasing the block size gave big improvements, for me. I was thinking the brand of disks also differs between your setups, maybe the Hitachis respond differently using smaller block sizes with all this 4k sector emulation going on?

Here are my results from 512 bytes to 128k bytes (I don't know what happened to my 32k one but it was slower), after 128k I saw no improvements, it kept around 265 - 275 MB/s up to 256M block size where I stopped testing.

dd if=/dev/zero of=/mnt/tank/testfile bs=512 count=200000000
102400000000 bytes transferred in 1584.639886 secs (64620360 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=1k count=100000000
102400000000 bytes transferred in 992.529623 secs (103170724 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=2k count=50000000
102400000000 bytes transferred in 656.292066 secs (156028094 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=4k count=25000000
102400000000 bytes transferred in 549.999603 secs (186181953 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=8k count=12500000
102400000000 bytes transferred in 496.744801 secs (206142067 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=16k count=6250000
102400000000 bytes transferred in 453.236900 secs (225930413 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=32k count=3125000
102400000000 bytes transferred in 497.709476 secs (205742516 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=64k count=1562500
102400000000 bytes transferred in 389.092841 secs (263176263 bytes/sec)

dd if=/dev/zero of=/mnt/tank/testfile bs=128k count=781250
102400000000 bytes transferred in 358.459472 secs (285666883 bytes/sec)

 

[b1ghen]

You made sure that you selected 4k sectors when creating both pools? I assume they must be 4k sector drives since they are 3TB.

Also I don't know if this load utilize multithreading or not, if it is single threaded one core of the Core i3 is probably way faster than one core of the Xeon, there is both an architecture and clock speed advance favoring the i3 if it's single threaded.

 

[davegibson]

The new sizing for dd seems to be painting a very different picture; my newer arrays are now looking faster than the old arrays. Thanks! I guess the real testing should be on iSCSI/NFS/etc. IO.

I appreciate the help on this one! I was hesitatnt to deploy the box until I felt confident that it was properly optimized, but we're moving forward with deployment.

Thanks again!
-Dave

 

[b1ghen]

The new sizing for dd seems to be painting a very different picture; my newer arrays are now looking faster than the old arrays. Thanks! I guess the real testing should be on iSCSI/NFS/etc. IO.

I appreciate the help on this one! I was hesitatnt to deploy the box until I felt confident that it was properly optimized, but we're moving forward with deployment.

Thanks again!
-Dave

Sounds great that it got some more reassuring results
And you are correct that benchmarking the way it's intended to be run is key. Local benchmarks are not really indicative of how it will work over a network protocol.