No increased performance when adding more vdev to mirror

[Eagleman]

Why am I not seeing improvements in read speeds when switching from a mirror to a stripe?

Code:

Samsung 850 EVO (512GB) x4 striped

root@freenas:/mnt/benchmark # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 79.603552 secs (1348861702 bytes/sec)

root@freenas:/mnt/benchmark # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 67.587923 secs (1588659307 bytes/sec)

Samsung 850 EVO (512GB) x4 mirror

root@freenas:/mnt/easy/benchmark # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 157.263621 secs (682765550 bytes/sec)

root@freenas:/mnt/easy/benchmark # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 66.430672 secs (1616334423 bytes/sec)

 

[tvsjr]

Your sig doesn't provide a ton of info about your system. First, I assume you have compression disabled for these tests... if not, they'll be meaningless. Second, if you're directly connecting your drives to your motherboard, the C612 chipset has 2 6Gbps SATA channels. 1.6GB/sec = 12Gbps... it's possible you're simply maxing out the throughput of the SATA controller.

 

[Eagleman]

Compression was disabled, otherwise you would see numbers multiplied by 10. Do you have a source that the 612 chipset only has 2 6Gbps SATA channels?

 

[Eagleman]

I just switched all SSDs from the onboard sata to my 2 IBM m1015's, so 4 SSDs are now connected to one of the M1015's.

I am still not seeing an improvement:

Code:

root@freenas:/mnt/benchmark # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 69.143770 secs (1552911878 bytes/sec)

root@freenas:/mnt/benchmark # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 71.759518 secs (1496305795 bytes/sec)

 

[CookiesLikeWhoa]

If I'm reading this right you have 4 Samsung SSD's that were in a mirror (I'm assuming 2 vdevs with 2 drives each) and have now taken those same drives and put them into a striped configuration?

The max read you should be able to get out of the mirrored set is 1.1GB/s, and it looks like you are getting 1.6GB/s. Something does not seem right there.

 

[Eagleman]

Yes, 2 vdevs with 2 drives each when using the mirror.

>The max read you should be able to get out of the mirrored set is 1.1GB/s, and it looks like you are getting 1.6GB/s.

Write speeds should be 1.1GB/s max since they write to 2 vdevs at the same time. When reading it uses all disks in all vdevs together, so should reach 2.2GB/s.

 

[CookiesLikeWhoa]

Yes which means you're only reading from "2" disks not all four. If you were reading from all 4 then there is no point in going striped because it would be the same performance (reading from 4 disks).

 

[Eagleman]

Yes which means you're only reading from "2" disks not all four. If you were reading from all 4 then there is no point in going striped because it would be the same performance (reading from 4 disks).

Your understanding of raid 10 is wrong, reading occurs from all "disks" in the pool. Writing occurs to each "vdev" in the pool. So theoretically reading should be as fast as with a mirror. A stripe however has less overhead since there is less coordination between reading/writing.

I've fixed my issue, I thought all my PCIe slots were x8 but some of them were x8 but with only 2 or 4 channels.

Now i've got the following numbers:

Code:

stripe (8)

root@freenas:/mnt/benchmark # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 30.535875 secs (3516328975 bytes/sec)

root@freenas:/mnt/benchmark # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 27.939059 secs (3843156781 bytes/sec)


mirror (4*2 vdevs)

root@freenas:/mnt/benchmark # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 63.173548 secs (1699669971 bytes/sec)

root@freenas:/mnt/benchmark # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 34.976912 secs (3069858876 bytes/sec)

As you can see on the above numbers read speeds are almost the same, only write speeds get doubles when going stripe since you have more disks to write to.

I would still like some confirmation that the C612 chipset only has 2 SATA III lanes. I wasnt able to found anything about this while googling @tvsjr.