Are these performance numbers right?

[bwegrzyn]

I have an old FreeNAS system running on 9.10 with 6x3tb disks in raidz2. The pool is about 80% full and I've been in the process of building a more powerful system, but the performance numbers seem to be a little off.

System 1 (9.10):
CPU: AMD A10-7850K
Memory: 24gb
HDD: SATA3 6x3TB in raidz2 (79% full, mix of WD Reds and Seagate Barracudas)

Code:

[root@nas] /mnt/zpool/Media# dd if=/dev/zero of=tmp.bin bs=2048k count=64k && sync
65536+0 records in
65536+0 records out
137438953472 bytes transferred in 472.456482 secs (290902885 bytes/sec)
[root@nas] /mnt/zpool/Media# dd if=tmp.bin of=/dev/null bs=2048k count=64k && sync
65536+0 records in
65536+0 records out
137438953472 bytes transferred in 458.838108 secs (299536920 bytes/sec)

System 2 (11.1):
CPU: Dual Intel Xeon E5-2680 v2
Memory: 32gb
HDD: LSI 9220-8i 4x8TB in mirrored vdevs (5% full, all WD Reds)

Code:

root@nas:/mnt/storage/media # dd if=/dev/zero of=tmp.bin bs=2048k count=64k && sync
65536+0 records in
65536+0 records out
137438953472 bytes transferred in 43.458964 secs (3162499557 bytes/sec)
root@nas:/mnt/storage/media # dd if=tmp.bin of=/dev/null bs=2048k count=64k && sync
65536+0 records in
65536+0 records out
137438953472 bytes transferred in 16.426381 secs (8366964789 bytes/sec)

Based on the results of DD, System 1 reads and writes right around ~275mb/s which seems right. System 2 is writing at ~3000mb/s and reading at ~8000mb/s. I knew the performance of the new system would be better, but these numbers are mind blowing so I'm pretty sure they're wrong.

The total size of the file being written and read is 128gb, which is 4x the memory capacity so I don't think it's being cached. What gives?

 

[garm]

Turn off compression on the dataset you write to

 

[bwegrzyn]

Thanks, that did it. Makes sense now that I think about it... the zero data would get compressed before being written.

Code:

root@nas:/mnt/storage/media # dd if=/dev/zero of=tmp.bin bs=2048k count=32k && sync
32768+0 records in
32768+0 records out
68719476736 bytes transferred in 196.811110 secs (349164622 bytes/sec)
root@nas:/mnt/storage/media # dd if=tmp.bin of=/dev/null bs=2048k count=32k && sync
32768+0 records in
32768+0 records out
68719476736 bytes transferred in 170.247937 secs (403643520 bytes/sec)