Bad read perfomance

[hugovsky]

Hi all.

Hope someone could give me a hand on this. Only tested with this version. Can't compare with older versions of freenas.

Build FreeNAS-9.2.1.3-RELEASE-x64 (dc0c46b)
Platform Intel(R) Pentium(R) CPU G2030 @ 3.00GHz
pool and zfs upgraded
RAIDZ2
4x WD red 3,5"
8GB ram non ecc (yet)
no optimizations whatsoever
Intel Cougar Point AHCI SATA controller

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD10EFRX-68JCSN0 01.01A01> ATA-8 SATA 3.x device
ada0: Serial Number WD-WMC1U5671723
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>

ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD10EFRX-68PJCN0 01.01A01> ATA-9 SATA 3.x device
ada1: Serial Number WD-WCC4J1725751
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>

ada2 at ahcich4 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD10EFRX-68JCSN0 01.01A01> ATA-8 SATA 3.x device
ada2: Serial Number WD-WMC1U5213900
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>

ada3 at ahcich5 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD10EFRX-68JCSN0 01.01A01> ATA-8 SATA 3.x device
ada3: Serial Number WD-WMC1U5672714
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>

Problem:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 701.048828 secs (153162202 bytes/sec)

dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 1615.517868 secs (66464249 bytes/sec)


The dataset does not have compression enabled.
TOP coomand data and graphs from disks attached.

 

[Yatti420]

Can you list full specs? Did you run tests on the drives before putting them into production use?? What about memtest?

 

[hugovsky]

memtest gives no errors 20+ hours working. I'll post exact specs later when I'm with the machine. I haven't tested speed with single drive. But I can try to move data to another location and test drives one by one. Do you think it could be a problem with the drives? There's nothing on smart. Whole system is relatively new, maybe 2-3 months.

 

[SweetAndLow]

You can test reading from each individual disk using /dev/sdaX. Just don't write to that device or you will lose your pool.

 

[hugovsky]

Full specs:

CPU: Intel® Pentium® Processor G2030 (3M Cache, 3.00 GHz) (ECC suport for future M/B)
http://ark.intel.com/products/74749/intel-pentium-processor-g2030-3m-cache-3_00-ghz

M/B: Asus P8H61-MX R2.0
https://www.asus.com/Motherboards/P8H61MX_R20/#specifications

Mem: 8GB Kingston non-ecc

Disks: 4x WDC WD10EFRX 1TB

Intel® H61(B3) chipset : 4 x SATA 3Gb/s port(s), blue

Intel NIC

 

[hugovsky]

You can test reading from each individual disk using /dev/sdaX. Just don't write to that device or you will lose your pool.

Sorry, can you explain better, please?

 

[hammong]

Performance looks OK to me.

You have compression turned off, so your write speed of 153 MB/sec with double parity (RaidZ2) and 5900 RPM reds looks about right. If you turn compression on, your read/write speeds might increase if the disk content is highly compressible. You would need to experiment with it on (lz4) and off and see how it goes.

Reading back the temp file and writing back to the same set of disks with double parity is showing the limits of your Pentium G2030 and 8GB of RAM - basically a torture test. You're still getting 66+ MB/sec, which is nothing to sneeze at.

With 4 disks, I'd probably run a 2x2 Mirror instead of RAIDz2 if performance is your top consideration.

Greg

 

[cyberjock]

2x2.. what language is that? That's nothing like how ZFS RAIDs are explained. How about dual mirrored vdevs.

And for home use, RAIDZ2 is probably a better choice than mirrored vdevs.

 

[hammong]

Dual mirror, 2x2 - what's the difference? It's two sets of two disks mirrored. How about a comment re: the OP question about performance, instead of picking out my terminology? LOL.

Greg

 

[cyberjock]

Dual mirror, 2x2 - what's the difference? It's two sets of two disks mirrored. How about a comment re: the OP question about performance, instead of picking out my terminology? LOL.

Greg

And you think that I have an answer to provide and haven't posted it so I can go fencing with you?

Sorry, but I'd have thought that my 9600+ posts would have made it clear if I had something to offer I'd have commented.

 

[hugovsky]

CPU doesn't seem to be the problem nor does ram as shown in the screen I've attached in the first post. Performance isn't really an issue but I think with this setup, I should have 100 MB/sec read and write. Don't you?

 

[cyberjock]

CPU doesn't seem to be the problem nor does ram as shown in the screen I've attached in the first post. Performance isn't really an issue but I think with this setup, I should have 100 MB/sec read and write. Don't you?

Well, to be honest.. you haven't yet provided ANY numbers involving transfer rates yet. Yes, you provided the reporting screenshots, but that says nothing else except the disks do about 80MB/sec. That's NOT an indication of transfer speeds across the LAN(which is generally what people care about).

So how about you mention what protocol you are using and what transfer rates you are actually getting? Because right now, with the info you've provided, there's nothing to give me any indication that your system is performing "poorly"(for whatever definition of "poorly" you have. Individual disk performance is NOT equal to network performance.

 

[hugovsky]

I've never mentioned NETWORK performance. I'm talking about DISK performance:

"Problem:

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 701.048828 secs (153162202 bytes/sec)

dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 1615.517868 secs (66464249 bytes/sec)"

This is what I'm talking about from the start. Based on this: http://forums.freenas.org/index.php?threads/notes-on-performance-benchmarks-and-cache.981/ seems like my hardware should do more. I'm just asking if it really my problem or is it normal. Shouldn't my read speed be higher?

 

[cyberjock]

Ok.. and here's why I asked you to provide actual throughput numbers for a protocol. Because the max a Gb connection can do theoretically is about 120MB/sec. So 2 disks can hit that limit with your benchmark. You have 4 disks. So this is where I ask the question "whats the problem". You aren't realizing you're bottleneck isn't going to be the disks, it's going to be your LAN.

As for individual disks, I don't even look at them until someone can actually prove that their configuration with whatever protocol they choose, isn't performing as expected. Disks will perform differently as firmware changes, the number of platters and heads change, etc. At the end of the day, for your setup with just 4 disks, all you'll need is a measly 30MB/sec from each disk to make the bottleneck for your setup the LAN and not the pool or its disks.

To me, you are getting way to far ahead of yourself with the "testing" you are doing. You haven't proven there's a problem.. at all.

 

[hugovsky]

I agree with you. Maybe I didn't explain myself very well. Those numbers are not from single disk benchmark. This are values I'm getting from writing and reading from my raidz2 pool. You say 30MB/sec. I'm not even hitting 20 MB/s per disk. Am I seeing this wrong? btw, transfer speed with nfs hits 30-40 MB/s from nas to linux server and macbook on gigabit network.

edit: corrected a few typos.

 

[cyberjock]

Wow.. then you do have a problem. I'd have expected far better.

So I'd try doing dd reads and dd writes to the invidual disks for benchmark testing... Rule out a disk itself as a problem

 

[hugovsky]

Thanks… that's what I was trying to know. Well, not the part of having a real problem…. but at least, now I can confirm I have a problem. Is destroying the pool the only way to test the disks?

 

[cyberjock]

You can do dd read tests without destroying your pool. Write test will require you to destroy your pool. :(

 

[hugovsky]

Raw read results to the 4 disk taken simultaneously:

dd if=/dev/ad4 of=/dev/null bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 145.263991 secs (147833171 bytes/sec)

dd if=/dev/ad6 of=/dev/null bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 151.244793 secs (141987278 bytes/sec)

dd if=/dev/ad8 of=/dev/null bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 142.655994 secs (150535816 bytes/sec)

dd if=/dev/ad10 of=/dev/null bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 143.723296 secs (149417924 bytes/sec)

So far, everything seems normal. Where should I go now?

 

[hugovsky]

I just took this snapshot from gstat while running this:

dd if=tmp.dat of=/dev/null bs=2048k count=50k

Why are this showing disks 100% busy if they're only at 12MB/s?