10GBe, stripe of mirrors, write is 2x faster then read ? Please help to sort this out.

[ysnk]

so striping has no effect on read either? more vdev should give better perf.

Typically the way to improve write performance is to try and cache your working set.

No effect, pretty much the same numbers...

 

[sfcredfox]

I'm not much of an expert yet, but I'm curious if a local drive test would be helpful or has already been done?

Have you confirmed whether cifs is playing a part?

I would be interested in seeing a DD write to the local pool, then he could do a read test to find out if its hardware issues or protocol or network issue right?

 

[ysnk]

Local drive is a

I'm not much of an expert yet, but I'm curious if a local drive test would be helpful or has already been done?

Have you confirmed whether cifs is playing a part?

I would be interested in seeing a DD write to the local pool, then he could do a read test to find out if its hardware issues or protocol or network issue right?

Local drive I used for test is a RAM drive, so it's definitely not a bottleneck...
How do I do DD write?

 

[sfcredfox]

I am probably out of my depth with ram drives. Not sure if the DD test is relevant then.

https://forums.freenas.org/index.php?threads/notes-on-performance-benchmarks-and-cache.981/

this has some good pointers for using DD. Have to ensure compression is disabled for writing zeros.

 

[ysnk]

DD write and read results:

Code:

# dd if=/dev/zero of=/mnt/pool1/tmp.dat bs=2048k count=50k    
51200+0 records in                                                            
51200+0 records out                                                           
107374182400 bytes transferred in 213.567299 secs (502765091 bytes/sec)       
# dd if=/mnt/pool1/tmp.dat of=/dev/null bs=2048k count=50k    
51200+0 records in                                                            
51200+0 records out                                                           
107374182400 bytes transferred in 149.011911 secs (720574493 bytes/sec)       

500 MB/s write, 720MB/s read...

 

[depasseg]

Is compression on or off for that dataset? Using a string of 0's compresses nicely and skews the test results. The recommended configuration for the test is to disable compression.

 

[Norleif]

If compression was on, he should see a lot more then 500MB/s, unless there really is a serious bottleneck in his system.

 

[sfcredfox]

If I'm not mistaken though, if the read rate directly from disk is 700ish, then this points to more of a network/protocol/initiator issue if he's seeing slow read on the initiator end, right?

 

[ysnk]

Is compression on or off for that dataset? Using a string of 0's compresses nicely and skews the test results. The recommended configuration for the test is to disable compression.

Compression was disabled for this test.

If I'm not mistaken though, if the read rate directly from disk is 700ish, then this points to more of a network/protocol/initiator issue if he's seeing slow read on the initiator end, right?

Yes, I'm sure it has nothing to do with the hardware. Most likely it's CIFS that slows it down. Is there any way to tweak it?

 

[sfcredfox]

I can't find anywhere you guys talked about the SMB version. OP, these guys might want to know what levels you have SMB set to? Min and max? I think Windows 7 does SMB2? I don't know how 10G play into that, since I don't have it :(

 

[ysnk]

SMB is set to 2.1 min, 3.0 max

 

[depasseg]

Are the dd results you posted in #25 run across the network to a CIFS share, or locally on the FreeNAS?

 

[ysnk]

Are the dd results you posted in #25 run across the network to a CIFS share, or locally on the FreeNAS?

I ran it in the freenas shell, so I guess locally.
Is it possible to run it across the network from Windows machine? How can I do it?

 

[ysnk]

Update: Disabling antivirus real-time protection increased the small files read speed by 25-30%. Large files read speed didn't change though, still 1/2 of write speed.

 

[L]

So what all this is saying to me, is...

locally read is better than write, over the network, write is better than read. Can you try with a different client? Something between the client and server is amiss. I have seen cases where the client kept aunthenticating on one or the other because of the way the permissions are set.