R620, XCP-NG and slow read performance

[RobMil]

Hello all,

I'm new to using FreeNas and would like to know if anyone can help me with diagnosing my slow read performance.

I currently have FreeNas setup as a VM on my xcp-ng server. The VM is configured with 64GB of ram, and direct access to the raid card that was flash into IT mode. I have 10 WD 2.5 blue 2TB drives attached to it in one Zpool using RAIDZ2.

My write speed is good. I performed a DD check directly on the FreeNas box and get a write throughput of 392MB/s, but my read speeds are on the slow side at 93MB/s. I'm just curious if I have something miss configured. I have syncing turned off, but I'm kinda of lost on what else to do. I know the WD Blue 2.5 drives are slow, but with 10 of them I figured the read speed would be a bit better. If I transfer a file off the NAS to my desktop I get a throughput of around 30MB/s using a SMB share. If I recopy the file a second time once it has been cached I get a full 110MB/s. My transfer speed to the NAS is 110MB/s, which makes sense it is just loading it to the memory. It seems like I need a L2ARC, but how much help would that truly be?

Thanks for any help you can provide!

 

[RobMil]

Correction, I provided 32GB of ram to the VM.

 

[RobMil]

I was just doing some further testing it seems to working well at the moment. Netdata reports I'm get read speeds above 120MB/s and my transfers are between 90-110MB/s. Maybe after I un-sync'd the entire pool it took time for FreeNas to get everything in order and it works good enough the my 1Gb connection.

 

[PhiloEpisteme]

I was just doing some further testing it seems to working well at the moment. Netdata reports I'm get read speeds above 120MB/s and my transfers are between 90-110MB/s. Maybe after I un-sync'd the entire pool it took time for FreeNas to get everything in order and it works good enough the my 1Gb connection.

Based on this I think part of the issue might have just been that data was not yet in the ARC and it needed a bit more time. Keeping an eye on the ARC hit ratio will help inform whether an L2ARC will help.

Of course, the best performance will be had by giving it more ram.

 

[RobMil]

How does the ARC work? It needs to get filled by the drives when copying, but if the read speeds are not much higher than the transfer speed the arc never really gets filled during the first copy. Afterwards, it is definitely all arc because there is no reading of that file.

 

[PhiloEpisteme]

How does the ARC work?

Here is a nice set of slides I just found on the topic. I can't promise they don't over simplify or are out of date in any way but you might find it helpful.

 

[RobMil]

Is there anything I can do to increase my read performance of my HDDs? I was under the impression that read speed was increased with RAIDZ2 and write speed was about as fast as a single hdd. But, I assume I'm not correct with this? Thanks again for the help!

 

[WOOGI]

Hate to dig up an older thread, but I am seeing the excat same issue on basically the same setup.

@RobMil - Did you ever find a solution?

My Setup

Freenas Box
2xE5645 Xeon CPU's
72gb of RAM
12x2TB 7.2K SAS Drives (6GB/s) in various Pool Setups (RAIDZ, Z2, Mirror etc)
10gb Link to Xen Server