Simultaneous Reads and Writes Performance

[FlangeMonkey]

Hi Guys,

So I was doing some performance testing on my MicroServer with 4x 3TB WD Reds, 16GB RAM and noticed an interesting performance statistic and wanted to see if anyone is getting similar results.

Average:
Write: 175-207MBps
Read: 202-227MBps
Simultaneous Write and Read: Write - 85MBps | Read - 26MBps

I'm looking at the simultaneous reads and writes and that's probably Seeks or Randoms I need to look at, but any idea's... should this be expected?

 

[EdvBeratung]

Since I am also using 4x WD Red 3TB on a test system I would be happy to give you my test results but to compare them with your results basically all relevant information is missing.
- Did you run the tests locally or over network? If Network what protocol did you use?
- What tools did you use to test, what configuration (record size, data amount, etc)?
- What FreeNAS version are you using?
- What RAID level are you using
- Do you have a caching HDD controller?
- Any other HDDs (SLOG, L2ARC)?
- Sync or async writes?
- Status of atime, compression, deduplication?

Without this information there is no chance to compare any performance results.
On my test system for example I am getting between 1.2 MB/s (sync writes, 4 kB record size, no SLOG) and 278 MB/s (async writes), up to 1800 MB/s (for async writes small enough to fit in the cache). Those numbers don't mean anything without knowing the context.

 

[FlangeMonkey]

Sorry, I completely forgot to reply...

- Did you run the tests locally or over network? If Network what protocol did you use?

Local
​

- What tools did you use to test, what configuration (record size, data amount, etc)?

DD to write sequential file
DD to read sequential file
DD to read sequential file while using DD to write another file
At the same file as the above, I also used zpool iostat 30
​

- What FreeNAS version are you using?

Latest 9.2.1.8
​

- What RAID level are you using

RAIDZ1
​

- Do you have a caching HDD controller?

Nope, all JBOD
​

- Any other HDDs (SLOG, L2ARC)?

Currently no SSD SLOG/ZIL or L2ARC
​

- Status of atime, compression, deduplication?

atime is default (on), no compression or dedup​

 

[solarisguy]

@FlangeMonkey, NAS = Network-attached storage, try timing a copy of some very large (i.e. substantially larger than 16GB) file using the client (Linux, Mac, Windows or whatever you are using...)

 

[FlangeMonkey]

@FlangeMonkey, NAS = Network-attached storage, try timing a copy of some very large (i.e. substantially larger than 16GB) file using the client (Linux, Mac, Windows or whatever you are using...)

That isn't going to help. I'm testing to file system not everything that sites above that.

I discovered that my 4x3GB drives where running at 3Gbps rather than 6 Gbps... Just saying!

I just built a new box with 5x4tb WD Reds in RAIDZ1 on a LSI 9211-8i (SATA3 6Gbps). I did some further testing (compression disabled, no dedup, no ZIL or L2ARC), again sequential read/writes using dd, with outputs from iostats and the results are:

Write: 340M
Read: 446M
Simultaneous Write and Read: Write - 217M | Read - 74.0M

 

[solarisguy]

1. Five 4TB hard drives in RAID-Z1 = bad idea. A better one: Six 4TB hard drives in RAID-Z2

2. If you are testing only and only the filesystem, then you are in the wrong place. Nobody here is a ZFS developer. Also the results of your testing are not really relevant in the real world. A hint: try limiting the writes to 130MB/s and see what the reads are going to be...

 

[FlangeMonkey]

1. Five 4TB hard drives in RAID-Z1 = bad idea. A better one: Six 4TB hard drives in RAID-Z2

2. If you are testing only and only the filesystem, then you are in the wrong place. Nobody here is a ZFS developer. Also the results of your testing are not really relevant in the real world. A hint: try limiting the writes to 130MB/s and see what the reads are going to be...

*Sigh*, I’m not "only and only" testing the file system. I’m asking a question is this expected? or similar to other people’s results. Granted its more of a question of seek and IO. But the question still stands and is the topic of this thread...

Also, it is "real world”, you have no knowledge of the infrastructure. Your assuming all I do is SMB and use FreeNAS as only a file server on a 1GB link. FreeNAS, in my opinion, is very flexible and doesn’t just serve as a file server, therefore performance at the File System is important to many people. Not to mention spinning disks are normally the bottleneck.

 

[HoneyBadger]

Short answer - Yes, a performance drop under simultaneous R/W is normal and expected. L2ARC and SLOG may help mitigate this for certain workloads.