Ridiculously slow DD write Speed

Status
Not open for further replies.

adamjs83

Dabbler
Joined
Sep 3, 2015
Messages
40
I am totally new to freenas and after reading this forum and the manual extensively I ordered my parts and assembled my machine which I have been testing the last few weeks. The full specs can be found in my signature below. This NAS is for home and home lab use exclusively.

I have one volume of spinning disks in raidz2 which I am able to write to over CIFS and get around 120MBps, better than I had hoped and the 1Gb link is clearly being saturated.

My other volume is 4 SSDs and is connected to ESXI using iSCSI over 2 multipathed links with and MTU of 9000 the entire way. With sync=always I get about 113MBps read and 40MBps write and 113/113 MBps with sync=standard. Setting all devices to the default MTU of 1500 slightly reduces all of those speeds These results are from a windows server 2012 R2 vm using CrystalDiskMark. The freenas gui reporting feature shows that each of my two Ethernet are splitting the traffic at about 450Mbps each, clearly not saturating the gigabit links.

In order to diagnose why the two iSCSI ethernet connections were not getting higher throughput I wanted to test the performance of the pool directly. My research pointed me to use DD. Below are the results.

Code:
adamjs83@sto01:/mnt/SAN-VOLUME % sudo dd if=/dev/zero of=testfile bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 220.526250 secs (232172 bytes/sec)

sudo dd if=testfile of=/dev/zero bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.089598 secs (571441707 bytes/sec)


To save time that's read of .23 MBps write and .57 GBps write

The read performance is ok for my needs but the write performance is abysmal and significantly slower than even the throughput reported in Crystal.

While the DD write is running this is the typical output of gstat for one of the 4 drives, the others are about the same.

Code:
L(q)  ops/s  r/s  kBps  ms/r  w/s  kBps  ms/w  %busy Name
1    340      0      0    0.0    114    456    0.0   69.0| gptid/10c674da-5e3c-11e5-93d7-0cc47a6ae2c8


I ran the same tests on my spinning pool and the results were .22 GBps write and .11GBps read. Not sure why those aren't reversed but plenty fast for my needs.

Does anyone have any ideas as to why I am getting this abysmal write performance from the SSDs on DD and how troubleshoot?

Additionally am I on the right track trying to figure out why the iSCSI is <500Mbps on each of the two Gb links?
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633

adamjs83

Dabbler
Joined
Sep 3, 2015
Messages
40
Thanks for that link. I have followed those steps and I still think the issue is with the dataset. The dd test shows the very slow write speed to the ssd volume and not the spinning volume. As another test I just created a test dataset on the ssd volume shared it over CIFS. I copied a 1GB file to the spinning disk volume and it averaged about 70MBps while the same file being copied from the same device to the ssd volume averaged about 12MBps. I now think that there is some kind of underlying issue that is causing this pool to write so slowly but I have no idea how to troubleshoot it.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Low performance with dd is probably the result of dd's block size set to 1KB, and the fact that dd runs its requests sequentially, one at a time. In combination with sync=always it creates huge overhead on ZFS layer and tons of _synchronous_ writes on SSD layer. Desktop SSDs without supercaps may be quite slow on such access patterns. Generally while this test could be valid in some situations, I would say that it is far from any typical workload, so make sure that you are testing what you really want.
 

adamjs83

Dabbler
Joined
Sep 3, 2015
Messages
40
Low performance with dd is probably the result of dd's block size set to 1KB, and the fact that dd runs its requests sequentially, one at a time. In combination with sync=always it creates huge overhead on ZFS layer and tons of _synchronous_ writes on SSD layer. Desktop SSDs without supercaps may be quite slow on such access patterns. Generally while this test could be valid in some situations, I would say that it is far from any typical workload, so make sure that you are testing what you really want.

That's good to know and make sense.

I guess what I want to test is where the bottleneck is that is causing me not to saturate the two nics. I was trying to test what the read/write performance is locally on the nas so that i could rule that out as the bottleneck before trying to troubleshoot the connection. I know that I have exceeded my iSCSI per NIC throughput over CIFS to a different volume so testing the speed on that volume seemed like the logical next step to me.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Now your dd test measure almost only a latency, not a throughput: for read you've got 570K sequential reads per second, that is good; but for writes, probably due to sync=always, you've got only 230 sequential writes per second, that is about a speed of good HDD.

If you want to test throughput, increase block size to something like 1MB, or even run several of those dd's same time and add their speeds.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
Is this dd test on your Freenas? or from your esx server?
 

adamjs83

Dabbler
Joined
Sep 3, 2015
Messages
40
Is this dd test on your Freenas? or from your esx server?
DD test is on Freenas. Only the Crystal test was on an ESXI guest.
 

adamjs83

Dabbler
Joined
Sep 3, 2015
Messages
40
So a few updates.

Using iperf from the ESXI host to freenas I am seeing throughput just under 950 Mbps.

CIFS to the Spinning Disk pool is at 100 MBps while CIFS to the SSD pool is barely sustain 40 MBps

DD using a block size of 1MB greatly improves the result to .16 GBps which is still significantly slower than my spinning disks at .23 GBps.

There is no difference in dd results with sync=always or sync=disabled but cifs jumps from 40MBps to 65MBps

Code:
adamjs83@sto01:/mnt/SAN-VOLUME % sudo dd if=/dev/zero of=testfile bs=1000000 count=50000
50000+0 records in
50000+0 records out
50000000000 bytes transferred in 311.731225 secs (160394583 bytes/sec)
adamjs83@sto01:/mnt/SAN-VOLUME % dd if=testfile of=/dev/zero bs=1000000 count=50000
50000+0 records in
50000+0 records out
50000000000 bytes transferred in 38.290554 secs (1305805080 bytes/sec)


The question still remains is why am I getting such bad iSCSI and CIFS performance to this volume and not to the spinning disk volume.
 
Last edited:

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874

adamjs83

Dabbler
Joined
Sep 3, 2015
Messages
40
So after changing all the cables and moving the drives to different ports there was no change.

I destroyed the volume and recreated it and everything is working perfectly. I guess there were some gremlins in there I was never going to find.
 
Status
Not open for further replies.
Top