Sync writes to pool with SLOG about 40% slower than writes directly to SLOG SSD

Status
Not open for further replies.

jesseosby

Cadet
Joined
Oct 18, 2015
Messages
2
I'm playing around with FreeNAS 9.3 and have found sync writes aren't as fast as I expected from my setup. First I used dd to test the max sequential write speed of my SLOG SSD:

[root@freenas] ~# dd if=/dev/zero of=/dev/ada0 bs=128k count=2000
2000+0 records in
2000+0 records out
262144000 bytes transferred in 2.431943 secs (107791992 bytes/sec)​

This is slower than the drive's write spec (170 Mbyte/s), but close enough to the limit of my gigabit Ethernet that it'd be fine if my pool could write that fast. But when I add the SSD to my pool as an SLOG and do the same write to the pool, I get this:

[root@freenas] ~# zpool add store log ada0
[root@freenas] ~# zfs set sync=always store
[root@freenas] ~# dd if=/dev/zero of=/mnt/store/tempfile bs=128k count=2000
2000+0 records in
2000+0 records out
262144000 bytes transferred in 4.080628 secs (64241090 bytes/sec)​

I.e. about 40% slower. My RAM is 32GB, so my transaction group size should be big enough to accept that whole 256MB write into the SLOG, right? What could be causing this? Are SLOG writes non-sequential or multi-threaded or something that would make them run slower than a single-thread sequential write to the same device?

This is somewhat academic, as when I was searching for an answer to this, I discovered that my SLOG SSD doesn't have a super-capacitor, so (as I understand it) sync writes won't give me any extra protection in this setup anyway - but I am still curious to understand.

Hardware:
Dell PE2970
2 x AMD Opteron 2389 (2.90Ghz quad-core)
32GB ECC 1067MHz
6 x SATA 7200 RPM 320GB - RAIDZ2
1 x Intel X25-E 32GB SLC - SLOG
 

BERKUT

Explorer
Joined
Sep 22, 2015
Messages
70
With SLOG pci-e HyperX Predator PCIe SSD (SHPM2280P2H/240G) I have max 95 MBytes/s speed write :)
 

kspare

Guru
Joined
Feb 19, 2015
Messages
507
It boils down to having a good slog device.

This is a Intel 750 PCIe 400GB drive. Works great for us!
 

Attachments

  • slog.JPG
    slog.JPG
    32.6 KB · Views: 429

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
In your first test you're writing to your SSD directly in asynchronous mode, so it's being told "Here's a pile of data, write it."

In the second test where the SSD is in place as a log device, it's being asked to do synchronous writes at single queue depth. This enacts a significant overhead penalty since the drive is now being asked "Write this one piece. Make sure it's safe. Okay. Write the next piece. Make sure it's safe. Okay."
 

jesseosby

Cadet
Joined
Oct 18, 2015
Messages
2
In your first test you're writing to your SSD directly in asynchronous mode

Ah, cool, I was just assuming dd would be doing a synchronous write. Based on your reply, I found https://romanrm.net/dd-benchmark which details how to do synchronous write with dd. It looks like the dd binary included with FreeNAS doesn't support the necessary flag, so I tested from Linux:

[root@livecd ~]# dd if=/dev/zero of=/dev/sda bs=128k count=2000
2000+0 records in
2000+0 records out
262144000 bytes (262 MB) copied, 2.02784 s, 129 MB/s

[root@livecd ~]# dd if=/dev/zero of=/dev/sda bs=128k count=2000 oflag=dsync
2000+0 records in
2000+0 records out
262144000 bytes (262 MB) copied, 3.26624 s, 80.3 MB/s

For some reason the speeds are faster under Linux than under FreeBSD, but the same ~40% speed drop is observed going from async to sync.
 
Status
Not open for further replies.
Top