SLOG not changing disk performance of volume during testing

[Eds89]

Hi,

I have been setting up a FreeNAS/ESXi all in one similar to what Stux has done here: https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/
I want to use a Samsung SM953 as my SLOG for a volume, to provide good performance for hosting VM disks. The SM953 has good read/write performance, with writes reportedly being around 850MB/s as per this whitepaper: https://s3.ap-northeast-2.amazonaws.com/global.semi.static/SM953_Whitepaper-0.pdf

When I create a volume of 2 mirrored vdevs (basically RAID 10), and attach a zvol as a datastore in ESXi via iSCSI, I am able to successfully create a test VM and run some basic disk benchmarks using crystal disk mark.
While this is obviously not the best testing tool, it does seem the quickest and easiest for me to use.

My issue is, that whether I have the Samsung SSD added to the volume as a SLOG or not, the disk performance remains the same. Reads for sequential operations are approx 900MB/s, with writes being about 400MB/s. During testing with the SSD added, FreeNAS looks to report no activity on the drive, so it doesnt look like it is being using for ZIL. These figures feel like what I would expect from a 4 disk RAID 10 of 7200 rpm drives (about 200MB/s read per drive, and about 100MB/s write per drive).

I have uploaded screens of the volume with and without SLOG, and the performance with and without SLOG

Where do I need to look to diagnose this, as while performance is good, I feel it should be better when using the SSD as a SLOG.

Thanks
Eds

 

[Zredwire]

If you are using iSCSI then ESXi does not automatically use Sync Writes. For data protection I would personally set Sync = always on the iSCSI volume in FreeNAS. But doing this will almost assuredly slow your system down. It will use the SLOG but using a SLOG with Sync writes is not as fast as using no SLOG with Sync off (which seems to be what your system is currently set at).

 

[Zredwire]

Here is a resource: https://forums.freenas.org/index.php?threads/setting-sync-always-adding-slog.55526/
Also to check the Sync settings use: zfs get sync, with the same parameters as in the resource I added above.

 

[Eds89]

If you are using ISCSI then ESXI does not automatically use Sync Writes. For data protection I would personally set Sync = always on the ISCSI volume in Freenas. But doing this will almost assuredly slow your system down. It will use the SLOG but using a SLOG with Sync writes is not as fast as using no SLOG with Sync off (which seems to be what your system is currently set at).

Ah yes forgot about that.
Have enabled sync writes on the zvol, and I can see some write activity on the SLOG. Crystal Disk mark doesn't seem to give different results, with writes still being around the 400MB/s mark.

Will writes only ever be as fast as the spinners can handle, even though the SSD can handle sequential writes up to 800+MB/s?

 

[Zredwire]

That is a good question. Maybe someone more versed with ZFS can answer. For burst I would think you could get higher. I believe the ZIL on the SLOG will hold a maximum of two transaction groups, with each transaction group being about 5 seconds. So a burst of less than 10 seconds, or at least 5 seconds would be really fast you would think. For longer copies, it must pause to clear the transactions out of the ZIL on the SLOG.

 

[Eds89]

I might lower the test file size and see if I can produce different results for SLOG vs no SLOG

 

[Zredwire]

I would be interested in your results if your able to set up an effective test. I would assume random writes would be a big improvement. If not then what good is a SLOG? Well I guess latency would be lower and IOPS higher on a decent SLOG.

 

[Eds89]

Ok, again not sure if my testing methodology is the most appropriate, but when I lower the test file size to 100MB (down from 1GB) the write speeds see no change with or without the SLOG.

I imagined the SLOG would behave as a cache would on a hardware RAID controller, so the data would go the cache (SLOG), then flush to disks, assuming async writes are enabled.

Maybe this is the expected behaviour?

 

[Zredwire]

Wow, on my system, with Sync enabled I see a huge difference between having a SLOG and no SLOG. Maybe something is up with your test. What write speeds are you getting now? When I look at your first post you talk about getting over 400MB/s write speeds. With 4 drives in mirror setup if your disk can do 100MB/s each then the max write you should get is 200MB/s. With writes you take a 50% penalty in mirrors, where reads take no penalty. With SYNC off or standard you should be able to get more as the ZIL used is in memory. With SYNC on and no SLOG the very best should be 200MB/s (though mine is much, much lower than that without a SLOG).

 

[Eds89]

I have tried these scenarios and measured sequential writes on a 1GiB file size:
No SLOG, 2 mirrored vdevs only, sync=always - write speeds apporx 40MB/s
No SLOG, 2 mirrored vdevs only, sync=standard - write speeds approx 430MB/s
No SLOG, 2 mirrored vdevs only, sync=disabled - write speeds approx 450MB/s
SLOG, 2 mirrored vdevs, sync=always - write speeds approx 370MB/s
SLOG, 2 mirrored vdevs, sync=standard - write speeds approx 450MB/s
SLOG, 2 mirrored vdevs, sync=disabled - write speeds approx 440MB/s

So I can see that if I want sync writes, I definitely need to use a SLOG. Think the issue with my first test was not setting the correct sync option for each test.
Disabled or standard sync seems to make little to no difference for sequential performance.

I am only assuming that each disk can do 100MB/s as that's what I see in the disk performance counters in FreeNAS. Given that's the case, it seems odd that I then get 400MB/s+ writes when it should be half that?

Would you imagine with sync disabled, I should get over 400MB/s as it is using memory? Is it perhaps the limitation of 400MB/s is somewhere else in the system? Perhaps memory, or if it is writing something to another drive?

 

[Zredwire]

Ok those numbers look correct. You are getting almost a 10 times increase in performance when using a SLOG with Sync = always. That looks correct. Once again I am sure there is someone more versed than me with ZFS but I believe the 400MB/s write you get is being capped by a combination of the type and number of spinning drives you have and the way ZFS uses its ZIL (both in memory or on SLOG). I know ZFS will not endlessly cache transactions. I believe the max it will cache is 3. One being written and two additional. But whatever the number it is capped. So for writes it will only cache so much (as opposed to reads where it will cache much more depending on memory) then it must flush to disk. So the speed of the disk and the number of disk will determine how fast they can write. Maybe the combination of the two is averaging out to about 400MB/s on your system?

 

[Eds89]

Just seems odd, as if I drop the file size down to something tiny, I would imagine as it doesn't take long to write, it should all be handled in cache/SLOG, report as being written very quickly, then flush to disk in the background.
One of the reasons I shelled out for an SSD to use as a SLOG, as I expected it to aid in write caching rather than relying on the speed of the spinning disks.

Maybe someone else out there can say for certain how it should behave and if my limit is just that of the spinning drives.

Thanks for your help thus far!

 

[Zredwire]

Yes maybe someone else can speak to how the ZIL works in ZFS and if it is the cap you are seeing. I will say that your SLOG is working pretty good though as it will never be as fast as RAM (not with current technology). So with RAM (which you get with sync=disabled) you get 450MB/s and with your SLOG you get 370MB/s.

Note: Both are faster than what you should be getting with a 4 Drive mirror (4 drive spinning disk) if there was no cache.

 

[Eds89]

So as a test, I removed the SSD as a SLOG, and create a new vdev/volume/target and created a new ESXi datastore.
I put a new VMDK on that datastore, attached it to my test VM and ran the same tests.

I got exactly the same speeds, for both reads and writes!

I'm now going to remove the drive, put it into a physical Windows machine and test what it is actually capable of. I can't think of where the bottle neck is. The storage network between FreeNAS and ESXi is 10G so an't be the problem (The fact reads are close to 1GB/s shows the bandwidth is there).

Hopefully someone else can comment, as I'd like to see if I can get more performance before I use this live.

EDIT: Attached the SSD direct to another VM, and it is capable of 1GB/s writes. Definitely a FreeNAS bottleneck somewhere

 

[Stux]

Testing the benefits of SLOG using a RAM disk!
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

Sync=disabled is actually faster than using a ram based ‘slog’

But the technique documented in my link is a valid test for working out your max gain

When using ESXi your critical vm datastores on iSCSI volumes should have sync=always set. To force sync

NFS datastores will automatically force sync because ESXi always writes sync writes to
NFS.

iSCSI tends to be faster. And NFS is easier/more pleasant to work with.

Slog is not a write cache. A slog invokes writing extra data to the slog. The original ZFS zil is written to the slog as a sync write and to the array as an async write.

Async writes can be reordered to make them much faster, but the sync write can only go as fast as you can sync write to your slog.

Also, async writes can be to disk cache.

 

[Eds89]

Ok, so the long and short of it;

• If I disable sync on a zvol, I get 900MB/s reads, 450MB/s writes. This is the maximum performance of the disks in the vdev?
• If I enable sync, and have a SLOG, am I still limited to the maximum performance of the vdev? or;
• Would an SSD with 1GB/s writes allow the sync write to complete faster and allow the async writes to continue in the background to the vdevs?

I'm just trying to determine at this point if I am actually encountering a bottleneck, or if I am just seeing the maximum performance I can get for this vdev.

Thanks

EDIT: Ok I really don't think what I am seeing is because of a limit of the underlying vdev, as if I create a standalone vdev on just the SSD, and then create a datastore in ESXi over iSCSI, I would expect to hit about 1.2GB/s reads and 1GB/s writes. This would be the limit of a 10G network connection between the two.
The SSD is capable of more, but gives me the same 900MB/s reads and 450MB/s writes. There must be something else I am missing surely?

 

[Stux]

Play with a RAM disk. That will show you your true maximums

 

[c32767a]

I'm just trying to determine at this point if I am actually encountering a bottleneck, or if I am just seeing the maximum performance I can get for this vdev.

Question from left field.

When you do your tests with the vdev and SSD SLOG, go into the freenas UI under Services > NFS, what number is in the "Number of servers" field? in the NFS settings?

 

[Eds89]

Play with a RAM disk. That will show you your true maximums

So you mean create a RAM disk, and add it as a vdev and see what performance I an achieve to that? Or add it as a SLOG to my existing pool? What will the resulting write speed show as my limitation? The physical disks?

One other thing to note, is when I was testing and getting 450MB/s, I was using 4 disks, so 2 mirrors.
I deleted the volume and recreated and am now using 6 disks, so 3 mirrored vdevs, and still get exactly the same read/write speeds. That to me shows the limitation is not the disks, but elsewhere?

Question from left field.

When you do your tests with the vdev and SSD SLOG, go into the freenas UI under Services > NFS, what number is in the "Number of servers" field? in the NFS settings?

I am not using NFS, I am using iSCSI. As such the service isn't running, but checked anyway and has 4 as the number.

 

[Spearfoot]

Adding a SLOG won't speed up your writes... Here are some interesting articles about the SLOG, key quote from the first one:

It is important to, first and foremost, clear up a common misconception I see about the ZIL. It is not a write cache. There is no caching of any kind going on in the ZIL. The ZIL's purpose is not to provide you with a write cache. The ZIL's purpose is to protect you from data loss. It is necessary because the actual ZFS write cache, which is not the ZIL, is handled by system RAM, and RAM is volatile.

http://nex7.blogspot.com/2013/04/zfs-intent-log.html
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/