Slow writes on ixsystems hardware

[Thomas_VDB]

IMHO, I would not feel comfortable at all with "sync=disabled" where VMs are concerned.

*** Side note, I was going to do some tests with NFS and various scenarios (No SLOG, S3710 SLOG, Samsung Evo SLOG, S3500 SLOG), but maybe not needed anymore..

It would make things clearer because the forum experts believe the Samsung is the culprit, and ixsystems believes its due to tuning. I don't know on what i d put my money...

 

[Mirfster]

Here are some actual tests I did on a quickly created NFS Share (Shoutout to @Mlovelace for helping me get this working with Server 2o12 R2).

So, apologies in advance for the images but a picture is worth a thousand words..

All tests were done against a new Pool composed of 2 x 3TB Mirror vDevs. No additional NFS Tuning was done, just created, shared it and attached to it from Server 2012 R2 DataCenter machine.

Just for a comparison, this is the iSCSI Pool (has 5 x 2TB Mirror vDevs) and an Intel S3710 200GB SLOG. I also have "sync=always":

 

[Thomas_VDB]

Here are some actual tests I did on a quickly created NFS Share (Shoutout to @Mlovelace for helping me get this working with Server 2o12 R2).

So, apologies in advance for the images but a picture is worth a thousand words..

All tests were done against a new Pool composed of 2 x 3TB Mirror vDevs. No additional NFS Tuning was done, just created, shared it and attached to it from Server 2012 R2 DataCenter machine.

View attachment 13638

View attachment 13639

View attachment 13640

View attachment 13641

Just for a comparison, this is the iSCSI Pool (has 5 x 2TB Mirror vDevs) and an Intel S3710 200GB SLOG. I also have "sync=always":
View attachment 13642

Very clarifying test!
So always choose an Intel zil above Samsung when using nfs, but iscsi runs circles around nfs...

 

[Mirfster]

So always choose an Intel zil above Samsung when using nfs, but iscsi runs circles around nfs...

I'm all about Intel. There are plenty of threads on NFS vs iSCSI to make ones head spin. Keep in mind that there was no tuning on the NFS. I am not that adept in it and wanted to mainly provide "side by side" tests to show the SSD SLOG perspectives.

IMHO, iSCSI is still better though, but NFS does have its "Use-Case"; same with CIFS... Forget AFP as far as I am concerned... :)

The iSCSI result has more IOPS going for it with 5 Mirrors vs the 2 Mirrors used in the NFS testing. Certain restrictions pertain to iSCSI, like not going above 50% full, etc.

A blazing NVMe for a SLOG will definitely make those numbers get higher, but perhaps not so much on my old hardware... ;)

 

[Thomas_VDB]

As i understand it, iscsi does not use syncing and therefore is the fastest. I believe you would get roughly the same speed results with nfs and sync disabled. With synced nfs the performance is slower and you can partly solve this with a zil.

Still my question remains: why is iscsi considered as safe if it doesnt use the zil when configurered 'out of the box' (sync=standard, not always).

 

[depasseg]

Here are some actual tests I did on a quickly created NFS Share (Shoutout to @Mlovelace for helping me get this working with Server 2o12 R2).

So, apologies in advance for the images but a picture is worth a thousand words..

All tests were done against a new Pool composed of 2 x 3TB Mirror vDevs. No additional NFS Tuning was done, just created, shared it and attached to it from Server 2012 R2 DataCenter machine.

View attachment 13638

View attachment 13639

View attachment 13640

View attachment 13641

Just for a comparison, this is the iSCSI Pool (has 5 x 2TB Mirror vDevs) and an Intel S3710 200GB SLOG. I also have "sync=always":
View attachment 13642

Awesome testing. Any chance you could add an NFS test with Sync Disabled?

 

[Mlovelace]

As i understand it, iscsi does not use syncing and therefore is the fastest. I believe you would get roughly the same speed results with nfs and sync disabled. With synced nfs the performance is slower and you can partly solve this with a zil.

Still my question remains: why is iscsi considered as safe if it doesnt use the zil when configurered 'out of the box' (sync=standard, not always).

It depends on what you consider "safe" and I think most people would recommend forcing sync for iSCSI back virtualization storage. Again, I would recommend that you read @jgreco post https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

To quote from it:

Why sync writes matter, especially on a fileserver

In the old days, when the VAX would crash and take your changes with it (because you hadn't yet saved to disk), you knew what work you had lost.

To a large extent, people I've talked to feel that this is still the case for a fileserver. Either the file is there or it is lost. In many cases, this is even true.

However, these days, we are using storage for much more complicated purposes. In the best cases, sync writes may already happen by default. In other cases, sync writes must be explicitly enabled.

Consider a VMware virtual disk device. Here is a massive disk file with lots of contents. It will probably exist for the lifetime of the associated VM. So one day in the VM's life, it is writing some minor update to the "hard disk". ESXi transmits that to the NAS, asking it to commit that to the "hard disk" (vmdk file). The NAS responds that it has been written, but then crashes before it actually commits the write to the actual disk. NAS reboots. Now the running VM has data that it thinks has been written to the "hard disk" but that is no longer actually represented in the data served by the NAS. So that might merely be a corrupted file on the VM disk, but that lost write could also have been critical metadata crucial to the integrity of the filesystem. The failure to write this with sync mode translates to some sort of danger.

Unfortunately, the hypervisor is not in a good position to judge the relative importance of a given write request, so the hypervisor simply marks all of the writes to be written sync. This is conservative and safe to do ... and it is dangerous to second-guess this, at least if you care about the integrity of your VM.

So the "good" news is that ESXi issues all NFS requests in sync mode, but the bad news is that ZFS performance handling large numbers of sync requests is poor without a SLOG device. This leaves a lot of ESXi users wanting to "fix" the "awful" performance of their FreeNAS ZFS. The choices basically come down to disabling the ZIL or disabling sync somehow, or providing a proper SLOG. The former options are dangerous to the integrity of the VM, and the latter is typically expense (for the SLOG) and learning (what you're reading right now, etc).

However, the same problem exists with iSCSI - and by default, iSCSI writes are async. The same risks to VM integrity exist. So for an iSCSI setup with VM's, setting "sync=always" is the correct way to go.

 

[Mirfster]

Any chance you could add an NFS test with Sync Disabled?

Sure, I'll run them and post those as well.

 

[Mirfster]

As requested, same Pool... NFS Results with "sync=disabled"

 

[Stux]

In iscsi the default is no sync. With a VM, you want sync.

So if you value the vm, then you want sync enabled.

So, you're back to the same problem.

 

[Thomas_VDB]

As requested, same Pool... NFS Results with "sync=disabled"
View attachment 13648

View attachment 13649

View attachment 13650

View attachment 13651

To have the complete story, could you test with iscsi and forced sync?
Thx.

 

[Thomas_VDB]

It depends on what you consider "safe" and I think most people would recommend forcing sync for iSCSI back virtualization storage. Again, I would recommend that you read @jgreco post https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

To quote from it:

Very interesting article.

I have now forced sync=always on the iSCSI setup, and write performance went down to 60MB/s. Better than NFS, but not so happy with it.

 

[Stux]

The SLOG needs to have Power Loss Protection, or its not really a SLOG. The Samsung does not. You should have it replaced with an Intel SSD. The Intel SSDs generally have steady random write performance, high write endurance, and PLP.

I would consider using an NVMe PCIe Intel SSD which are capable of insane random and sequential write performance. Do you have an available 4x PCIe slot?

Edit: http://lkcl.net/reports/ssd_analysis.html, the conclusion is fairly damning "Conclusion: don't buy anything other than Intel SSDs" and as far as I can tell, this is still the situation today.

 

[Stux]

The SLOG needs to have Power Loss Protection, or its not really a SLOG. The Samsung does not. You should have it replaced with an Intel SSD. The Intel SSDs generally have steady random write performance, high write endurance, and PLP.

I would consider using an NVMe PCIe Intel SSD which are capable of insane random and sequential write performance. Do you have an available 4x PCIe slot?

Edit: http://lkcl.net/reports/ssd_analysis.html, the conclusion is fairly damning "Conclusion: don't buy anything other than Intel SSDs" and as far as I can tell, this is still the situation today.

Intel Datacentre grade PCIe SSDs

http://www.intel.com.au/content/www/au/en/solid-state-drives/intel-ssd-dc-family-for-pcie.html

And the consumer version:

http://www.intel.com.au/content/www/au/en/solid-state-drives/solid-state-drives-750-series.html

For when endurance is not so critical.

 

[Mirfster]

To have the complete story, could you test with iscsi and forced sync?

Here are both for iSCSI. First one is with "sync=always"

 

[Thomas_VDB]

Here are both for iSCSI. First one is with "sync=always"
View attachment 13657

View attachment 13658

Thanks for your extensive testing.
I am happy that you are seeing the same results as I am : iSCSI forced sync roughly cuts the write performance in half.
However you are getting overall higher speed values. I think I will exchange the Samsung 850 Pro ZIL SSD for an intel SSD.
Everybody is sure that I can swap the ZIL without destroying the pool? What would be the step-by-step instructions for that?

 

[Mirfster]

Everybody is sure that I can swap the ZIL without destroying the pool? What would be the step-by-step instructions for that?

Been doing it the entire time through the testing. ;)

However, I have not been using [Replace], instead I completely [Remove] the Drive (in the GUI) and then add the new one using [Volume Manager] - [Manual Setup]. This way is a little more dangerous, since you for sure do not want to incorrectly have it added as a "Member Disk", but as a "ZFS Extra" instead.

The reason for this is that my disks are not clear and I would end up with a message similar to this if I used the [Replace] (Of course I could just whack the new drive first with gpart destroy /dev/%disk%, but too lazy)

Message I am avoiding by NOT using [Replace]:

So here is the way I am doing it:

With new SLOG now attached:

 

[Thomas_VDB]

Been doing it the entire time through the testing. ;)

However, I have not been using [Replace], instead I completely [Remove] the Drive (in the GUI) and then add the new one using [Volume Manager] - [Manual Setup]. This way is a little more dangerous, since you for sure do not want to incorrectly have it added as a "Member Disk", but as a "ZFS Extra" instead.

The reason for this is that my disks are not clear and I would end up with a message similar to this if I used the [Replace] (Of course I could just whack the new drive first with gpart destroy /dev/%disk%, but too lazy)

Message I am avoiding by NOT using [Replace]:
View attachment 13665


So here is the way I am doing it:
View attachment 13662

View attachment 13672

View attachment 13668

View attachment 13669

View attachment 13670

With new SLOG now attached:
View attachment 13673

Great tutorial. I will follow it to the letter
Thanks!

 

[Mirfster]

FYI, I'm going to be tearing down the Pool later today so if there is any last minute tests desired let me know and I will try to fit them in.

 

[Thomas_VDB]

Hi all,
@Mirfster, first of all, thanks for your extensive testing.

I was thinking about the iSCSI performance hit with sync=always :
We have an UPS (APC SmartUPS 1000), to which our FreeNAS + 2 hosts are connected (no USB/serial connection - UPS is just a power buffer).
What if I connect the UPS only to the FreeNAS, not the hosts, and disable iSCSI sync (sync=standard).
In case of a power failure, the 2 hosts would immediately be killed, but the FreeNAS would live on for a few minutes. So this would make sure
that the last write commands are committed to disk. I presume that a few minutes power buffer would be more than enough?

Could you guys confirm (or comment) on this idea?