Slow writes on ixsystems hardware

[Thomas_VDB]

Guessing that da0 is the SLOG?

According to this, I should guess so :

 

[Thomas_VDB]

This is the disk performance of da0 during the migrating of the VM. You can see the part where the sync was enabled.
I am guessing that the top performance of 15MB/s might be the issue.
Question is, is it cause or effect?

 

[Mirfster]

According to this, I should guess so :

Yep, that's your SLOG alright. ;)

 

[jpaetzel]

ESXi artificially sets every NFS write as a sync write, which has to go through the SLOG device. The best solution would be to start using iSCSI (The hardware offload with VAAI is a huge win there) A more appropriate SLOG device would help your current write performance if you continue using NFS + ESXi.

 

[Thomas_VDB]

ESXi artificially sets every NFS write as a sync write, which has to go through the SLOG device. The best solution would be to start using iSCSI (The hardware offload with VAAI is a huge win there) A more appropriate SLOG device would help your current write performance if you continue using NFS + ESXi.

Hi, I understand the NFS sync issue, and as a last resort we will switch to iSCSI.
However I am having a hard time believing that by replacing the slog SSD with another brand of SSD, the write performance will at least quadruple. Mlovelace mentioned that he was getting 240MB/s with an Intel DC S3700 (same price category as our Samsung 850 Pro).

 

[depasseg]

This is the complete config :

Did this config come from ixsystems? Are they the ones who configured the 850's as both a SLOG and L2ARC?
SLOG performance is all about latency. The lower the better.

 

[Sakuru]

I bought a Samsung 950 PRO and it performed terribly as a SLOG. I don't know why. I guess there's just something about Samsung SSDs that doesn't play nice with a SLOG workload. Oh well, as I mention in my signature I should have gotten an Intel 750 instead.

I am rather concerned that someone at iX Systems sold you a box with an improper SLOG. I'm also concerned they called it a ZIL instead of using the proper term. @jkh Do you know what happened here?

 

[zambanini]

i had a similar party with an ixsystem offer. they offered a system which would have (regarding the existing data) only 30% free space. vsphere and iscsi would not like that

 

[Mlovelace]

ESXi artificially sets every NFS write as a sync write, which has to go through the SLOG device. The best solution would be to start using iSCSI (The hardware offload with VAAI is a huge win there) A more appropriate SLOG device would help your current write performance if you continue using NFS + ESXi.

This all sounds oddly familiar... ;)

 

[Thomas_VDB]

Did this config come from ixsystems? Are they the ones who configured the 850's as both a SLOG and L2ARC?
SLOG performance is all about latency. The lower the better.

Yes this is what they suggested after we mentioned our budget.

 

[Thomas_VDB]

Okay so everyone agrees the Samsung 850 Pro is the root cause here? This means that we can either replace the samsung with an Intel SSD, or switching to iSCSI.
Can the ZIL-SSD be replaced with another disk without destroying the pool? If we should go with iSCSI, do we leave the samsung ZIL seated?

 

[Mirfster]

Can the ZIL-SSD be replaced with another disk without destroying the pool?

Yes, I have "Removed" and added SLOGs without issues from a Pool/Volume

 

[jkh]

I am rather concerned that someone at iX Systems sold you a box with an improper SLOG. I'm also concerned they called it a ZIL instead of using the proper term. @jkh Do you know what happened here?

I don't know anything about the box in question (at the very least, someone would need to give me a serial number since I otherwise have zilch to go on in this conversation).

 

[Thomas_VDB]

I don't know anything about the box in question (at the very least, someone would need to give me a serial number since I otherwise have zilch to go on in this conversation).

Hi jkh, thank you for joining this topic.
The system has this serial : A1-50397. Quote KL160526.
Thomas.

 

[Mirfster]

While I don't use NFS (went with iSCSI for ESXi and MS VMs), I wonder if anyone has used the suggestions at "Recommendations for NFS Protocol"? Not that I know if these are "better" or not, just curious.

 

[Thomas_VDB]

Hi,

I was contacted by iXsystems and they were willing to swap the ZIL for Intel type. However they doubt that it will have much effect as the Samsung has better specs in terms of speed than the Intel SSD.
They believe that tuning the parameters is the way to go. I must say that I understand their reasonings.

In the mean time, I have switched from NFS to iSCSI and performance is now at expected level. And I also get VAAI, multiple connection support,...
So me happy.
Because of the asynchronous character of iSCSI, the concerning ZIL is now not used anymore. So no more performance issues there, but is this not as dangerous as using NFS with sync disabled?

Apparently with FreeNAS and ESXi, iSCSI is the way to go.

 

[Mlovelace]

but is this not as dangerous as using NFS with sync disabled?

You can alway force sync writes for iSCSI if/when you get a proper slog device.

 

[Mirfster]

IMHO, I would not feel comfortable at all with "sync=disabled" where VMs are concerned.

*** Side note, I was going to do some tests with NFS and various scenarios (No SLOG, S3710 SLOG, Samsung Evo SLOG, S3500 SLOG), but maybe not needed anymore..

 

[Sakuru]

I was contacted by iXsystems and they were willing to swap the ZIL for Intel type.

Awesome! :D

 

[Thomas_VDB]

You can alway force sync writes for iSCSI if/when you get a proper slog device.

Okay, but i doubt that iscsi is typically used like that, no? In sync=standard mode, the zil is not used with iscsi. So is iscsi in this mode then not equally unsafer as nfs with sync disabled?