iSCSI performance with sync write always

[eroji]

Below is the current setup of my FreeNAS. I have all the drives in a mirror+stripe setup and 10GbE connection directly to the ESXi host (dual port, in roud-robin). I have 2 iSCSI volumes of 4TB and 2TB made available to ESXi. With sync write always, I am getting about 400MB/s (via CrystalDiskMark) sequential write, and when sync write is disabled, I am getting around 1600MB/s. I understand sync write will significantly drop write performance, but do the numbers look normal?

2x E5 2660
128GB RAM
Intel X520-DA
Intel 750 400GB for ZIL
4x4TB HGST 7200RPM
2x4TB WD Red
6x5TB Toshiba 7200RPM

 

[sfcredfox]

I don't claim to be any kind of expert, but iI've been doing what you're doing for a while. If I understand your configuration, my opinion is probably yes, that looks right-ish.

How do you have those drives arranged into pools? Are you doing mirrors pairs for those two iSCSI stores? Not sure what controller you're using either.

Crossing into iSCSI storage for VMs (typically a heavy random workload), things get complicated.

I'm not sure if your system in it's current configuration is built well for heavy SYNC IO? I don't see you listing anything looking like a SLOG. Remember that SYNC writes have to be committed to non-volatile storage somewhere, so if you aren't providing a SLOG, then it's going to the default ZIL location in the pool. That usually slows things down. Many posts about this topic on forum. See insights to SLOG post by jgreco. I'm guessing this is the area you need to study and evaluate.

Something else to consider, people around here beat up bench marks like these. You're testing sequential reads/writes, which can sometimes show the max potential performance of the hardware and software in the configuration you have, but it's probably not anywhere close to real performance expectation for VMs. Maybe you already realize that. If you are trying to gauge how the storage will perform for your VMs, you probably want to switch to random IO testing.

I'm running a hand full of VMs that cover every function you could expect (database, media, etc.), and they don't usually tax the system in the same way you are. If I run a sequential, my pool of 20 old terrible disks in groups of Z1 can still do ~500MBps. Probably just a function of having so many. If I did them in mirrors like you're supposed to, it would probably be better performance, I just wouldn't have enough space.

If you want to test real performance, throw a couple VMs on there and generate real workloads, then use your gstat/zilstat/zpool iostat tools to gauge whats going on. That might give you better data.

 

[eroji]

Thanks for the reply. I incorrectly listed the Intel 750 SSD as ZIL. It is however set as the SLOG device. This should explain why sequential write is coming back at 1600MB/s when sync is disabled. I understand the benchmarks I used would not be a good real world measurement, but I am purely using them as something to gauge the performance hit of turning sync write on. The drives are set to mirror in pairs and the pairs are striped for the pool. I am not sure if this is ideal but I recall reading somewhere that this would be a better performing setup (without sacrificing even more storage while still retaining expandability) than a raid-z1 or z2.

 

[Ericloewe]

I incorrectly listed the Intel 750 SSD as ZIL. It is however set as the SLOG device.

SLOG devices contain the ZIL. ;) It's improper terminology to call it a ZIL (since that can also live on the pool), but it's mostly understandable.

I am not sure if this is ideal but I recall reading somewhere that this would be a better performing setup

Right. If single-drive redundancy isn't good enough for you, you can still do three-way mirrors. More vdevs give you more IOPS, so RAIDZ is at a disadvantage in that area.

 

[cyberjock]

I really, really wish people would stop using Crystaldisk. It's not a good tool to use for ZFS, at all.

Anyway, the write number could be reasonable for sync = always.

Thanks for the reply. I incorrectly listed the Intel 750 SSD as ZIL. It is however set as the SLOG device. This should explain why sequential write is coming back at 1600MB/s when sync is disabled.

slog and zil are somewhat synonymous. They aren't exactly, but if you say you have an ssd for the zil, then we know what you mean.


You clearly are even more confused than you realize. The only time an slog device would be used is when sync is set to standard or enabled. When it is disabled, zero writes are made to the slog. None.. at all. Any sync writes are immediately returned even though the write is not being treated as synchronous. In essence, sync=disabled means "go as fast as RAM will let you go". That is why you get such amazing performance with sync=disabled. It is also why it is extremely dangerous because the writes are supposed to be in non-volatile storage when they are set as sync writes.

I understand the benchmarks I used would not be a good real world measurement, but I am purely using them as something to gauge the performance hit of turning sync write on.

Hate to break it to you, but the Crystal benchmarks aren't even good enough for that when using ZFS. We keep telling people not to play the benchmark games, and people keep on ignoring us.

 

[eroji]

Thanks for the clarifications. Taught me a couple things that I seemed to have missed before I ventured into ZFS.

 

[Rick Arman]

I really, really wish people would stop using Crystaldisk. It's not a good tool to use for ZFS, at all.

Anyway, the write number could be reasonable for sync = always.



slog and zil are somewhat synonymous. They aren't exactly, but if you say you have an ssd for the zil, then we know what you mean.


You clearly are even more confused than you realize. The only time an slog device would be used is when sync is set to standard or enabled. When it is disabled, zero writes are made to the slog. None.. at all. Any sync writes are immediately returned even though the write is not being treated as synchronous. In essence, sync=disabled means "go as fast as RAM will let you go". That is why you get such amazing performance with sync=disabled. It is also why it is extremely dangerous because the writes are supposed to be in non-volatile storage when they are set as sync writes.




Hate to break it to you, but the Crystal benchmarks aren't even good enough for that when using ZFS. We keep telling people not to play the benchmark games, and people keep on ignoring us.

so if sync is left at disabled, you want to make sure you have a UPS to prevent loss of power which could potentially corrupt the VM if the data is not written to disk? is that the danger?

 

[Ericloewe]

Geez, was that question really worth the 2.5-year necro?

If the application relies on sync write functionality being honored and it isn't, you're vulnerable to all sorts of things - power failure, hardware failure, software bugs that panic the system, Linus from LinusTechTips walking in the room, bad batteries in your UPS going out before you can successfully shutdown, ...

It may be perfectly acceptable for a lab environment, for instance, but it's up to you to examine the risks and determine how much pain you're willing to tolerate.

 

[Rick Arman]

sorry for the late entry here but I'm prepping for my build (i have all the parts). i just want to make sure I understand everything, even at the risk of getting yelled at by the mods lol

 

[Ericloewe]

sorry for the late entry here but I'm prepping for my build (i have all the parts). i just want to make sure I understand everything, even at the risk of getting yelled at by the mods lol

It's just that it's more suited to a new thread. Unless there's a relatively strong link to an old thread, just make a new one. If you feel it might be relevant, just add a link to the older thread.