How to throttle write and txg on new OpenZFS Pool version

[Bjarke Emborg Kragelund]

FreeNAS 9.2.1.2-RELEASE

Used to throttle my pool via :

vfs.zfs.write_limit_shift : 9
vfs.zfs.txg.timeout : 1
vfs.zfs.txg.synctime_ms : 200
With 4 X SLOGs we very seldomly saw > 10 ms write latency to the pool. We use it as a NFS Server for ESXI hosts.

Do not know exactly when it happened, but now the first two settings are not even applicable anymore. Hence we are experiencing > 100 seconds write delay yet again :-(

Can anyone guide me to which settings you can 'play' with on the current ZFS pool version?

( That is, if I got it right? )

Brgds.
Bjarke Emborg Kragelund

 

[Bjarke Emborg Kragelund]

The answer to the above - I think - lies somewhere in here : https://github.com/illumos/illumos-gate/commit/69962b5647e4a8b9b14998733b765925381b727e

Now to the most important part... understanding it!

Brgds.
Bjarke Emborg Kragelund

 

[cyberjock]

Wow. Those settings are.... out there! Where did you get those settings from?

 

[eraser]

What version of FreeNAS did you upgrade from?

Looks like "vfs.zfs.write_limit_shift" and "vfs.zfs.txg.synctime_ms" are also missing on my FreeNAS 9.2.1.2 installation.

However, I did spin up a new FreeBSD 9.2 system and those two sysctl's do exist there. Interesting.... wonder why they are missing from FreeNAS?

 

[dlavigne]

I know the first one get deprecated in 9.2.0. From http://doc.freenas.org/index.php/Tunables:

The ZFS version used in 9.2.0 deprecates the following tunables:
vfs.zfs.write_limit_override vfs.zfs.write_limit_inflated vfs.zfs.write_limit_max vfs.zfs.write_limit_min vfs.zfs.write_limit_shift vfs.zfs.no_write_throttle
If you upgrade from an earlier version of FreeNAS® where these tunables are set, they will automatically be deleted for you. You should not try to add these tunables back.

It sounds like the second one was deprecated by one of the new feature flags introduced somewhere in the 9.2.1.x series.

If there is a performance implication, create a bug report at bugs.freenas.org and post the issue number here. In the report, explain your use case and hardware.

 

[Bjarke Emborg Kragelund]

Hello,

@eraser - I guess it is the pool-version that defines which settings are applicaple - not the OS. After a zpool upgrade on my home-system - the old settings are not applicable either.

@cyberjock, Found the information about getting a responsive pool with minimal latency at various sites.

https://wiki.freebsd.org/ZFSTuningGuide

Also a lot of it form here in this forum.

After a lot of testing and tweaking this is what we have been in production with for about 6 months. A very responsive pool with latencies below 2 ms with 10 - 20 normal production VMs. So we have been quite happy.

But all things come to an end. We now have to come to grips with the new methods og throttling write and especially txg.

This page : http://code.metager.de/source/history/zfs/linux/zfs/man/man5/zfs-module-parameters.5 mentions a description of the 'new' variables. Will be searching for it tommorow.

Brgds.
Bjarke Emborg Kragelund

 

[HoneyBadger]

This page : http://code.metager.de/source/history/zfs/linux/zfs/man/man5/zfs-module-parameters.5 mentions a description of the 'new' variables. Will be searching for it tommorow.

Found it for you now, at least in github format.

https://github.com/zfsonlinux/zfs/blob/master/man/man5/zfs-module-parameters.5

The "ZFS I/O Scheduler" section is at the bottom.

 

[vitek]

Let us know how the tweaking for low latency goes with the new parameters :).

 

[Bjarke Emborg Kragelund]

Thanks to everyone - blitzen schnell support

I will look into it ASAP.

 

[Bjarke Emborg Kragelund]

Hello everybody,

I have managed to get the pool to a state which is very similar in latency / responsiveness as from before we upgraded the pool with the newest feature-flags.

The below settings are taken from ( not quite :) ) understanding this :

https://github.com/illumos/illumos-gate/commit/69962b5647e4a8b9b14998733b765925381b727e
http://code.metager.de/source/history/zfs/linux/zfs/man/man5/zfs-module-parameters.5
https://github.com/zfsonlinux/zfs/blob/master/man/man5/zfs-module-parameters.5

A lot of tweaking and testing has gone by... and finally we are seeing responsive VM's again.

We use the FreeNAS as a NFS-only for VMware ESXi hosts.

The below settings keep at least 10 running VMs - with some activity - below an average of 10ms - both reads and writes. As we only have 5 vdevs in this LAB-pool, we are quite content.

System | Tunables

vfs.zfs.cache_flush_disable : 1 ( default = 0, should only be disabled if you have battery-backed the NVRAM on your HBA - if I have understood it right? )
vfs.zfs.txg.timeout : 1 ( default = 10? )
vfs.zfs.vdev.async_read_max_active : 1 ( default = 3 )
vfs.zfs.vdev.async_write_max_active : 1 ( default = 10 )
vfs.zfs.vdev.max_active : 32 ( default = 1000, should not be in effect as the sum of all max_active's is below this number )
vfs.zfs.vdev.sync_read_max_active : 1 ( default = 10 )
vfs.zfs.vdev.sync_read_min_active : 1 ( default = 10 )
vfs.zfs.vdev.sync_write_max_active : 1 ( default = 10 )
vfs.zfs.vdev.sync_write_min_active : 1 ( default = 10 )

Even if we have shortened the vdev max_active's so drastically ( from 10 to 1 in most cases ), our pool still delivers ~ 300 - 400 MBps when doing transfer directly from and to the head. And there are no longer 'bursty' spikes.

Exactly as in the "old" days :)

As this is a very preliminary result you should do a lot of testing yourselves.

But it is a start!

Best regards
Bjarke Emborg Kragelund

 

[jgreco]

You cannot have "battery-backed the NVRAM on your HBA". HBA's do not provide cache.

Please feel free to add commentary and a link to this thread to the end of bug 1531, which Jordan recently closed. I was not convinced that this was actually fixed, though it appears to be a ton better by default.

 

[jgreco]

However, I did spin up a new FreeBSD 9.2 system and those two sysctl's do exist there. Interesting.... wonder why they are missing from FreeNAS?

Because FreeNAS is using newer ZFS code than FreeBSD 9.2.

 

[jgreco]

Wow. Those settings are.... out there! Where did you get those settings from?

Probably originally sourced from the extensive discussion in bug 1531, which I believe fully explains the reasoning and effects.

 

[Bjarke Emborg Kragelund]

Hi jgreco et al,

I agree, the new 'tunes' are a great improvement. And the standard settings are much more sound.

The fact that it is now throttled per vdev and not per pool, is a major step forward.

I will be more attendant the next time I do a pool upgrade :)

Thx!
Bjarke Emborg Kragelund

 

[eraser]

Because FreeNAS is using newer ZFS code than FreeBSD 9.2.

Apparently it is also using ZFS code that is even newer then the version in FreeBSD 10.0-RELEASE (which only came out two months ago).

 

[jgreco]

Yes. It's called current code.

 

[eraser]

Yes. It's called current code.

So FreeNAS users are effectively beta testers for code that is not yet deemed ready for the FreeBSD STABLE/RELEASE branches?

http://www5.us.freebsd.org/doc/handbook/current-stable.html

 

[cyberjock]

So FreeNAS users are effectively beta testers for code that is not yet deemed ready for the FreeBSD STABLE/RELEASE branches?

http://www5.us.freebsd.org/doc/handbook/current-stable.html

That's one way to look at it. :P

 

[jgreco]

Is it different than the code in STABLE or CURRENT?

 

[cyberjock]

Is it different than the code in STABLE or CURRENT?

no clue.