No, its like this... 2 scenarios: 1 with ESXi exclusive and one with ESXi and normal users.
ESXi exclusive:
So if you have 10 VMs all doing sync writes with ESXi, the last thing you want to do is delay them. This does only one thing... makes those VMs go idle while waiting for their turn to write data. So obviously this is not too useful. The obvious answer is to upsize your server, add an L2ARC, etc. Trying to artificially choke VMs because one VM is particluarly busy doesn't solve your problem, it simply creates a new one(VM performance is in the crapper on purpose now).
ESXi with file sharing for normal users:
So you have 10VMs doing sync writes and the normal users using some number of shares. Here's the catch... normally, NFS shares don't have alot of sync writes. So those users aren't throwing lots of load at the server that requires a sync write, so the server can simply write the data at its next transaction. So the only thing you'd really be doing by throttling back sync writes is forcing VMs to go idle while waiting for their sync writes to complete since they are being throttled. And, they're being throttled for no reason because your other users don't have sync writes. So you're basically back to the "ESXi exclusive" category, with some extra writes that aren't even sync writes. Non-sync writes don't appreciably affect pool performance except in VERY large numbers. And if your numbers are that big, you should have built a bigger server to begin with. So who cares?
And you can even add a 3rd category: users that use any other sharing service than NFS(since they don't support sync writes). They're just like the above paragraph except that instead of saying "they don't have a lot of sync writes" you know for 100% certainty that they have zero sync writes. So they can
always undoubtedly wait for the next transaction to write their data without any waiting.
So I'll ask again.. how is this supposed to benefit anyone if you are either deliberately adding latency to a VM as a "just in case" another VM needs to do a sync write; or deliberately adding latency to a VM so that another user can write their non-sync write data to the server at its next regularly scheduled transaction anyway. Sounds like a no-win situation for both outcomes.
It looks to me like you're throttling back VMs on purpose, and for no benefit. On top of it, there is a technical solution to the problem, rightsize your server.
In their example it sounds like they have a single large pool that's divided up to different customers(presumably under contract or something) that don't even know who they are sharing resources with or that their server is actually shared with other users. The whole purpose of the QoS system they are discussing is so that if you have 10 different customers on the server 1 client can't make the server so busy that the other 9 clients can't use it. So you deliberately throttle one user so that the other users get a chance for server cycles. This makes sense in that the customers with expected loading can pay for their contract and enjoy the server resources. But the guy that wants more server resources will be throttled from excessive loading(and perhaps might want to purchase a larger contract, maybe even have a dedicated server all to themselves).
In short, what they are doing and what FreeNAS are doing is both using ZFS. But that's where the similarities end. The client side of the house is totally different(and that's where their QoS system allows them to be more profitable). Other than ZFS, they don't really share much else in common.
So, I'll go back to what I said in the first post unless you can explain where I am in error...
I'm not even sure what to say. I see no link between how they did their build and why you'd even want to think about throttling sync writes. They are using the same hardware and "selling" a service to various customers. They are doing what they are doing to ensure that each customer gets their fair share of I/O.
FreeNAS servers are usually used by the same customer(aka.. the company). So trying to "sell" service to different customers is stupid because you don't have different customers. You have one customer.. the company.
That's like you being upset because the Grocery store checkout lines are too long, so the store manager says "I know the solution" and immediately closes 1/2 of the checkout lanes. Does that even make sense? Hell no.