L2ARC would be almost completely useless on a server soaking up backups (dedup exempted).
L2ARC would be mostly useless on a general purpose fileserver with light to moderate usage patterns.
L2ARC becomes fairly useful on a very busy departmental fileserver where the pool is being hit hard enough to be >80% busy for substantial periods.
ARC and L2ARC are usually very important on a highly fragmented VM datastore pool.
Absent any sort of context about what the purpose of the filer is, I can give you any answer on the spectrum from yes to no and be totally correct for some use model that is not yours. I don't care to waste my time explaining all the possible scenarios, because I could write a small book trying to cover them all.
By the way, how'd you end up with 148GB of RAM? 128 + 16 + 4 ...?
For an environment with 25-30 semi-active VM's, what you probably want to do is to figure out an estimate of how much data is actually being accessed on a regular basis. This is called the "working set." You can define "regular basis" as desired, that could mean "once an hour" or "once a day." And of course it is more than just a bit of a guess. You can take a little bit of guidance from the solid state hybrid disk guys for things like booting a Windows box, where they figured that 8-24GB of flash was sufficient to make their devices feel really fast. The difference with ZFS is that because it is a COW filesystem, what might seem to be linear read requests of the VM disk file are actually highly fragmented, and L2ARC is the mitigation strategy for that, so the SSHD numbers are probably a lowball estimate of what you want per VM.
So, what you want for your pool are mirror vdevs, and disks that are large enough to make sure your pool is only ~25-50% full when loaded up with the stuff you want to use it for. Keeping pool utilization low boosts write speeds by allowing contiguous space allocation. For example, if you want 8TB of datastore storage, expect to be buying 32-64TB of raw hard disk, depending on how zippy you want it to be.
If you had 8TB of datastore storage, you might figure that as much of 1/8th of that is your working set, so you definitely want to go with at least 500GB L2ARC, but you may even have enough RAM to go out to 1TB L2ARC, depending on how you configure your pool. It's very important to note that your options are contingent on your design choices. If you go putting 1TB of L2ARC on a 128GB RAM system with iSCSI and a 4K volblocksize, you will probably melt down, for example.
But, yes, unless you really want to be filling most reads from your pool storage, you do want L2ARC for VM storage. You want as much as you can safely support and reasonably afford, in most cases.
