rungekutta
Contributor
- Joined
- May 11, 2016
- Messages
- 146
Hi,
I’m intrigued by the new special allocation class vdevs (catchy name!) but have some questions for this great community if anyone can chip in...
-First of all; my TrueNAS is used to store VMs hosted on ESXi. Also acts as media server, backup server for all desktops and laptops, etc. Raidz2 x6 and ssd slog. Performance has been ok, although not fantastic when several VMs are doing heavy i/o at the same time. In any case I enjoy tinkering and would like to see if I can push it further.
- ESXi keeps the VMs in vmdk files on an NFS share. Record size on that dataset is 16k.
- First question; since ESXi runs the VM’s file system inside a flat (vmdk) file on the NAS, presumably it constantly reads and writes in small chunks at different locations across the file. And given the 16k record size, presumably the files (many gigabytes obviously) are split evenly into many 16k block, as TruesNAS/ZFS has no understanding of the contents of the file. So related to that - presumably the “small file blocks” in the special vdev would be of little use here as all blocks are 16k, so I would either have to set the threshold to catch all of them (then a in effect moving all VM storage to the special vdev), or none? Am I thinking about this in the right way?
- And second question, somewhat related. Do you think metadata storage on an ssd special vdev would be helpful here? Or would the metadata typically be small enough to fit in RAM (ARC) anyway? (currently 6x3TB pool and 32GB RAM in my case).
In short I can totally see the strength of the special allocation class vdev in a use case where the ZFS file system is used directly by the host OS and in situations requiring small file i/o (e.g. compiling stuff). I just don’t really understand if the same benefits can be reached for hosted VMs via NFS.
Thoughts, or direct experience?
I’m intrigued by the new special allocation class vdevs (catchy name!) but have some questions for this great community if anyone can chip in...
-First of all; my TrueNAS is used to store VMs hosted on ESXi. Also acts as media server, backup server for all desktops and laptops, etc. Raidz2 x6 and ssd slog. Performance has been ok, although not fantastic when several VMs are doing heavy i/o at the same time. In any case I enjoy tinkering and would like to see if I can push it further.
- ESXi keeps the VMs in vmdk files on an NFS share. Record size on that dataset is 16k.
- First question; since ESXi runs the VM’s file system inside a flat (vmdk) file on the NAS, presumably it constantly reads and writes in small chunks at different locations across the file. And given the 16k record size, presumably the files (many gigabytes obviously) are split evenly into many 16k block, as TruesNAS/ZFS has no understanding of the contents of the file. So related to that - presumably the “small file blocks” in the special vdev would be of little use here as all blocks are 16k, so I would either have to set the threshold to catch all of them (then a in effect moving all VM storage to the special vdev), or none? Am I thinking about this in the right way?
- And second question, somewhat related. Do you think metadata storage on an ssd special vdev would be helpful here? Or would the metadata typically be small enough to fit in RAM (ARC) anyway? (currently 6x3TB pool and 32GB RAM in my case).
In short I can totally see the strength of the special allocation class vdev in a use case where the ZFS file system is used directly by the host OS and in situations requiring small file i/o (e.g. compiling stuff). I just don’t really understand if the same benefits can be reached for hosted VMs via NFS.
Thoughts, or direct experience?
Last edited: