I've spent the last day or so reading up on the different types of vdevs in a pool and I'm basically just looking for someone to confirm or correct my understanding. I also have a couple of questions that I'll get to after.
vdev types:
Data: Stores the files themselves, and everything else if no special vdevs are used.
Cache: I believe this is what people refer to as L2ARC, basically a pool-specific extension of the RAM-based ARC. Can improve read speeds by caching some files on higher speed drives. Should not be used on a system with less than 32/64GB (couldn't find a strong consensus there) or it may hurt performance by using up RAM. Should be less than 10x the total system RAM in size. Should be high speed and high endurance (since it's written to a lot), but failure isn't a huge deal as it won't cause data loss. This won't really do anything unless the system is getting a lot of ARC misses.
Log: I believe this is what people refer to as SLOG, a separate, higher speed vdev for write logs. Can improve speeds for synchronous writes. A synchronous write is when the ZFS write-data (not the files themselves, but some sort of ZFS-specific write log) is written to the RAM cache (ARC) and the pool (ZIL or SLOG if available) at the same time, vs an asynchronous write where it's written to ARC, then eventually gets moved to the pool. SLOG basically replaces the ZIL, but with faster storage, allowing sync writes to complete faster. Should be high speed,but doesn't need to be super high endurance like cache, since it sees a lot less writes. (Edit: I don't actually know this to be true. jgreco's guide on SLOGs says it should be high endurance, so maybe I don't understand exactly what the 'intent log' data is) Won't do anything for async writes, and general file storing is usually mostly async.
Hotspare: A backup physical drive (or multiple drives) that are kept running, but no data is written to. In the event of a disk failure, the hot spare can be used to replace the failed disk without needing to physically move any disks around. Hotspare disks should be the same disks as whatever disks they will replace.
Metadata: A Separate vdev for storing just the metadata of the main data vdev(s), allowing it to be run on much faster storage. This speeds up file browsing or searching, as well as reading lots of files (at least, it speeds up the locating of the files, not the actual reading itself). If this vdev dies, the whole pool dies, so this should be a 2/3-way mirror. Should be high speed, but doesn't need super high endurance like cache.
Dedup: Stores the de-duplication tables for the data vdev(s) on faster storage, (I'm guessing) to speed up de-duplication tasks. I haven't really come across many posts about this, so I don't really know what the write frequency looks like.
I've also seen people mention small-file vdevs as a way to improve read/write speed for lots of small files. I've also seen mentions of special vdevs, which seem to be a combination of metadata and small-file, however, I haven't been able to find anything like that in the current TrueNAS Scale GUI. I'm interested in the small-files vdev and it seems like maybe that's part of what the metadata vdev does, but I can't seem to find a concrete answer on that. Does the small-file vdev still exist? If so, how do I go about that?
Also, I've seen some people saying that a metadata vdev can increase overall speed of a pool, but from my understanding, it only allows the files to be searched and found faster, not actually written or read any faster. Does this increased finding speed actually make a big difference in general pool usage? Or does it speed things up in another way?
Sorry for the long post, I just didn't want to make a bunch of smaller posts since these are all related.
vdev types:
Data: Stores the files themselves, and everything else if no special vdevs are used.
Cache: I believe this is what people refer to as L2ARC, basically a pool-specific extension of the RAM-based ARC. Can improve read speeds by caching some files on higher speed drives. Should not be used on a system with less than 32/64GB (couldn't find a strong consensus there) or it may hurt performance by using up RAM. Should be less than 10x the total system RAM in size. Should be high speed and high endurance (since it's written to a lot), but failure isn't a huge deal as it won't cause data loss. This won't really do anything unless the system is getting a lot of ARC misses.
Log: I believe this is what people refer to as SLOG, a separate, higher speed vdev for write logs. Can improve speeds for synchronous writes. A synchronous write is when the ZFS write-data (not the files themselves, but some sort of ZFS-specific write log) is written to the RAM cache (ARC) and the pool (ZIL or SLOG if available) at the same time, vs an asynchronous write where it's written to ARC, then eventually gets moved to the pool. SLOG basically replaces the ZIL, but with faster storage, allowing sync writes to complete faster. Should be high speed,
Hotspare: A backup physical drive (or multiple drives) that are kept running, but no data is written to. In the event of a disk failure, the hot spare can be used to replace the failed disk without needing to physically move any disks around. Hotspare disks should be the same disks as whatever disks they will replace.
Metadata: A Separate vdev for storing just the metadata of the main data vdev(s), allowing it to be run on much faster storage. This speeds up file browsing or searching, as well as reading lots of files (at least, it speeds up the locating of the files, not the actual reading itself). If this vdev dies, the whole pool dies, so this should be a 2/3-way mirror. Should be high speed, but doesn't need super high endurance like cache.
Dedup: Stores the de-duplication tables for the data vdev(s) on faster storage, (I'm guessing) to speed up de-duplication tasks. I haven't really come across many posts about this, so I don't really know what the write frequency looks like.
I've also seen people mention small-file vdevs as a way to improve read/write speed for lots of small files. I've also seen mentions of special vdevs, which seem to be a combination of metadata and small-file, however, I haven't been able to find anything like that in the current TrueNAS Scale GUI. I'm interested in the small-files vdev and it seems like maybe that's part of what the metadata vdev does, but I can't seem to find a concrete answer on that. Does the small-file vdev still exist? If so, how do I go about that?
Also, I've seen some people saying that a metadata vdev can increase overall speed of a pool, but from my understanding, it only allows the files to be searched and found faster, not actually written or read any faster. Does this increased finding speed actually make a big difference in general pool usage? Or does it speed things up in another way?
Sorry for the long post, I just didn't want to make a bunch of smaller posts since these are all related.
Last edited: