Awesome info on all fronts jgreco. One last question before I can set my mind down and actually get some work done today. Why is it that when I turn compression off I get higher measured IOPs using zpool iostat 1 than with it on? I read the articles above but unless I missed it I didn't see that referenced.
Offhand guess would be that when you compress the data you're probably pushing less data to the pool. If you disable compression, more data is stored. More IOPS. Not useful IOPS though, since it's more IOPS for about the same amount of work.
Also, a poster above said to set sync=always but I am not sure where to do that. I don't have any datasets yet, just a volume:
I understand vdev, zpool, some what of the SLOG and ZIL, but zvol and dataset is a little confusing.
ZFS creates its big storage pool out of vdevs. Looking at the big picture of the "back end", the goal of ZFS is to create a storage pool and then be able to do useful things with the storage. The characteristics of the pool are defined by the choice of vdev type, and any modifiers such as SLOG or L2ARC.
You can grow the pool at any time by adding more vdevs. You cannot remove vdevs, a significant weakness of ZFS currently.
Now let's move to the "front end." ZFS allows us to use its pool and allocate those resources to storage needs.
Think of a dataset as being a filesystem created using (part of?) the pool as storage.
A zvol is a block storage abstraction that you create using (part of?) the pool as storage.