Pools and redundancy - getting clear on some technical points about ZFS

[Stilez]

I've read the docs and googled around, and I understand somewhat how pool layout would work. So for example, that the pool isn't redundant (only vdevs have redundancy); that data is striped or balanced across vdevs, that ZFS will attempt to even out its use of vdevs in some way or other (and data should therefore be balanced across vdevs); that ZFS can be manipulated via CLI but this may not always update FreeNAS internal knowledge of the pool; etc. What isn't clear to me is some of the detail:

1. Suppose a 4TB pool is made up of a 3TB + 1TB drive: will the data always be striped "arbitrarily" across them, or is there a choice to keep files within a single disk in the pool (the files in the pool are held some on each drive, but individual files are held either on one drive or the other)
   
2. Suppose a pool has 2 x 1TB and 2 x 2TB, is there any practical difference between configuring the pool as mirrored stripes vs striped mirrors? For example:
   ((1+1 mirrored) striped with (2+2 mirrored)), or
   ((1+3 striped) mirrored with (1+3 striped))
   
3. When a new disk/vdev is added, the docs say that ZFS tries to balance across disks, and therefore unbalanced disks will be inefficient. But what exact metric is ZFS trying to balance out - its write activities, the amount of used space, or the amount of free space? This distinction would be important as a workload or stored data varies between mostly-reads and mostly-writes.
4. If a server has multiple top-level volumes/pools, can a file system on one of them contain a symlink to a directory on another of them? If so, what issues might arise (if any)?

 

[fracai]

1) nope. If you want control over which device gets files you'll need to create a pool for each. When striped, each device will get writes proportional to it's share of available space.

2) I would think you'd want to go with the stripe of mirrors. I only have experience with Z2 though.

3) balancing refers to how ZFS will write to where the available space is. If you have a stripe of two 1 TB drives that is 50% full and replace one with a 6 TB drive. The new drive will get the majority of writes as there is more free space available there. This is also why I'd prefer the stripe of mirrors in (2). That way you're balancing the available space better.

4) a symlink will be fine; I can't think of any issues

 

[wblock]

1) Pretty sure ZFS gives no user control over what goes where. It does not make a huge difference, given that the volume fails if either drive fails. It's not like you you can put important files on one drive and still have them if the other fails. Please avoid RAID0. This sort of scheme has no redundancy. If you can live with that (please don't), add each drive as a separate volume, a "stripe of one disk". Then manually put files on one or the other. But again, prefer volumes with redundancy.

 

[DrKK]

Stilez:

These are really good questions. The answers aren't totally easy. Suffice it to say: It does what you logically expect it would probably do. The two books by Allan Jude and Michael Lucas (FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS) go into some detail as to how this works.

Generally: You have no control over which drives are used. If they are presently unbalanced (for example, you've added a fresh vdev to a well-seasoned vdev), then indeed almost all of the new data will go on the new vdev by design. If you start with a compound pool with multiple vdevs in it, the striping across them (all vdevs are striped together) is equitable.

 

[Stux]

A mirror of stripes is not a configuration you can make using pure ZFS.

You make a pool out of mirrors. The data is then striped across the mirrors, as ZFS sees fit, which in essence means it balances the writes out across the available space.

Or put another way, there is no way to make a vdev which contains vdevs.

 

[DrKK]

A mirror of stripes is not a configuration you can make using pure ZFS.

You make a pool out of mirrors. The data is then striped across the mirrors, as ZFS sees fit, which in essence means it balances the writes out across the available space.

Or put another way, there is no way to make a vdev which contains vdevs.

Yeah I was going to point this out myself.