Is my available space correct?

[jgreco]

Yeah, a cursory search of "veeam nfs backup" reveals that NFS may not be an option for a Veeam backup repository on Windows. But that's outside my bailiwick; perhaps @Dan Lee will enlighten us about that.

Lazy Grinch didn't even notice the Veeam, or at least to remember it. Good catch, glad someone's checking my homework.

 

[Dan Lee]

wow, such great information; quite refreshing actually. I knew going into this project that there'd be a learning curve; heh. I believe that curve just got real steep. I'm running the sdelete -z option now and I'm sure that'll run for quite sometime but I will update you on the results of that task.

@Spearfoot and @jgreco The reason I went iSCSI instead of CIFS or NFS was because our main backup target is a Nexsan Storage Shelf with iSCSI connections. It felt like iSCSI was the easiest to incorporate because we already had the iSCSI vlan and network configuration in place and it was simply adding another target to the Windows iSCSI Initiator. Our daily/monthly production backups land there and this FreeNAS storage was to be an archive type backup keeping a monthly, quarterly and yearly backup copy of each backup job. Performance wasn't important, space utilization was. I also had a $1500 budget and was hoping to be able to hold around 12-14TB of veeam backup copies.

Having a better understanding of ZFS (but still barely enough to scratch the surface, just make me really dangerous) I think CIFS would probably be best case scenario as far as Veeam is concerned. I'm still wrapping my head around the mirrors and need to process the information that was linked above. But I do realize that a rebuild is in my very near future and I'm okay with that as I haven't lost any data; yet.

 

[jgreco]

So that's all good.

If I can take the stage for a moment, one thing I'd like to point out to you is that unlike a Nexsan storage shelf, which I'm guessing will be using SAS hard drives and pricey software, if it's anything like most of the other array vendors, you can help control costs in a ZFS environment by going cheap. Which I'm guessing you're already doing.

In particular, instead of a 4TB SAS HDD at $250-$400/ea, use of a lower grade drive like a WD Red NAS 4TB for $130 allows for greater amounts of raw storage to be included in a pool. Especially in an environment where performance isn't the key, what you do need to realize is that there's still a need to build in to your system design the things that make ZFS perform well. And you kinda have to not obsess over it too much.

For example, for good iSCSI performance, you shouldn't exceed around 50% of the pool capacity, so that ZFS has a better chance at being able to allocate space quickly. Also, mirrors perform better than RAIDZ, and have fewer "quirks". However, to get the same data protection level per vdev as RAIDZ2, you need a three way mirror. So here's the thing. Our VM filer here has 52TB of 2TB HDD's, all 2.5", in a 2U enclosure. 24 in front and two spare in back. These are grouped as eight vdevs that are each three-way mirrors. I can lose any disk without losing redundancy, which is the design goal. This gives a pool of 16TB. Of that, no more than 8TB should be used. So to deliver 8TB of high redundancy usable space, we use 52TB of raw disk.

Some people can't get over that. But the neat thing is, you don't actually need super-high-quality storage. And we don't use that. We use cheap 2.5" laptop hard drives. They're about half the speed of 2.5" SAS nearline "enterprise" grade drives. But they're also like 1/4th the cost. Better overall reliability characteristics, and I threw some of the saved money at high speed L2ARC, so the box has 1TB of L2ARC, which means that most of the time the array is *only* writing updates, and reads for all the commonly accessed stuff is being serviced by the ARC and L2ARC.

ZFS is piggy with resources, but if you play your cards right, you can get really sweet performance out of it and get it to do other amazing storage tricks. Make sure that when you're designing a system, that you give some thought to issues such as redundancy, what kind of accesses are going on, whether or not cheaper disk might be acceptable, etc.

 

[Dan Lee]

Status Update: sdelete finished it's work last night which freed up 497GB; FreeNAS is now Alerting to 94% spaced used and my Veeam backup copy worked last night. The conclusion being once there isn't any room, Windows will construe a volume with available space as write protected when the reality is the back-end is actually full. I'm still trying to wrap my head around how the space is set/assigned/decided. The variable size that I set on the zvol when creating the iSCSI share is completely up to me then? I could have put any number in there, say 50T and that's what would have been reported to Windows? But in reality, with the way ZFS utilizes space, knowing I only had 20TB of raw space but not wanting to use more than 50% for best performance then I should have set that option to 10T? This would have then shown Windows only a 10TB disk and FreeNAS wouldn't have gotten bent out of shape, right? However, because the Block Storage written to by Windows won't actually release data it's leaving ZFS out in the dark on what the front end is assuming. By switching to CIFS, ZFS will have more insight into the actual file operations and report the size changes appropriately. Is all that more or less accurate?

@jgreco In regards to your VM filer example you gave; I definitely fall into the group of people struggling with having to use 52TB of raw space for only 8TB of usable space. Because performance is of no concern for me in this; I would almost rather spin up a Windows Storage Server with RAID5 or maybe RAID6. How would one design a ZFS storage pool with adequate protection to loose a disk or two but not have to use a 4:1 ratio on raw to usable, or is that not possible with the mindset of how/what ZFS is designed to do.

 

[jgreco]

Status Update: sdelete finished it's work last night which freed up 497GB; FreeNAS is now Alerting to 94% spaced used and my Veeam backup copy worked last night. The conclusion being once there isn't any room, Windows will construe a volume with available space as write protected when the reality is the back-end is actually full. I'm still trying to wrap my head around how the space is set/assigned/decided. The variable size that I set on the zvol when creating the iSCSI share is completely up to me then? I could have put any number in there, say 50T and that's what would have been reported to Windows? But in reality, with the way ZFS utilizes space, knowing I only had 20TB of raw space but not wanting to use more than 50% for best performance then I should have set that option to 10T? This would have then shown Windows only a 10TB disk and FreeNAS wouldn't have gotten bent out of shape, right? However, because the Block Storage written to by Windows won't actually release data it's leaving ZFS out in the dark on what the front end is assuming. By switching to CIFS, ZFS will have more insight into the actual file operations and report the size changes appropriately. Is all that more or less accurate?

It's the correct high level understanding, yes. It gets more complicated when you're using RAIDZ because what's happening for block storage may or may not be optimal. It is possible to tease block storage and RAIDZ into consuming 100% additional overhead.

@jgreco In regards to your VM filer example you gave; I definitely fall into the group of people struggling with having to use 52TB of raw space for only 8TB of usable space. Because performance is of no concern for me in this; I would almost rather spin up a Windows Storage Server with RAID5 or maybe RAID6. How would one design a ZFS storage pool with adequate protection to loose a disk or two but not have to use a 4:1 ratio on raw to usable, or is that not possible with the mindset of how/what ZFS is designed to do.

It all depends on what you're trying to do. RAIDZ is the default choice for people who want to make the most of their raw disk, but the tradeoffs are low performance, and the fact that RAIDZ is really only good at storing large sequential files.

Your Windows Storage Server with RAID6 will deliver a much more consistently suck-tacular level of performance, at a cost. You lose all the cool features but you gain a certain kind of more-comprehensible predictability. To build that RAID6, you need to use a hardware RAID controller ($$$$) and enterprise class drives certified to work with it ($$$$). But pretty much any idiot can set it up.

ZFS was built on the idea that a modern CPU was an acceptable alternative to pricey specialized RAID silicon, so you pay more for the host system but you omit the RAID controller, and you can use cheap NAS or desktop grade disks. The problem is that ZFS fundamentally expects to be able to burn resources in order to deliver significantly enhanced performance. It is possible (note I didn't say easy) to get ~2000 random write IOPS out of a single hard disk if you play by ZFS rules, something you will not be doing on a conventional RAID controller and HDD. You get that by using a much larger disk than is needed, which allows ZFS to write contiguous transaction groups.

As previously mentioned, I sleep fine at night with the 52TB issue because I paid less for the 52TB than I would have for 8TB of storage and a RAID controller that performed at a similar level.

 

[Spearfoot]

Status Update: sdelete finished it's work last night which freed up 497GB; FreeNAS is now Alerting to 94% spaced used and my Veeam backup copy worked last night. The conclusion being once there isn't any room, Windows will construe a volume with available space as write protected when the reality is the back-end is actually full. I'm still trying to wrap my head around how the space is set/assigned/decided. The variable size that I set on the zvol when creating the iSCSI share is completely up to me then? I could have put any number in there, say 50T and that's what would have been reported to Windows? But in reality, with the way ZFS utilizes space, knowing I only had 20TB of raw space but not wanting to use more than 50% for best performance then I should have set that option to 10T? This would have then shown Windows only a 10TB disk and FreeNAS wouldn't have gotten bent out of shape, right? However, because the Block Storage written to by Windows won't actually release data it's leaving ZFS out in the dark on what the front end is assuming. By switching to CIFS, ZFS will have more insight into the actual file operations and report the size changes appropriately. Is all that more or less accurate?

@jgreco In regards to your VM filer example you gave; I definitely fall into the group of people struggling with having to use 52TB of raw space for only 8TB of usable space. Because performance is of no concern for me in this; I would almost rather spin up a Windows Storage Server with RAID5 or maybe RAID6. How would one design a ZFS storage pool with adequate protection to loose a disk or two but not have to use a 4:1 ratio on raw to usable, or is that not possible with the mindset of how/what ZFS is designed to do.

We don't want to scare you off from using FreeNAS! I don't presume to speak for @jgreco, but what he was describing is an optimized, fail-safe virtual machine server; which differs quite a bit from your use-case (i.e., reliable and capacious backup storage).

IMHO, your needs will be well-served using RAIDZ2, which has a much higher storage efficiency vs. the 3-way mirrors he described. If you were to replace all 6 of your 4TB drives with 8TB devices in a RAIDZ2 configuration, you would have the capacity of 4 drives; ~32TB of available space, less overhead. I've deduced from previous posts that your backup set size is ~8TB, so this configuration should give you plenty of space, even assuming you have no choice but to use iSCSI, which works best on sparsely populated pools. And if NFS or CIFS is an option, you could use a larger percentage of the pool's space without a crippling performance penalty.

 

[jgreco]

That could be true. If there was any time where iSCSI might work well with RAIDZ, it'd be for something like backups.

 

[Dan Lee]

8TB of backups is the low end and in all actuality; I'd like to get archival backups of all my backups which would amount to closer to 30TB if I were to include monthly, quarterly and yearly restore points of everything. I honestly completely underestimated the required raw usage when designing this solution having applied all my years of windows raid building assumptions to ZFS. I read through the manual and looked at configured systems but just didn't grasp the notion of how ZFS handles things. I'd assume that when a system is built using cheap 2TB 2.5" drives, they are only 5400rpm disks? It would seem that disk speed is of no consequence with the memory and cpu being leveraged for the majority of performance along with the density many drives. My chassis is a 16 bay chassis for 3.5" drives, so I'd probably add more disks than replace the Seagate ST4000DM000 Drives I'm currently using. If drive speeds don't matter, then I wont concern myself with RPMs, but look for density and capacity.

I can use NFS with Veeam; that's not a problem. Like I had stated earlier, the only reason I went with iSCSI was because that was what was done before me. I don't want to use iSCSI when I rebuild this; but a decision will be to use NFS or CIFS. iSCSI will apparently perform faster for Veeam but since the repositories aren't VMWare datastores; I loose the benefit of direct iSCSI access for maximum speed on backups.

 

[jgreco]

Drive speeds matter to some extent, but are not really a dominant factor. When you're building something with a conventional hardware RAID6 controller, your choices are slow, medium, and fast (5400, 7200, 10K+). ZFS can of course benefit from faster seek times, but consider this:

Three cheap disks that are each half as fast as a single expensive disk are up to 50% faster at reads than the single expensive disk, and much more reliable as a group.

For the stuff that you're reading all the time, fetching that from ARC (system memory) or L2ARC (cache SSD) is tons faster.

If write speed performance is important, moving from 50% full to 25% full (by doubling the drive size) approximately triples write speed once you reach steady state fragmentation levels:

while doubling the RPM of the disk from 5400 to 10K does not result in such a spectacular increase. Of course, you can increase *both* and that's still better.

 

[Spearfoot]

8TB of backups is the low end and in all actuality; I'd like to get archival backups of all my backups which would amount to closer to 30TB if I were to include monthly, quarterly and yearly restore points of everything. I honestly completely underestimated the required raw usage when designing this solution having applied all my years of windows raid building assumptions to ZFS. I read through the manual and looked at configured systems but just didn't grasp the notion of how ZFS handles things. I'd assume that when a system is built using cheap 2TB 2.5" drives, they are only 5400rpm disks? It would seem that disk speed is of no consequence with the memory and cpu being leveraged for the majority of performance along with the density many drives. My chassis is a 16 bay chassis for 3.5" drives, so I'd probably add more disks than replace the Seagate ST4000DM000 Drives I'm currently using. If drive speeds don't matter, then I wont concern myself with RPMs, but look for density and capacity.

I can use NFS with Veeam; that's not a problem. Like I had stated earlier, the only reason I went with iSCSI was because that was what was done before me. I don't want to use iSCSI when I rebuild this; but a decision will be to use NFS or CIFS. iSCSI will apparently perform faster for Veeam but since the repositories aren't VMWare datastores; I loose the benefit of direct iSCSI access for maximum speed on backups.

Ah, 16 drive bays! You could set up a pool with 2 vdevs, each vdev comprising 8 x 4TB drives in a RAIDZ2 configuration. That would give you ~48TB, less overhead, so your utililization would be a little over 60%. Not too bad.

Or... you could use a vdev of 4TB HDD (8 x 4TB RAIDZ2 giving ~24TB) plus a second vdev of 8TB HDD (8 x 8TB RAIDZ2 giving ~48TB) to get ~72TB of storage.

I don't believe you ever told us how much RAM you have; adding so much additional storage might require more RAM as well.

In any case, it sounds like you can upgrade your existing server and make it quite viable as a backup target.

 

[Dan Lee]

Ah, 16 drive bays! You could set up a pool with 2 vdevs, each vdev comprising 8 x 4TB drives in a RAIDZ2 configuration. That would give you ~48TB, less overhead, so your utililization would be a little over 60%. Not too bad.

Or... you could use a vdev of 4TB HDD (8 x 4TB RAIDZ2 giving ~24TB) plus a second vdev of 8TB HDD (8 x 8TB RAIDZ2 giving ~48TB) to get ~72TB of storage.

I don't believe you ever told us how much RAM you have; adding so much additional storage might require more RAM as well.

In any case, it sounds like you can upgrade your existing server and make it quite viable as a backup target.

My system has 32GB of non-ECC memory(please don't think too ill of me; i realize now it's the single point of failure) . My rebuild will consist of a new mobo with 32GB ECC memory.

 

[jgreco]

My system has 32GB of non-ECC memory(please don't think too ill of me; i realize now it's the single point of failure) . My rebuild will consist of a new mobo with 32GB ECC memory.

It's unlikely to be a significant issue in the short term as long as it's well-tested.