Replacing HDDs in Offsite Backup - Best Practice Question

[fricker_greg]

those three drives with the checksum error were the last three that were added and resilvering during the reboot

 

[Chris Moore]

I am scrubbing the pool now again, and I will follow up with long smart tests of the drives, but is this just an error due to shutting down during the resilver and one that I don't need to worry about since it fixed the problem?

I would do the zpool clear as that would 'clear' the error and then if you do a scrub with no new errors, you should be good

 

[fricker_greg]

I would do the zpool clear as that would 'clear' the error and then if you do a scrub with no new errors, you should be good

Thank you, I will do that now

 

[Chris Moore]

PS. You have to give the pool name with zpool clear, for example zpool clear tank

 

[fricker_greg]

I will follow up on above, but one last question here, I can see in the top level of my pool that the TiB corresponds to what it should for 6 8TB HDDs, but the 19.8TiB after accounting for my raidz3 pool is only 21.7TB, is that due to the inefficiency of having a 6 drive raidz3?

 

[Chris Moore]

I will follow up on above, but one last question here, I can see in the top level of my pool that the TiB corresponds to what it should for 6 8TB HDDs, but the 19.8TiB after accounting for my raidz3 pool is only 21.7TB, is that due to the inefficiency of having a 6 drive raidz3?

The numbers you say, "19.8TiB after accounting for my raidz3 pool is only 21.7TB," don't appear to align with the graphic you posted, so it leaves me wondering what you mean. Here is the rundown, based on my own calculations starting simply with the number and size of drive you used.

ZFS makes a reservation of space in your 6 drive pool for 'parity an padding' based on your desire to have 3 drives of redundancy (RAID-z3). Part of that data is the redundancy data that allowed you to remove three drives and still recover the array, but another component of that data is the checksum data that is stored to allow ZFS to determine with accuracy whether the data that was retrieved is the same as what was stored. The result is, even though you have about 32.5TiB of disks in the pool, about 17TiB of that space is reserved and only about 15TiB is available to you for storage. Even so, you should keep utilization under 80% (for performance reasons) which drops the number even lower, leaving you with only 11.7TiB (roughly) that you should use. It is this sacrifice of capacity to redundancy that causes many people to choose RAID-z2 instead of RAID-z3. With the system being remote, I can understand the desire to have more redundancy but in this case it leaves you with only the rough capacity of two out of your six drives that can be used for data.

Did that clarify the situation?

PS. In practice, you can be between 80% and 90% utilization but above 90% the pool performance is supposed to drop off significantly because the algorithm used to store data is changed.