ZFS Disk Numbers

[londadmin]

Hello All,

I am preparing to install FreeNAS on a system for a Veeam backup target. We will be utilizing a SuperMicro CSE-847BE2C-R1K28LPB (similar on NewEgg). We will have 36x 6TB disks. I need as much space as I can possibly get, while keeping some fault tolerance. I have read that ~12 disks is the most you want for a vdev. Is this due to I/O issues, fault tolerance issues, or another reason? As a backup storage target, IOPs aren't hugely important. Is it okay to have a 36 disk vdev, attached via iSCSI to the Veeam box, or should we split these? We have about 20-30TB of VMs that need backing up. With this much data, I feel like a pretty well thought out retention/archive plan needs to be implemented either way (maybe tape). Sorry if any of this is general knowledge, I have only ever created small FreeNAS solutions. Thanks for the assistance. Please let me know if I've missed anything.

londadmn

 

[MatthewSteinhoff]

I don' think there is a technical reason you can't put 36 6T drives in a RAIDZ2 array. That would give you 204T of storage, give or take.

But, man, it just feels wrong.

I know you say performance and IOPS are way down on the list but a single RAIDZ2 pool will feel REALLY slow. Especially if you ever have to do a restoration. With 36 drives spinning in the same pool, statistically, you'll have failures and performance will be brutal during drive replacement.

I'd do three, 12-drive groups into one pool for 180T of storage. Doing three groups will give you three times the throughput and IOPS. Instead of cratering performance during a drive replacement, it will only be one-third as bad.

Cheers,
Matt

 

[londadmin]

Thanks Matt! I guess that shows my inexperience. I thought the "drive groups" were the pool. So I can create 3 RAIDz2 drive groups, and make them one pool? And it would present to Veeam as one large drive over iSCSI?

 

[MatthewSteinhoff]

Yes.

Devices (disks) make up VDEVs which make up Volumes on which Datasets live

In our case - a much smaller case - we have a Volume made up of two VDEVs. Each of those VDEVs is a six-disk RAIDZ2 group. For us, that balances capacity with performance.

If you don't need all 200T of space on day one, you may want to think about staggering your disk purchases. You could build the first RAIDZ2 group of 12 drives giving you 60T of storage space. When you're 75% full, can add another 12-disk RAIDZ2 VDEV to the existing pool and have it expand. Best of all, you're not stuck buying 6T drives. A year from now, you may be able to purchase 8T drives for the same price. You could start with 60T, add 12x8T drives and end up with a 140T Volume. Best of all, adding a second VDEV to your pool will double your IOPS and throughput (give or take reality).

Cheers,
Matt

 

[londadmin]

Awesome. Sounds good Matt. Thanks again. One more quick question. I have been told that formatting the iSCSI volume on Server 2016 as ReFS can speed up backups. Do you know if this has any truth? Or is this more of a Veeam question?

 

[MatthewSteinhoff]

I'm not the right guy for that question since I don't really use any of those components.

(What little I know about ReFS is that it is slower than NTFS when integrity checking is turned on and faster without integrity checking. Since ReFS duplicates what ZFS is doing behind the scenes, ReFS might be faster if you run it without those duplicated features. But, yeah, check with someone who actually knows.)

Cheers,
Matt