480 TB z3 Backup Server

[CharlesN]

We are having to do this build DIY, we have talked to iXsystems and 45 Drives and other NAS systems. The main issue is cost at two or three times the budget. The NAS side has tech limitations like Max pool sizes ect. Please let me know what you think about the below config. We are also going to add a 2nd system in Q2 that will be identical to replicate to off-site. I am fairly new to ZFS and FreeNAS ( I do have an 8 bay T630 at home for the last 3 months and love it).

Use Case
Veeam Image backups (75 local vSphere) (150 remote PC's)
Cold/Warm storage for decommission server images
Cold/Warm Storage from DB Backups

Shares
NFS and MinIO

Pool Config
1 Pool 4 x VDEV (15 disk) z3 (480TB)

Host Server
Dell PowerEdge R630 6xSAS12 + 4xNVMe
2 x Intel E5-2667v4 3.2GHz/25M/2400MHz 8-Core 135W
256GB 8 x Dell 32GB DDR4 LRDIMM, 2400MT/s Load Reduced
1 x HBA330 12Gbps SAS HBA Mini Mono Controller (NON-RAID) ( Local SAS12 HBA for OS and Cache )
2 x Dell 1.6TB PCIe NVMe Mix Use TLC 2.5" SSD, PM1725a ( cache ? )
2 x Dell 480GB 12Gbps SAS Mix Use MLC 2.5 SSD PX05SVB048Y ( OS Install )
1 x Intel XL710 Dual Port 40Gb QSFP+ rNDC ( Dual-link to HA Switch Stack )
2 x Dell SAS 12Gbps Host Bus Adapter External Controller Low Profile ( 1 Card for each controller on JBOD multipath , one available port on each for expansion)
1 x iDRAC8 Enterprise
2 x Dell 1100W 80 Plus Platinum Efficiency Power Supplies
1 x 5 Years Dell ProSupport and Next Business Day On-site Service

DAS JBOD
WD Ultrastar Data60 (1ES0350)
60 bays (SAS-3) 4U
60 x 10TB SAS 12Gb/s 512e ISE

 

[jgreco]

Without taking the time to look up all the rest of the hardware specifics, ...

You might be inflating the cost unnecessarily with the SAS drives. Your massive JBOD is spec'd at around $40K MSRP (findable for around $24K). 600TB raw. Or if we look at pool space (480TB), $50 per TB.

A typical Supermicro 36-bay JBOD is around $2100. And if you don't mind shucking drives, a 12TB HDD can be gotten for $180 when on sale. This has tradeoffs of course, but the cost differential between 56 12TB SATA HDD's and two of those JBOD's is around $10K less than your big one. Or around $30 per TB. Downside is that you're providing your own warranty on the drives.

Depends how desperate you are to control costs, of course. Some of us fight to keep our dollars. :^)

 

[CharlesN]

I Completely agree that we could go More DIY, This is for production use and the 5-year warranty are important. We also have not been fans of supermicro. The thoughts on SAS were dual controllers and higher-end drives ( I know this can be debated for days but in production, DCs SATA does not fly in reviews).

I guess my main questions are about the Pool setup, caching and general specs of the Host.

 

[jgreco]

You're not fans of Supermicro but you're fine with Western Digital?

Okay, fine, whatever.

It's probably worth noting that multipath support in FreeNAS might be weak and maybe fragile, or at least confusing and not something most of the forum users will have experience with.

It's hard to know if you're going too big here. You've arrived at the 480TB by Veeam's sizing recommendations?

I'm not sure about the XL710.

As for the HBA's, be aware that you'll be required to flash the latest IT firmware on there. Whatever FreeNAS is expecting, I don't remember offhand for the 12Gbps HBA's. Be aware that Dell might not be so happy about that. If not, you may be better off buying the retail HBA's and flashing those.

 

[Jessep]

Veeam repositories work best on ReFS (merges, synthetic full, etc. are much faster, as in 2-5x faster for some things).

This would require iSCSI presented to a Windows Server 2016/2019 box to then format ReFS.

 

[CharlesN]

That's disappointing,