Production System Backup Server Design Questions

[Scharbag]

Hello.

I have long used TrueNAS at home but I am looking for some advice for a production storage system to store Veeam backups for work. Short history is we have an appliance that is not performing and we may want to repurpose the hardware. Hardware is:

Code:

SuperMicro X11DPH-T with dual Xeon Silver 4214 CPUs
SuperMicro 3U Case (cannot remember model right now - SAS3 expander)
1TB ECC RDIMM
Currently uses a RAID card but will be changed to 2 LIS-9300 HBAs (1@8i, 1@8e)
Supermicro 44 3.5" drive shelf
8@14TB SAS Drives
13@16TB SAS Drives
4@4TB SAS SSDs
10/25GigE Mellanox NIC (2 port)

Our SANs store about 120TiB of live data. A lot of it is vmdks as we are a mostly virtualized shop. We do also have Exchange, SQL and SAP.

So, while using the existing hardware, we were considering using ZFS for flexibility and reliability. Not sure how we will mount into Veeam yet - XFS on a ZVOL, iSCSI or NFS are all options but it would be great if others could provide some suggestions. I use NFS for my Veeam backups in my house and they work great. But, I am also not backing up 200VMs etc.

Next, deduplication... Is there ANY value in using dedupe in a backup system? Given we do have some larger SSDs and a boat load of RAM, it is tempting. But I have read that it is not really worth it (5G/T really sucks up memory fast). Perhaps it is just better to buy more 14TB or 16TB SAS drives?

Finally, vDev layout for maximizing ingest. TrueNAS seems to recommend 9+2+1 RZ2 vDevs for a Veeam solution. Is this still the case? With a 44 disk chassis, this would allow for 4 parallel vDevs in the pool (spares, L2ARC and special vDevs would be in the main 3U chassis). Based on this layout, with 16TB disks, we would effectively have ~500TB of space. Given ZFS slows down when it gets full, this should give us a bit of time and some history (Veeam is pretty efficient at dedupe too).

Anyway - any advice is greatly appreciated regarding a backup storage target.

Cheers,

 

[blanchet]

I backup 150 VM with Veeam and TrueNAS Core.
I have also a 44 disks JBOD with 18TB disks

I use 4 x 11 vdisk raidz3 for veeam backup

For Veeam backup the best setup is XFS Hardened Repository

Therefore I setup a bhyve virtual machine with Debian on TrueNAS Core (with serial console and without VNC)
+ XFS Hardened Repository
Use Zvol for the virtual disks.

SOLVED - Quick guide to install Debian 10 in bhyve with the serial console

Abstract To have a lighter Linux virtual machine, you can enable the serial console and then remove the VNC device after the operating system installation with VNC. But do you know that you can directly run the Debian 10 installer on the serial console without VNC ? It is faster because you will...

www.truenas.com

Veeam uses XFS fast clone to deduplicate data so you can store a long backup history without consuming too much space.

Stay away from iSCSI + ReFS

Windows Server 2022 ReFS: in-place upgrade issue (BSOD boot loop)

Veeam Community discussions and solutions for: Windows Server 2022 ReFS: in-place upgrade issue (BSOD boot loop) of Veeam Backup & Replication

forums.veeam.com

Stay away from ZFS deduplication

 

[Scharbag]

I backup 150 VM with Veeam and TrueNAS Core.
I have also a 44 disks JBOD with 18TB disks

I use 4 x 11 vdisk raidz3 for veeam backup

For Veeam backup the best setup is XFS Hardened Repository

Therefore I setup a bhyve virtual machine with Debian on TrueNAS Core (with serial console and without VNC)
+ XFS Hardened Repository
Use Zvol for the virtual disks.

SOLVED - Quick guide to install Debian 10 in bhyve with the serial console

Abstract To have a lighter Linux virtual machine, you can enable the serial console and then remove the VNC device after the operating system installation with VNC. But do you know that you can directly run the Debian 10 installer on the serial console without VNC ? It is faster because you will...

www.truenas.com

Veeam uses XFS fast clone to deduplicate data so you can store a long backup history without consuming too much space.

Stay away from iSCSI + ReFS

Windows Server 2022 ReFS: in-place upgrade issue (BSOD boot loop)

Veeam Community discussions and solutions for: Windows Server 2022 ReFS: in-place upgrade issue (BSOD boot loop) of Veeam Backup & Replication

forums.veeam.com

Stay away from ZFS deduplication

What sort of ingest rates does this system have?

 

[NickF]

Shameless plug...

Resource - TrueNAS SCALE: A “Datacenter-in-a-box"

As a pre-requisite, consider reading my TrueNAS SCALE virtualization guide! https://www.truenas.com/community/resources/getting-started-with-virtualization-on-truenas-scale.214/ This resource is meant to be a series. Part 1 will cover both the...

www.truenas.com

You may find this to be an interesting design. Don't need to do alot of these things at the application layer with Veeam, at least not when you have ZFS.

 

[blanchet]

What sort of ingest rates does this system have?

Item	Description
Source	All-flash-array Netapp C190
Proxy	Veeam as a VM on VMware cluster
Network	10 Gbps
Target	Bhyve VM 2 vCPU, 8 GB of RAM, vdisk 80TB
Processing Rate	130 Mb/s - 250 Mb/s

In my case, the primary bottleneck is the proxy.
The processing rate depends on the number of concurrent jobs.

 

[Scharbag]

Thanks for the info. We need to get to an ingest rate of ~1GBps to make this stuff work. We can run many proxies on our hosts which all have 10/25GigE NICs.

At some point, I think we just need to budget for a proper backup like Exagrid or StoreOnce as all of these home grown solutions seem to be lacking in ingest rate.

Cheers,