Hello Everyone,
I am having an issue with Veeam backup performance. It lists our bottleneck as Target for every backup job, and our backup jobs are taking way longer than expected. We just built out an entirely new Veeam/backup infrastructure a few weeks ago and want to see if anyone has an ideas on how we can improve our speed. Here is an overview of our setup:
We have 4 separate physical locations across the US. Sites 1-3 are production sites, and site 4 is a dedicated DR site. All sites are connected with IPSec VPN Tunnels.
Each site has a Dell R740XD2 server with the following specs:
- Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
- 4 x 32GB DDR4 ECC Ram
- 24 x 18TB storage drives
- 2 x Samsung 256GB SSD boot drives
- Dual 10G NIC cards
Each of these 4 servers is running TrueNAS Core with all of the storage drives in a single pool (2 x RAIDZ2 VDEVs) totaling about 300TB of useable space.
Our Veeam server is running as a VM on a separate host at the DR (4th) site. Each of the 3 production sites has a Veeam Proxy / Wan Accelerator VM setup so that the backups can process locally (and then subsequently be copied offsite).
Here is an example of the issue I am facing. A job that processes about 15VMs took 23 hours and 43 minutes to run an INCREMENTAL backup. The job had a process rate of 17MB/s which I feel is really slow for an environment that is all SSD storage and 10G network. It lists the Target as the bottleneck; however, when I look at the TrueNAS metrics I barely see any utilization.
- CPU usage in TrueNAS is always below 10% (most of the time sitting at 0%)
- RAM Usage is below 70%
- The 10G NIC is hovering between 1-7MiB/s inbound traffic when processing backups
- Disk Latency is below 10ms during backup
- Disk Busy is below 1% on average during backup
A few notes about my setup
- The repositories are setup using a NFS connection
- My backups are in reverse incremental mode
- Compression level is set at optimal
- Backup proxy is on a different VLAN than the hosts being backed up, but if that was causing an issue I would assume it would list Proxy or Network as the bottleneck
I am beating my head against the wall trying to figure out why TrueNAS shows practically no utilization, but Veeam is listing it as 99% of the bottleneck of the job.
Maybe I am just over-estimating the speed at which these backups are able to processes, I know reverse incremental is slower, but almost 24 hours to run a incremental backup seems insane. I don’t know if I need to change the repository to SMB instead of NFS as a test? I have seen people saying to NEVER use SMB because it can cause corrupt backups, but other people say NFS is not developed well and can lead to performance issues. Any input or troubleshooting steps would be greatly appreciated.
I am having an issue with Veeam backup performance. It lists our bottleneck as Target for every backup job, and our backup jobs are taking way longer than expected. We just built out an entirely new Veeam/backup infrastructure a few weeks ago and want to see if anyone has an ideas on how we can improve our speed. Here is an overview of our setup:
We have 4 separate physical locations across the US. Sites 1-3 are production sites, and site 4 is a dedicated DR site. All sites are connected with IPSec VPN Tunnels.
Each site has a Dell R740XD2 server with the following specs:
- Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
- 4 x 32GB DDR4 ECC Ram
- 24 x 18TB storage drives
- 2 x Samsung 256GB SSD boot drives
- Dual 10G NIC cards
Each of these 4 servers is running TrueNAS Core with all of the storage drives in a single pool (2 x RAIDZ2 VDEVs) totaling about 300TB of useable space.
Our Veeam server is running as a VM on a separate host at the DR (4th) site. Each of the 3 production sites has a Veeam Proxy / Wan Accelerator VM setup so that the backups can process locally (and then subsequently be copied offsite).
Here is an example of the issue I am facing. A job that processes about 15VMs took 23 hours and 43 minutes to run an INCREMENTAL backup. The job had a process rate of 17MB/s which I feel is really slow for an environment that is all SSD storage and 10G network. It lists the Target as the bottleneck; however, when I look at the TrueNAS metrics I barely see any utilization.
- CPU usage in TrueNAS is always below 10% (most of the time sitting at 0%)
- RAM Usage is below 70%
- The 10G NIC is hovering between 1-7MiB/s inbound traffic when processing backups
- Disk Latency is below 10ms during backup
- Disk Busy is below 1% on average during backup
A few notes about my setup
- The repositories are setup using a NFS connection
- My backups are in reverse incremental mode
- Compression level is set at optimal
- Backup proxy is on a different VLAN than the hosts being backed up, but if that was causing an issue I would assume it would list Proxy or Network as the bottleneck
I am beating my head against the wall trying to figure out why TrueNAS shows practically no utilization, but Veeam is listing it as 99% of the bottleneck of the job.
Maybe I am just over-estimating the speed at which these backups are able to processes, I know reverse incremental is slower, but almost 24 hours to run a incremental backup seems insane. I don’t know if I need to change the repository to SMB instead of NFS as a test? I have seen people saying to NEVER use SMB because it can cause corrupt backups, but other people say NFS is not developed well and can lead to performance issues. Any input or troubleshooting steps would be greatly appreciated.
