Performance with Veeam

[DougATClark]

Hey all,

I have two FreeNAS boxes that I've inherited from a predecessor that are being used as iscsi storage for Veeam backup. I'm having some slowness with having the files merge together and their support is thinking that it may be an i/o issue with the storage (FreeNAS). I'm looking for any suggestions or help that anyone can provide on getting to the bottom of this. I'm going to apologize in advance in I mix up some of the terminology, trying to keep this all straight.

Since I upgraded from 9.x to 11.x I ran into alerts indicating out of chains, so my only tunables are dev.mps.0.max_chains and hw.mps.max_chains both are 4096. Thats seemingly made the daily warnings go away.

Hardware:
Supermicro X9SCL-F motherboard
Intel E3-1230 V2 CPU
32gb ECC RAM
LSI 9201-16i IT mode
16 WD30EFRX (WD Red) 3TB drives
Intel 82599ES 10gig card
OS installed on a 32gig Supermicro Sata-DOM

Setup:
FreeNAS 11.1-u1
Single Volume with a single dataset with a single zvol
Two 8 disk raidz2 arrays
lz4 compression on the dataset, no dedupe
9216 MTU on the 10gig cards ( confirmed functional jumbo frames end to end )

Reporting Charts:


There are a bunch of disk charts but they don't really lend themselves to being screen captured and posted up. I have a feeling they are going to be some of the important ones, is there a better way to dump the disk stats to the post?

thank you!

 

[bigphil]

Run the commands "zpool status" and "zpool list" on the FreeNAS box and post the output here in code tags.

 

[nightshade00013]

Well one thing that will be helpful since using iSCSI is how full the pool is. If you are over 50% usage it can cause issues since the system is working with blocks rather than data.

Code:

zpool list 
zpool iostat 1 
spool status

Use iostat while you are actively using the pool and let it run for a bit. The list will show how full and some other info and the list will show if there are errors. Post the info and lets see what you have.

https://forums.freenas.org/index.php?threads/useful-commands.30314/ has a lot of stuff that you will probably want to look at.

 

[DougATClark]

Here's the outputs, there are 3 of the merge jobs running right now so hopefully it will show what usage is:

 

[bigphil]

check "gstat" as well to make sure there is nothing wrong...like one disk is super busy and the others aren't. If nothing is obviously wrong there, it may be that you're hitting the limits of the disk system. I assume the merge you're talking about is "synthetic full backup" and in this case it's going to be quite an expensive operation on the disks as they are reading and writing from the same pool of disks at the same time.

 

[DougATClark]

Here's 4 different snips of it running:


Thanks for all the replies so far

 

[bigphil]

Something else to keep in mind is that if you are indeed doing synthetic fulls in a forever type rollup fashion, this is random IO intensive and with your pool layout you don't have a lot of IOPS. If you confirm that this is the type of backup job you have, I suggest switching to Active Full backups instead. This is sequential and will perform much better in your case.

EDIT: just saw your screenshots...your disks are very busy. I suggest you switch your backup strategy to Active Full instead of synthetic.

 

[DougATClark]

Something else to keep in mind is that if you are indeed doing synthetic fulls in a forever type rollup fashion, this is random IO intensive and with your pool layout you don't have a lot of IOPS. If you confirm that this is the type of backup job you have, I suggest switching to Active Full backups instead. This is sequential and will perform much better in your case.

The jobs are running their default, Incremental. The job description says best for targets with poor random I/O. In some reading today I'm finding that it may be recommended to get a low latency SSD for SLOG/ZIL. I do not have anything like that configured currently. Would my use case benefit from that?

 

[bigphil]

It depends on how your current config is. if you're using iSCSI through VMware with the zfs option sync=disabled or sync=standard, then SLOG will not help you and only slow you down. you can check with this command "zfs get sync VeeamVolume"

 

[DougATClark]

The output is showing "VeeamVolume sync standard default" so looks like its enabled. Now that you mention VMware i guess I left the host part out, whoops. I'm presenting the storage to a windows 2012 r2 based system using Microsoft's iSCSI initiator.

 

[bigphil]

Can you post a pic of the backup job options so we can confirm the settings. Assuming you're on Veeam 9+, open the job > storage > advanced > "backup" tab. Also...are you using CBT (changed block tracking)?

 

[DougATClark]

Yes, CBT is enabled.

 

[bigphil]

So you never do a full backup...just forever incremental? I wonder if the backup chain gets too long if it could cause a performance issue. You might throw an Active Full in there and see if performance is better after that. As for SLOG. You could set the sync option to "disabled" for that dataset and see if performance increases. Do some research on the ZIL and what the sync options does so you have a better understanding of what the implications are when configuring this setting.

 

[DougATClark]

Yeah, I mean when it hits the max retention it rolls it up... which is what it does on a daily basis now. That merge operation is whats taking many hours and what i'm trying to get to the bottom of.

 

[bigphil]

Ok, then you need to add an active full backup in there to prevent that. That should fix the problem.

 

[DougATClark]

Just as a followup to this thread if anyone else happens to find it in the future: Changing job types to enable full's on a regular schedule has indeed taken care of this.