Running with sync disabled for backups only?

[sonicc]

A little background:
FreeNAS 9.10 installed on a custom desktop PC
16GB RAM
4x 2TB SATA 7200rpm drives in Raidz2
MB: Asus P8H61-M LX2 (https://www.asus.com/Motherboards/P8H61M_LX)
CPU: Pentium G620 (http://ark.intel.com/products/53480/Intel-Pentium-Processor-G620-3M-Cache-2_60-GHz)

I've setup a FreeNAS device to be used as storage device for backups for VMWare Images. I have a script that runs every night, takes a snapshot, and then copies the virtual disks to the backup device.
Previously, the backups were stored on a Synology Device which was fairly quick - only have about 4-5 VMs that need to be backed up anyway and was able to hit 400-500 mpbs transfer speeds which was good enough.

After setting up the new FreeNAS device and attempting to back up to it, I noticed the transfer speed was hovering around 30-40 mbps. This made no sense, because even with low specs of the hardware I used for FreeNAS device it should at least be the same as the Diskstation which has about 1GB of RAM. So I tested the ports made sure everything is on 1Gbit network, and when tested with iperf I get 800mbps - 1gbps speeds from FreeNAS to a VM on one of the hosts and back. So no issues with hardware.

Looking more into the issue, I learned about disabling sync. This improved the speeds from measly 30-40mbps to 200-400mbps. But many people say this is a fairly risky thing to do...which makes sense to me if you run "live" VMs from the FreeNAS device.

My question is, does it make much difference if this device is just used for backups? Does sync matter much in this situation? Also, would upgrading t0 16GB of RAM help at all?

Thanks in advance.

 

[sonicc]

Additionally, with sync disabled the Interface Traffic looks like this:

and I see massive write latency in VMWare...in some cases up to 70,000ms

If I enable sync, latency doesn't go over 20ms and the graph is more consistent without spikes like in the above pictures, but I can only get about 30-40mbps transfer speeds

Am I just running into the issue of hard drives not being able to keep up with data that's being dumped to FreeNAS?

 

[Robert Trevellyan]

You might need to tune your transaction group settings.

 

[SweetAndLow]

Intel nic would fix most of your problems I think. You can disable sync of you want or disable it on the client using mount options.

Sent from my Nexus 5X using Tapatalk

 

[Spearfoot]

Am I just running into the issue of hard drives not being able to keep up with data that's being dumped to FreeNAS?

I don't believe so, since even the slower 3Gb/s SATA disks can typically saturate a gigabit connection.

How, exactly, are you copying the image files? Rsync? Something else?

 

[sonicc]

You might need to tune your transaction group settings.

I'll have to look into this. What do you suggest adjusting it to?

Intel nic would fix most of your problems I think.

Any particular Intel NIC or just a PRO gigabit?

How, exactly, are you copying the image files? Rsync? Something else?

Not sure if you're familiar/heard of it, the script is ghettoVCB. It is run by MKSBackup from a remote machine. On the basic level, it uses vmkfstools to thin the disk and copy it to the backup datastore (FreeNAS)

 

[sonicc]

If I added a 120GB SSD as ZIL/SLOG would that help to hit 200-300mbps sustained with sync enabled? That should be good enough for backups.

Also, I've ordered a Intel Gigabit card.

 

[SweetAndLow]

You can easily get 1gbpa speeds without a slog device. I would disable sync and keep testing until you find your bottleneck. A slog will not be faster than disabling sync. Try testing with dd to get an idea of local write speeds. Example dd if=/dev/zero of=./testfile bs=1M count=10000

Make sure to test on a dataset with compression disabled.

Sent from my Nexus 5X using Tapatalk

 

[HoneyBadger]

To answer your main question, sync=disabled on a backup target is perfectly fine in my opinion. In case of a failed copy, you still have the source snapshot to pull from again.

Need some more detailed specs. What model of Pentium and motherboard is in play here, and what size/model are the SATA drives? I just want to make sure you're not running Core2-era gear here.

I'll have to look into [tuning transaction groups]. What do you suggest adjusting it to?

How full are your disks? A four-drive RAIDZ2 shouldn't have any trouble keeping up with the ~100MB/s stream that 1GbE provides, and your net graph is capping out at ~600Mbps or about 60MB/s.

 

[Robert Trevellyan]

What do you suggest adjusting it to?

You may see more consistent results by reducing the duration from 4 seconds to, say, 1 second. I seem to remember @jgreco writing about this a couple of times. I don't know the details - ZFS tuning is not something I have any expertise with.

EDIT: here's some background: http://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/

 

[sonicc]

To answer your main question, sync=disabled on a backup target is perfectly fine in my opinion. In case of a failed copy, you still have the source snapshot to pull from again.

Need some more detailed specs. What model of Pentium and motherboard is in play here, and what size/model are the SATA drives? I just want to make sure you're not running Core2-era gear here.

How full are your disks? A four-drive RAIDZ2 shouldn't have any trouble keeping up with the ~100MB/s stream that 1GbE provides, and your net graph is capping out at ~600Mbps or about 60MB/s.

I've also updated specs in OP:
4x 2TB SATA 7200rpm drives in Raidz2
MB: Asus P8H61-M LX2 (https://www.asus.com/Motherboards/P8H61M_LX)
CPU: Pentium G620 (http://ark.intel.com/products/53480/Intel-Pentium-Processor-G620-3M-Cache-2_60-GHz)

Currently using about 1TB out of some ~6TB available.

My issue with the graph is that with sync disabled it shows spikes like that of ~600mbps, then nothing, then spike again. With synch enabled, the graph is a constant 30-40mbps line, without it dipping like that.

 

[sonicc]

Installing Intel NIC seemed to have helped a little. For now, backups are finishing in about 6hrs which is good enough.

Still can't figure out what kind of limitation i'm hitting. Might just upgrade to a server board with Xeon in a couple of months.

 

[HoneyBadger]

Have you done a local dd test on the machine itself to see what kind of numbers the local drives are able to sustain?

If you're only on single-link gigabit, I'd think your network would be the bottleneck first.

 

[sonicc]

When I run dd test:

# dd if=/dev/zero of=testfile bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.384110 secs (133295159 bytes/sec)

# dd if=testfile of=/dev/zero bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.698456 secs (73304541 bytes/sec)

So it seems pretty decent.

As for the network, I don't think it's a bottle neck as it is not staying maxed out. A gigabit network should be able to stay at ~600mbps consistent, right? Also this is a dedicated gigabit network between the VMWare hosts and FreeNAS machine, nothing else on it. The issue I'm running is the network spikes of ~600mbps followed by latency which shows up in VMWare up to 60,000ms. One other thing I am suspecting is that the onboard SATA controller can't keep up...

 

[SweetAndLow]

Your dd tests are very slow and you did the read test wrong. The of should be /dev/null and make sure to write 50-100GB for a good test. With your layout I think 300MB/s is a good speed to see.

Your network should be able to run at 900mbps non stop.

Sent from my Nexus 5X using Tapatalk

 

[sonicc]

I wanted to post an update to this thread. I have finally resolved this issue.

It seems the problem was with dedup being enabled on the dataset and I had nowhere near the memory for it (about 5GB per 1TB of storage). So I went in and disabled ZFS deduplication and immediately noticed a big improvement in speed, on both the FreeNAS server and VMWare host. On the FreeNAS bandwidth graphs I am hitting 600-900mbps consistently which was not the case before. Write latency in VMWare is now stays around 0-50ms. Previously, it would spike up to those speeds, then drop, repeatedly. On VMWare I was still seeing massive latency spiking up to 60,000+ ms.

However, what had me looking into this some more was the fact that backups went from taking 6 or so hours to taking 12+ hours to complete and very often the backup process would fail due to time-outs. After doing some research I came across an article that talked about dedup on FreeNAS and memory requirements and then it all made sense. The index table was getting filled with more files, causing slower write speeds due to lack of RAM.

Before with dedup enabled

After disabling dedup (the one drop off is the end of previous transfer):