Troubleshooting steps for slow Timemachine backups

[mhelms1961]

Hello,

I recently installed FreeNas 11.2 on a surplus Dell T3600 with 36gb ram, 4 core Xeon processor with a 635 watt power supply. The server OS is running off a Samsung EVO 128gb SSD connected to SATA port 0 on the Motherboard. Two of 3 8TB Western Digital Red NAS HDD are connected to the SATA ports on the mother board and one is connected to a PCIE card in one of the slots on the board. The volume is set up as a stripe configuration yielding total capacity of just under 24tb.
The FreeNas server is connected to my home LAN via Cat6 cable as well as My Macs. My router is a Netgear 7000 and I typically get around 120mbps transfer speeds when coping files between shares on my Macs and PCs. I can typically backup 16tb of data using Timemachine to a direct connect Mediasonic storage enclosure in about 24 hours for the 1st backup. This connection is via USB 3 on a late 2013 MacPro Cylinder. I have setup a Timemachine dataset on FreeNas for each Mac and gave the necessary storage quota for each Mac. I am currently testing a Timemachine backup of the MacPro on the FreeNas server and the speed is extremely slow compared to the direct connect scenario. I was wondering if there is a list of troubleshooting steps to follow so I can determine the weak link in my backup scenario. Thanks!

 

[Constantin]

Time Machine never broke any speed records in my experience. The most transfer I can recall was around 50MB/s. USB 3 should yield 150MB+ on a fast DAS array. Did you make any changes such as opt for a encrypted backup this time? Are you using AFP or SMB for time machine?

 

[mhelms1961]

Hi, I am using AFP, my understanding that is required for timemachine. I am trying to find out troubleshooting steps to see if I can sort out the problem with the speed. For instance do I test network speed 1st? Could it be a bus issue? Just not sure where to start. I did not use encryption.

 

[Constantin]

well, I'd mount the TM share and dump a large video file and see what happens Then, dump a folder with a lot of little files. Each time, see what the activity program shows re: network throughput. Then compare to time machine once it's started running. But every time, use the share that TM is sitting on.

 

[Dirk Wagner]

As I wrote a week ago in the German language board, I am facing the same problems.
I am running OS X 10.13.6 on my new iMac...
After upgrading to 11.2, the first Backup of that iMac on to the same sparsebundle I used for the old one, was incredible slow.

Than i deleted the whole sparsebundle - and tried an complete new backup.
This is now running since Friday, allready completed almost 3 TB of the 4 TB needed to be backed up.

Network speed ist mostly around 10MBps...

Using 9.1, backing up the iMac did not take this time...

Ciao

dirk

 

[mhelms1961]

well, I'd mount the TM share and dump a large video file and see what happens Then, dump a folder with a lot of little files. Each time, see what the activity program shows re: network throughput. Then compare to time machine once it's started running. But every time, use the share that TM is sitting on.

Hi, I copied a 5.64gb video file to the folder containing the timemachine sparse bundle on the NAS server and it took 40 seconds. I copied a folder full of small files as suggested and that went very fast as well, less than 5 seconds for about 15 files averaging 125mb each. In fact the transfer speed from the DAS on the MacPro to the FreeNAS server was FASTER than between two DAS connected via usb 3.0 to the same machine. So there must be something quirky about the timemachine setup on FreeNAS. I stopped the backup as only 564gb out of 13tb had backed up for the past 24 hours. I also had an issue with Chrome on the MacPro as I was seeing the spinning beach ball and could not force quit the browser so had to reboot the MacPro. So I can conclude no issue with LAN speed or any write issues with the datapool. I can rule out any issues with the bus on either the motherboard or the PCIe SATA card.

 

[anodos]

Do the servers have slog devices? In my testing with OSX 10.14 and Time Machine over SMB on FreeNAS, the MacOS client at certain points was sending about 30 fsync requests per second (which translate into zil writes).

 

[mhelms1961]

Timemachine has been running about 13 hours now and has completed 950gb out of 13tb of backup. much slower transfer rate than what was observed when I simply copied a 5.64gb video file (<40 seconds) or a folder containing 15 150mb files +/- (<5 seconds). I copied the test files to the same share holding the Timemachine sparse bundle. I had to create an AFP share for Timemachine as I am told that Timemachine does not support SMB yet. I must be missing some sort of subtle tweak. I am new to FreeNas as I had been using shares between PCs and Macs as well as DAS and a HD connected to the Netfgear 7000 router. I looked at the FreeNAS 11.2 dashboard this morning and under the "bandwidth" section it indicated " ->20.601MBps" & "<-0.4642MBps" on the primary NIC. The PC is an old Dell T3600 CAD workstation built in late 2013 with 32gb ram and a 4 core Xeon processor with 635 watt power supply.......... Any tweak ideas will be appreciated unless I simply have to accept that the 1st backup will take days on FreeNAS versus hours on DAS................

 

[anodos]

Timemachine has been running about 13 hours now and has completed 950gb out of 13tb of backup. much slower transfer rate than what was observed when I simply copied a 5.64gb video file (<40 seconds) or a folder containing 15 150mb files +/- (<5 seconds). I copied the test files to the same share holding the Timemachine sparse bundle. I had to create an AFP share for Timemachine as I am told that Timemachine does not support SMB yet. I must be missing some sort of subtle tweak. I am new to FreeNas as I had been using shares between PCs and Macs as well as DAS and a HD connected to the Netfgear 7000 router. I looked at the FreeNAS 11.2 dashboard this morning and under the "bandwidth" section it indicated " ->20.601MBps" & "<-0.4642MBps" on the primary NIC. The PC is an old Dell T3600 CAD workstation built in late 2013 with 32gb ram and a 4 core Xeon processor with 635 watt power supply.......... Any tweak ideas will be appreciated unless I simply have to accept that the 1st backup will take days on FreeNAS versus hours on DAS................

I just added initial support for time machine over SMB to freenas/master. It will probably be added to a 11.2 U-release. Manipulating sparsebundles over the network is always going to be nasty because the default band size is 8MB (so your 13tb backup is actually comprised of 8MB files that have to be written and updated). You can type "zilstat 1" to quantify zil activity. As I mentioned the protocol over SMB makes tons of requests to flush to stable storage. This has an impact on performance, that might be partially mitigated by adding an appropriate SLOG, but that said, the performance will never be on par with direct attached storage.

 

[mhelms1961]

I just added initial support for time machine over SMB to freenas/master. It will probably be added to a 11.2 U-release. Manipulating sparsebundles over the network is always going to be nasty because the default band size is 8MB (so your 13tb backup is actually comprised of 8MB files that have to be written and updated). You can type "zilstat 1" to quantify zil activity. As I mentioned the protocol over SMB makes tons of requests to flush to stable storage. This has an impact on performance, that might be partially mitigated by adding an appropriate SLOG, but that said, the performance will never be on par with direct attached storage.

Hi anodos, thanks for your reply. Dumb question as I am a total newbie to FreeNAS and coming up to speed!! Can I put the SLOG on the same internal SSD where the FreeNAS OS resides? Or should I put the FreeNAS OS on an attached USB thumbdrive, boot from that, and reserve the internal Samsung EVO 850 128gb SSD attachded to SATA 0 on the motherboard for the SLOG alone? Below is my understanding of how adding an appropriate SLOG may help.

If I understand correctly the By default, the short-term ZIL storage exists on the same hard disks as the long-term pool storage (In my situation a pool comprised of 3x8TB WD Red NAS Drives in a stripe configuration) at the expense of all data being written to disk twice: once to the short-term ZIL and again across the long-term pool. Because each disk can only perform one operation at a time, the performance penalty of this duplicated effort can be alleviated by sending the ZIL writes to a Separate ZFS Intent Log or “SLOG”, or simply “log”. (No clue how to do this task btw)While using a spinning hard disk as SLOG will yield performance benefits by reducing the duplicate writes to the same disks, it is a poor use of a hard drive given the small size but high frequency of the incoming data.

The optimal SLOG device is a small, flash-based device such an SSD or NVMe card, thanks to their inherent high-performance, low latency and of course persistence in case of power loss. You can mirror your SLOG devices as an additional precaution and will be surprised what speed improvements can be gained from only a few gigabytes of separate log storage. Your storage pool will have the write performance of an all-flash array with the capacity of a traditional spinning disk array.

 

[Constantin]

Though I'm sure it has been done, I'd keep them completely separate.

In 11.2, the boot drive doesn't show as part of the co-hort that you can add to a pool.

 

[mhelms1961]

Though I'm sure it has been done, I'd keep them completely separate.

In 11.2, the boot drive doesn't show as part of the co-hort that you can add to a pool.

Hi Constantin, Thanks for the information. I do have some 128gb usb flashdrives laying around. Rather than messing around with my current setup can I plug one of those into an available usb port and use that for the SLOG?

 

[Constantin]

I suppose so, but I wonder how quick they are vs. a sata drive.

Plus, I’d feel more than queasy entrusting SLOG data to a USB drive, regardless of manufacturer unless you mirrored it and even then... just no. Ssds are cheap insurance.

 

[mhelms1961]

OK, I have an additional internal SATA port and power available for another Samsung EVO 850 that I can dedicate to the SLOG. Not sure if it is overkill but BestBuy has a 500gb on sale for $74.99 or I can get a 250gb for $54.99

In any event, can you share how I actually "point" the SLOG from the default location to that new drive? Again I am a complete Newbie to this but if I come up with a Timemachine solution that provides reasonable transfer rates, I'm sure it will be helpful for those wanting to transition their Timemachine backups to FreeNAS. I'll be sure to share the results with the group if I can figure this out!!

 

[Constantin]

Select Storage, then your pool, then use the three dot thingie to the right to give you the option to extend or edit the pool (sorry at work, not at home)

Then select the spare SSD and then the button at the button that allows you to add cache. See here.

 

[mhelms1961]

Select Storage, then your pool, then use the three dot thingie to the left to give you the option to extend or edit the pool (sorry at work, not at home)

Then select the spare SSD and then the button at the button that allows you to add cache. See here.

Thanks, I will try this out when I get home! I'll let the group know how it goes.................

 

[mhelms1961]

Well the timemachine backup is still slow even after adding a 500GB Samsung EVO 860 cash to the pool. Just for grins I used a utility on my Mac called black magic speed test which tests disk read writes on DAS and well as Network Shares. black magic indicated 105MBs write speeds and 110MBs read speeds on the FreeNAS server compared to 250/300MBs R/W for my DAS. I would assume then that the best I could expect would be at least 3x longer to backup to NAS versus DAS. Any other tweaks to try to speed up the process would be appreciated as cache had no impact.

 

[Constantin]

I don't know what NIC you're using but if it's 1GB/s ethernet, then 105MB/s is close to the limit of 125MB/s that 1GB/s network can theoretically put through.

I agree that the slowness of your TM is baffling, however. The only remedy I can suggest at this point is the following:

1. Make a encrypted TM sparsebundle on a local disc (your DAS, for example)
2. Use Carbon Copy Cloner to transfer the sparsebundle to the NAS
3. Then point TM via its preferences to the NAS TM sparsebundle. Enter applicable passwords, etc. and you should be good to go.

Incremental backups may not bother you as much as the initial build.

 

[mhelms1961]

I don't know what NIC you're using but if it's 1GB/s ethernet, then 105MB/s is close to the limit of 125MB/s that 1GB/s network can theoretically put through.

I agree that the slowness of your TM is baffling, however. The only remedy I can suggest at this point is the following:

1. Make a encrypted TM sparsebundle on a local disc (your DAS, for example)
2. Use Carbon Copy Cloner to transfer the sparsebundle to the NAS
3. Then point TM via its preferences to the NAS TM sparsebundle. Enter applicable passwords, etc. and you should be good to go.

Incremental backups may not bother you as much as the initial build.

Thanks for the information, the NIC is an integrated Intel® 82579 Gigabit Ethernet Controller on the T3610. I believe I will go ahead and follow your steps above as I was contemplating that as a next step. Incrementals won't be so much of a pain as you say. It is just that the initial build is a real
P.I.T.A. !

 

[Constantin]

Sorry I didn't have a silver bullet. I keep very little on my laptop. Thus, backups don't take long either.

One more thing: if you have scratch space with WIP that you don't need to have backed up (let's say you drop it on the server nightly), I'd exclude it from backups via TM (see preferences). That keeps your backup sets small.