Delete cancelled rsync holding files?

[silmaril]

I am new to freenas 9 and was setting up an rsync pull task to migrate data from my old freenas 7 server. After starting the transfer I realized that I had left "Compress" and "Delay Updates" checked. With no clear way to stop the existing rsync task, I terminated it through the console command "killall rsync" and then changed the settings and restarted the rsync pull. It seems to be working okay, but I am concerned that I may have several gigabytes of temp data from the original rsync task in a holding directory somewhere. Does anyone know where this data would be stored and/or how to safely delete it?
Thank you

 

[fracai]

rsync is most likely resuming the transfer, unless you changed the destination directory. It's doubtful that you'll end up with anything incomplete; rsync transfers in place using hidden temp files (prefixed with a "." and a unique extension). To be sure though, you could stop the transfer again and delete the destination directory before starting the transfer again.

Also though, the rsync tasks are meant for keeping something in sync over time, rather than an initial transfer like you're doing. While I can't think of a concrete case where your use will cause a problem you'd be better off running the transfer on your own from the command line.

Are you planning to keep the old server running where you'll need to keep the two in sync? Or will the old server be decommissioned with new files only going to the new server?

 

[silmaril]

I didn't change the destination directory, but the initial task had "Delay Updates" checked. According to what I've read:
"when checked, the temporary file from each updated file is saved to a holding directory until the end of the transfer, when all transferred files are renamed into place"
So that's why I'm wondering if there is still a holding directory with these incomplete transfers that were never renamed into place.

I will probably decommission the old server eventually. Possibly not necessary to use rsync, but I heard it had decent error checking.

 

[fracai]

Ah, gotcha. I missed the implication of that checkbox. I'm not familiar with how FreeNAS maps the options to rsync flags, so I'm not sure where this temporary directory would be. Short of someone else arriving to document the behavior, I'd suggest canceling the transfer, deleting the second partial transfer, and then manually looking for the holding directory from the first transfer. Alternatively, you could start another "Delay Updates" transfer and use "ps -axww | grep rsync" to identify the arguments that are being used. This will likely indicate where the holding directory is and you can determine if anything still needs to be cleaned up.

 

[fracai]

Actually, if you're transferring from an old FreeNAS system, could you just use replication? Or was the old system not using ZFS?

 

[silmaril]

I could, but the rsync is already 600GB into the task. If anyone is familiar with where these files are stored (assuming they weren't cleaned) that would certainly be my preference.

 

[Robert Trevellyan]

I could, but the rsync is already 600GB into the task. If anyone is familiar with where these files are stored (assuming they weren't cleaned) that would certainly be my preference.

It's pretty well explained in the rsync documentation, assuming FreeNAS is using the --delay-updates flag, which seems likely.

 

[fracai]

From the rsync man page:

--delay-updates
This option puts the temporary file from each updated file into a holding directory until the end of the transfer, at which time all the files are renamed into place in rapid succession. This attempts to make the updating of the files a little more atomic. By default the files are placed into a directory named lq.~tmp~rq in each file's destination directory, but if you've specified the --partial-dir option, that directory will be used instead. See the comments in the --partial-dir section for a discussion of how this lq.~tmp~rq dir will be excluded from the transfer, and what you can do if you want rsync to cleanup old lq.~tmp~rq dirs that might be lying around. Conflicts with --inplace and --append.

So, unless FreeNAS is also using the "--partial-dir" argument, you should be able to look for "lq.~tmp~rq" directories. These also shouldn't interfere with the ongoing transfer.

 

[silmaril]

Okay, I will look for those, thank you both very much!