remote IP mirroring? (delta/full/bandwidth cap)

[mediahound]

Although I have not read some of the more recent ZFS information it's possible you already have this, but the last time I was considering a migration to ZFS about two years back I hit a wall where I absolutely needed to back up to a remote IP address and didn't see a solution so didn't get much further.

My wishlist (though i'm not sure how to implement it obviously or if i'm even being fully clear/pick my brain if it doesn't make sense):
- Intelligent coordination between local and remote backups (ie - set my paranoia levels independantly, instant backups locally for anything I create, and maybe i'm content just uploading it all to a remote server once every two weeks)
- Limited remote backups (following bandwidth caps, ie limits per day, per hour, preferring mirroring at night, etc) This may even have to vary based on other use, ie - send backups towards the end of the month based on whats expected to be left, etc.
- Support "sneakernet" migration of files (for when bandwidth is limited, or bandwidth caps are in effect). This could include demand-based backups, ie it just fills a designated sneakernet drive until it's full then tells me to mail it to whomever or wherever I want to import that data to have a full mirror. This might include multiple stages of backups shared with multiple people.
- Delta/difference based backups and sync-ing of remote directories. I dont just mean changed files, I mean working with the above - I mail you a 4tb drive full of my last months... whatever data. A few files maybe arrive non-intact. (maybe for some reason I had to send a non-ZFS format not wanting to reformat the drive) So it's smart enough to just download updates for those files by recognizing they are different. Delta also means sending just the difference - if I sent you a 100gigabyte file, and 3 bytes are changed, it should only send 3 bytes corrective info.
- Priority marking (something important enough you want it pushed/FTP'ed right now to not risk loss) for certain files or data, vs lower priority stuff that can wait for a physical mailing that may take 2 months.

Basically adding some intelligence to work around real world scarcity (forcing full backups/mirrors over unlimited bandwidth to an unlimited server is easy, my wishlist would require something a little more integrated)

 

[cheezehead]

Have you looked at ZFS replications? http://doc.freenas.org/index.php/Replication_Tasks That may take care of your problem.

 

[mediahound]

This seems to cover at least some of it yes, I still dont fully understand snapshots but if that means it only sends the delta of changed files (like if 5 bytes changes in a 5gig file, it only sends 5 bytes of updated data) then yes that does exactly what I want to do. Or at least everything except scheduling around bandwidth caps. (scheduling being data-size specific, rather than time-specific)

 

[cyberjock]

This seems to cover at least some of it yes, I still dont fully understand snapshots but if that means it only sends the delta of changed files (like if 5 bytes changes in a 5gig file, it only sends 5 bytes of updated data) then yes that does exactly what I want to do. Or at least everything except scheduling around bandwidth caps. (scheduling being data-size specific, rather than time-specific)

If you go reading up on ZFS snapshots via Google, you will figure out how snapshots work and if they will work for you. :)