ZFS replication with "overlapping" snapshots

[aadje93]

I'm about to build my "Backup" machine which is almost a 1:1 copy (missing SLOG/L2ARC and bit slower CPU to save some power as lightning speed is less important)

My hardware:

2 freenas boxes built in Supermicro SC846's with expander.
Connection served by a LSI9211-8i in each server.

24x 4TB WD RED in both servers. (48 disk total).

Main server having 128G RAMM (overkill). Xeon E5-1620 v3 (3,5ghz quad) And 2x 120GB Samsung 840EVO L2ARC, 2x Intel DC 3500 100G in mirror as SLOG.

The backup server will have also 24x 4TB WD RED in striped mirrors (12x mirror) same as main server to keep iops up, and keep disk management "easy".

This server has 64G ramm, Xeon silver 4112 (2.6Ghz quad) and NO L2ARC or SLOG. I'm hoping the ZIL on the pool will be enough to keep up given the random IOPS the 12 mirrors will give, in case its not enough the main server will get a nice upgrade to its SLOG preparing for 10gbit networking with an intel NVME PCIE model, and the intel sata ssds will go from main to backup server to server as slog.

It will be the same pool layout; 24x WD RED 4TB in mirrored vdevs, same as the main host. But now i'm wondering.

In "old" versions of freenas ZFS replication would be unstable when there were overlapping snapshots;

Every hour kept for 7 days,
Every day kept for 30 days,
Every week kept for 12 months

If replicating the "hourly" one, ZFS replication would be unstable/fail.

Is this still the case?

Now for my true question;

I'm looking at a 5 minute replication interval for the whole recursive snapshot of the pool -being a new task - (as it shouldn't matter, if i save up for 1 hour of 5 minutes, the amount of data/hour is still the same, but every 5 minutes spreads out the load better? (less change of overloading the network for >10minutes). I'm using the hourly/7days one on the CIFS share for "shadowcopy", and then if it was >7days i can restore it daily from the 30days kept one. The case i lose something >30days before noticing it it wasn't that important to me :)

Secondly, is ZFS replication a sequential write to the destination? so importance of the SLOG is not noticeable compared to not having a dedicated SLOG (SSD).

Off course my whole pool is Sync=allways.

I'm looking forward to your help, the documents wasn't really clear on this matter.

 

[PhilipS]

I'm not sure on the stability of having multiple replication schedules for the same snapshot - I do have it setup this way on my home system and haven't run into any issues though.

You may want to check out the rollup script here: https://github.com/fracai/zfs-rollup

You can have one replication task for your desired interval, and then configure the rollup script to destroy the expired ones based on your desired settings - like 5 minutes for a day, daily for a week, weekly for a month, etc. This allows you to have a single snapshot for your "shadowcopy"/previous versions. Also look at the clearempty script which will remove the snapshots that haven't added any data to help keep things tidy.

This is how I have our business servers setup and it has been working well.

 

[aadje93]

Thanks for those rollup scripts, that will definitly help me!

Another question though,

what are some "best practices" for a VM storage-set/target. I'm running Xenserver (XCP-ng to be exact, the community clone). I've read NFS is bad, but ISCSI has the fragmentation problem.

I'm now thinking about ISCSI with a "Syn allways" on the zvol, but i'm not 100% sure whats better, a NFS store or ISCSI?

And how about the recordsizes? For torrents i know its best to give a temp dataset with 16K, and then "on completion transfer to another location" which is another dataset to force a re-write sequential to the long term storage, but how about VM NFS dataset/Zvol?

 

[Kevin Horton]

In "old" versions of freenas ZFS replication would be unstable when there were overlapping snapshots;

Every hour kept for 7 days,
Every day kept for 30 days,
Every week kept for 12 months

If replicating the "hourly" one, ZFS replication would be unstable/fail.

Is this still the case?

This setup of overlapping snapshot intervals and durations looks very similar to what I’m replicating to my backup server. I’ve never noted any issues, and I’ve been using this setup since FN 9.10 (I’m on 11.1-U5 now).

What were the symptoms of this unstable replication on older FreeNAS versions?