Data Protection tools for PVs in SCALE?

[SlaterTh90]

Hi,
I have been playing with RC1(.1) for a bit now, and so far I really like it. I have not encountered any breaking bugs yet.

One thing I am not sure about though is how I am supposed to back up PVs. They are not available in the UI like a normal dataset (which makes sense since they are managed by zfs-localpv). Since PVs are supposed to contain application state, it would be nice to make use of the same data protection tools available for normal datasets (cloud sync, snapshots on shedule). Is there a solution for this I have not discovered yet, or maybe functionality planned in the future?

 

[truecharts]

As we are the only one using PVC/PV storage, it might be worthwhile asking our community directly next time ;-)

You're right, PVC's are managed by zfs-localpv which prohibits them from being mounted. This part is something we collaborated on with iX to check and both of us cannot see a reasonable way to mount them them would not cause additional issues for the user on other area's.

To be clear before hand: We're still working on an initial backup advisory, but are not done yet. So most of this is preliminary.
An official Apps backup guidance or restore procedure also does not exist.

Lets for clearity seperate two forms of Backups:
- File based backups: Copy files from one destination to a backup destination (rsync, cloudsync)
- ZFS aware backups: ZFS replication to an off-site ZFS system, keeping the zfs dataset structure. (zfs replication)

Replication of the SCALE Apps, requires backing up the `ix-applications` dataset (optionally excluding `ix-applications/docker`), however for easy restore it's absolutely essential to backup the complete ZFS dataset structure, so you would need to use ZFS aware backups. File based backups like cloudsync are currently not wise to use.

We're currently working with our team and partners to create a backup quick-start guide to, at least, prevents dataloss. But even for us restore testing might be out-of-scope for the moment (we know it can be restored, but it might require some more manual work than usual).

In the future we will also look on how to integrate external Kubernetes aware backup solutions like Velero. However currently that is not our focus.

 

[SlaterTh90]

As we are the only one using PVC/PV storage, it might be worthwhile asking our community directly next time ;-)

I already asked some questions in your discord, which is what ultimately led me here :)
Basically searching for the best way to backup databases from apps that use the postgres dependency and thus have no hostpath options. Probably local zfs-replication and then cloud backup of those datasets is what I will use for now.