Core backup skipping media content folder

[iDenny]

Hello guys,

As the title says, what is the easiest way to backup FreeNAS core (jails, configs) skipping media folders (which in my case are /mnt/disk1/(movies|series))?
My FreeNAS is installed on a USB-drive inside a mini-server (HP microserver gen8), while jail data is stored on /mnt/disk1/jail-XXX.
I want to backup everything and use it as bare-metal backup copy. Which high-level tools do you use to achieve it? I mean dd still works, but not recommended...

Will it be possible to just wipe the disk and reinstall system from the backup image later (same hardware)?

Media content is not so important... However all configs, jails, crontab scripts and lot of other stuff are really important. I don't want to spend another 2 weeks configuring everything from the scratch.

 

[Nick2253]

If you're doing everything correctly (aka, through the GUI), then getting your config working is as simple as restoring a backup. You can make a backup of the config database from System -> General -> Save Config. Cyberjock has a script somewhere that does it automatically if you want.

To make a backup of your data, there are a couple different strategies. The easiest is probably to use ZFS send/receive using snapshots, so to get everything working again would be as simple as sending the data over. Perhaps because of my own lack of understanding with regard to ZFS, I don't think there's an easy way to just copy all the files off, because a simple file copy does not understand the details of the datasets you've created. In other words, you'll have to either notate those in some way and manually recreate them, or use something (like ZFS send/receive) which is aware of this metadata.

Perhaps a strategy would be to create a backup script that reads this data from the zvols/datasets, and stores it in a config file. Then, for restore, it reads out this data and recreates the appropriate zvols/datasets, and that would give you the framework to allow you to recopy the files (using rysnc).

 

[iDenny]

Sorry for the late reply mate.

1. Backup through GUI - piece of cake, dude, already done. However, this backup doesn't include all custom scripts and stuff.
2. Are we talking about jail data or "everything" data?

I mean, shouldn't it be easy to just create a copy of running image, save it somewhere (e.g. USB, HDD) and later apply this image on the "clean" hardware? I'm not an expert of the FreeBSD system environment, but it feels like this could be possible to achieve. The only problem is that in this case, I don't have a clue where to start. So far it feels like dd is the easiest way (?). Clone system USB -> Write it to new USB -> Manually restore/re-download stuff into /mnt/disk1/(movies|series)

 

[Nick2253]

What do you mean by "custom scripts and stuff"? If you're setting up stuff like cron jobs, they should be done from the GUI, and then they will be part of the config backup. If you have any scripts that you are referencing in these commands, those scripts should not be placed in the boot volumes. You should create a dataset on you array specifically for this data, and lock that dataset down appropriately.

A dd backup of the boot USB drive is possible, but the point is that in a properly configured system, a config backup will be at least as good as a full USB backup, and the config backup only requires a tiny amount of space relative to a full image backup.

 

[iDenny]

Thank you for your answer Nick2253,

>What do you mean by "custom scripts and stuff"?
- I use custom scripts to check the VPN connection, unrar the downloads and others. Everything is written in bash.

>If you're setting up stuff like cron jobs, they should be done from the GUI, and then they will be part of the config backup.
- Woah, I missed this part completely. Then, just put everything in the cron GUI, and that's it? <-- I guess this is solution to my problem. However, the obstacle here is that I need some of the scripts to run inside specific jails, not main OS.

>If you have any scripts that you are referencing in these commands, those scripts should not be placed in the boot volumes. You should create a dataset on you array specifically for this data, and lock that dataset down appropriately.
- All my custom things live inside the jail datasets, e.g. /mnt/media/jail-NAME/vpntest.sh.

>A dd backup of the boot USB drive is possible, but the point is that in a properly configured system, a config backup will be at least as good as a full USB backup, and the config backup only requires a tiny amount of space relative to a full image backup.
- The main cause of the headache is that I tweaked my FreeNAS at many different places and I don't remember every change I made. I guess the best way to solve this problem is to go back and search for all changes I've made and try to merge everything in to one big script and then put it in the cron GUI.

I will update the thread after a couple of weeks when I manage to put everything together.

 

[Nick2253]

I need some of the scripts to run inside specific jails, not main OS. ... All my custom things live inside the jail datasets

You've got me confused here. If your custom scripts live inside of jails, then the backup of the jail should capture those scripts. But yet we've been talking about copying out the boot drive...

The main cause of the headache is that I tweaked my FreeNAS at many different places and I don't remember every change I made.

A quick and dirty way to figure this out, if you can accept the down-time, is to backup your config (once you've got everything put in the GUI), install FreeNAS on a new boot device, and restore your config. Anything that's not working properly tells you what else you configured outside of the GUI :)

The most common thing I see people do incorrectly is cron jobs and tweakables. The GUI has places for all these settings. If you run across a setting that you need to set, but can't seem to do it through the GUI, post here, and we can either point you to it, or get it added as a feature request, so you won't have to worry about that setting for future backups.

 

[iDenny]

Ahaa... :D:D I decided to follow your advice like here bellow:

>A dd backup of the boot USB drive is possible, but the point is that in a properly configured system, a config backup will be at least as good as a full USB backup, and the config backup only requires a tiny amount of space relative to a full image backup.

>A quick and dirty way to figure this out, if you can accept the down-time, is to backup your config (once you've got everything put in the GUI), install FreeNAS on a new boot device, and restore your config. Anything that's not working properly tells you what else you configured outside of the GUI :)
- Yeeeep. I think that it is the best solution. My backup solution will get better for every time I need to reinstall whole system .

>The most common thing I see people do incorrectly is cron jobs and tweakables. The GUI has places for all these settings. If you run across a setting that you need to set, but can't seem to do it through the GUI, post here, and we can either point you to it, or get it added as a feature request, so you won't have to worry about that setting for future backups.
- That's awesome. Then, I have to say that I would really prefer "paste your code here" functionality in FreeNAS GUI, which is implemented in, like, dd wrt. It is really an easy way to run and manage your "check-VPN-if-not-connected-kill-Transmission" scripts.

 

[Nick2253]

I would really prefer "paste your code here" functionality in FreeNAS GUI

FreeNAS does have a shell-in-browser capability, if that's what you're looking for: https://doc.freenas.org/9.3/freenas_shell.html

Though your thread definitely has me thinking that it would be a great ability to add full bash scripts to the cron jobs, and then the middleware could translate that into a script file, and a cronjob pointing at that script.

 

[iDenny]

Update1: Funny thing. 2 days after the last post my USB-drive became degraded in FreeNAS. I was forced to rescue it using this guide: https://unix.stackexchange.com/questions/311086/cloning-a-bootable-usb-drive-to-another-stick-with-same-size.
For some reason, bmaptool wasn't able to produce metadata about my USB-drive. Anyway, I decided to burn newly created image to another USB (v.3 & larger capacity). When I booted up the system using my new boot stick I noticed that everything became much slower and kinda "glitchy". Then the whole system failed resulting in endless disk check at boot https://forums.freenas.org/index.php?threads/freenas-boot-import-problem.60663/.
However, I'm not sure that it have to do something with boot stick. I got the impression that 11.1 STABLE isn't as stable for my hardware as it should be.