SOLVED For those who love ZFS ...

[NASbox]

I see this is marked "SOLVED", but there really isn't any problem, so I hope it's OK to post here:

pfSense 2.4 ships with ZFS now! Is anyone else running this?
I just installed it on my J1800 board with dual Intel 24GB SLC mSata drives... been going a few days, seems to work great!

I was thinking about switching to ZFS, I'd really love to be able to roll back an update if I don't like it. What is the resource usage like?

Would it make sense to run ZFS on this setup?

CPU Type Intel(R) Celeron(R) CPU J1900 @ 1.99GHz
Current: 1826 MHz, Max: 1993 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: No
Memory usage 26% of 3960 MiB
SWAP usage
0% of 8191 MiB
Disk usage:
/
85% of 101GiB - ufs
/var/run
6% of 3.4MiB - ufs in RAM
--

All that said, I would not worry too much about using BTRFS. Just be aware of it's worts, moles and pimples. (ZFS has it's own share of such, it's just 8 or more years older.)

In a Desktop setup, what's the overhead on ZFS like compared to EXT4? I've got a linux Mint system that I use for much of my day-to-day that I need to rebuild-I have a 128GB SSD for boot/system and 2 1TB that are currently using the mdam RAID - I'd love to ZFS those drives to RAIDZ1.

Do you backup to FreeNAS, or removable USBs?

I'd like to have a backup setup that snaps the system (and all the dot files with program configurations) - but omits most of the user data, and then a separate backup for user data. The root of the home directory is such a mish-mash that I haven't figured out how to separate it.

 

[Arwen]

...
In a Desktop setup, what's the overhead on ZFS like compared to EXT4? I've got a linux Mint system that I use for much of my day-to-day that I need to rebuild-I have a 128GB SSD for boot/system and 2 1TB that are currently using the mdam RAID - I'd love to ZFS those drives to RAIDZ1.
...

ZFS definitely is slower on a less powered computers like my laptop with a dual core 1.5Ghz Celeron and only 2GB of memory. (My SSD is fast, so it's less noticeable than it could be.)

That said, I decided that data security and reliability are more important than the fastest. For example, I would not switch my Linux hosts to BTRFS even if they doubled the speed. And EXT2/3/4 have little to none data integrity checks.

...
Do you backup to FreeNAS, or removable USBs?
...

Both. Then backup my FreeNAS to an external disk.

...
I'd like to have a backup setup that snaps the system (and all the dot files with program configurations) - but omits most of the user data, and then a separate backup for user data. The root of the home directory is such a mish-mash that I haven't figured out how to separate it.

Designing a backup solution that meets needs and budget can be hard. I do that for a living and still have to make compromises.

 

[NASbox]

ZFS definantly is slower on a less powered computers like my laptop with a dual core 1.5Ghz Celeron and only 2GB of memory. (My SSD is fast, so it's less noticable than it could be.)

Well, I guess that answers the questions about resource requirements-that's certainly not a fast machine, and 2GB is very tight for memory as well.

I totally get where you are coming from regarding data integrity/security. Do you snapshot to provide protection against ransomware and other accidents?

Just curious what you run on that laptop - distro/desktop/and applications... just browser, or libre office or what.... and how you find the performance. I would expect it would be super slow, but if it's not then I need to update my thinking.

If you are able to run ZFS on that machine, I guess I shouldn't have a problem on an i5-6500/16MB.

Designing a backup solution that mets needs and budget can be hard. I do that for a living and still have to make compromises.

I think I need a new workflow... when I'm doing "stuff" like experimenting with scripting, or making quick scripts by doing command | command >file and editing the output, I've been just cd ~ for a place to work, but that means I get left with ~ full of crap to pollute a backup. I'm pretty new to the nix world, so I'm wondering if that's what most people do, or if there is a common "best practice" for that type of thing.

 

[Arwen]

...
Do you snapshot to provide protection against ransomware and other accidents?
...

Yes. I run hourly snapshots, for 1 day's worth, (24 total). Also run hourly snapshots to be kept a week, (only 7). This overwrites the previous day snapshot. Complicated, but does what I want.

...
Just curious what you run on that laptop - distro/desktop/and applications... just browser, or libre office or what.... and how you find the performance.
...

My home systems run Gentoo, (except of course, my FreeNAS), with Xfce4, (since Gnome 3 is basiclly useless).

I have all the stock desktop tools, browsers, LibreOffice, GIMP. And they work very well. Even when I run more than one, (though I have 4GB of swap configured on my SSD).

...
I think I need a new workflow... when I'm doing "stuff" like experimenting with scripting, or making quick scripts by doing command | command >file and editing the output, I've been just cd ~ for a place to work, but that means I get left with ~ full of crap to pollute a backup. I'm pretty new to the nix world, so I'm wondering if that's what most people do, or if there is a common "best practice" for that type of thing.

I put real projects in there own directory under ~/src/ and new, to be used regularly, scripts in ~/bin/ and help files in ~/help/.

One of the scripts I wrote creates a backup file with the date at the end, puts it in a backup subdir .BK and optionally edits the file. This allows me to keep a history of the changes to my scripts or help files without something like RCS / GIT / SVN. A simple diff between 2 versions lets me know what was changed.

 

[NASbox]

Yes. I run hourly snapshots, for 1 day's worth, (24 total). Also run hourly snapshots to be kept a week, (only 7). This overwrites the previous day snapshot. Complicated, but does what I want.

I assume you wrote a script to do that? Any chance you would be willing to share?

My home systems run Gentoo, (except of course, my FreeNAS), with Xfce4, (since Gnome 3 is basiclly useless).

I have all the stock desktop tools, browsers, LibreOffice, GIMP. And they work very well. Even when I run more than one, (though I have 4GB of swap configured on my SSD).

Sounds like you have tuned your hardware very well and are relying on fast swap to an SSD to make up for lack of other resources. I Googled Gentoo... from what I can see you need to be a guru to manage that distro in that it has to almost completely compiled from source.

I put real projects in there own directory under ~/src/ and new, to be used regularly, scripts in ~/bin/ and help files in ~/help/.

One of the scripts I wrote creates a backup file with the date at the end, puts it in a backup subdir .BK and optionally edits the file. This allows me to keep a history of the changes to my scripts or help files without something like RCS / GIT / SVN. A simple diff between 2 versions lets me know what was changed.

Do you end up with work files - maybe proof of concept, downloads etc.? I find that I collect a lot of that type of stuff, that I don't want to get rid of immediately, it ends up collecting.

 

[Bidule0hm]

No need for a script, you just setup multiple snapshots tasks in the GUI. Personally I've done this:

- every 10 minutes, keep for 10 hours
- every 1 hour, keep for 50 hours
- every 1 day, keep for 8 days
- every 1 week, keep for 1 month
- every 4 weeks, keep for 1 year

 

[NASbox]

No need for a script, you just setup multiple snapshots tasks in the GUI. Personally I've done this:

- every 10 minutes, keep for 10 hours
- every 1 hour, keep for 50 hours
- every 1 day, keep for 8 days
- every 1 week, keep for 1 month
- every 4 weeks, keep for 1 year

I'm sure you aren't going to have to worry too much about ransomware (or stupid users as long as they aren't root) much either! For an active dataset that 's a great idea you are not going to have to worry much about losing files to anything but catastrophic hardware failure or operating system issue. I'm wondering what the overhead of this type of thing is, so a couple of follow up questions if I may.

Approximate data set size?
How much the data turns? What sort of activity.... just a few text files (like software development), or big media files (video/audio that won't compress)
How much overhead for snapshots?

The reason I was looking for a script is that I was going to use the ideas for (1) my workstation (thinking about converting to zfs) since the discussion relates to laptop snapshots, and (2) I have quite a few "very low turnover" pools that I use as archive storage. A much better snapshot strategy for these is to do a diff to make sure I know the changes are what I expect them to be, and make a manual snapshot. I've noticed that the downside to too many snapshots is that zfs list becomes quite laggy, and for these types of workflows it's much better to have snapshots that are time points like a git commit rather than just regular saves. Since I do a lot of work at the command line, a script is very convenient, and having something that would create a snapshot and delete that last one from the stack (keeping 3 or 4 would be all I need) would be very helpful.

Other than zfs diff POOL@snapshot POOL and

Code:

wc -l

the output is there another way to get a count of files not protected by the last snapshot that would be less overhead?

 

[Bidule0hm]

Yep, I already used the snapshots twice (in 3 or 4 years) to recover some deleted file for a family's member... That's exactly what I have snapshots for (and ransomware too of course) :)

I have 4 datasets: 2 with video files (0.5-5 GB, 3.7 TB total) and 2 with all the documents a user usually have (text, doc, web sources, PDF, zip, 3D files, CAD files, images (1000s of them), ... 1 TB total).

The 2 video datasets don't have the 10 min and 1 h snapshots tasks because they don't change as much as the others. The other 2 see a lot of changes (but relatively small changes, 1 GB/day max).

If I add the numbers of the 'used' column of the snapshots in the GUI (keep in mind it not that's simple to see space used by snapshots but it's the only simple mean we have) I get about 150 GB and I have 450 snapshots (and the number stay nearly the same as older snapshots are deleted) for nearly 5 TB of data, not bad I think.

Ok, for your use case I can understand a script is better. What I would do is use a script to make the snapshots and have an automatic script (CRON...) to delete the oldest snapshot when their number is too big.

I don't know the answer to your last question, sorry.

 

[Arwen]

I assume you wrote a script to do that? Any chance you would be willing to share?
...

Something like below as a root cron. Thinking about it later, I realized that I did not have check if the dataset existed before running the destroy command. I already send the error output to NULL, so it would not matter. But, I have not gotten around to changing it, (it works).

Code:

#!/bin/bash
#
# 0 * * * * /root/bin/snap_home >/dev/null 2>&1
#
export MY_HOME_DATASET=`df -h /home | tail -1 | cut -f1 -d' '`
export MY_DAY_OF_WEEK="dow_`date +%w`"
export MY_HOUR_OF_DAY="hod_`date +%H`"
#
/sbin/zfs list ${MY_HOME_DATASET}@${MY_DAY_OF_WEEK} >/dev/null 2>&1
if [ ${?} -eq 0 ]; then
	/sbin/zfs destroy ${MY_HOME_DATASET}@${MY_DAY_OF_WEEK} >/dev/null 2>&1
fi
/sbin/zfs snap ${MY_HOME_DATASET}@${MY_DAY_OF_WEEK} >/dev/null 2>&1
#
/sbin/zfs list ${MY_HOME_DATASET}@${MY_HOUR_OF_DAY} >/dev/null 2>&1
if [ ${?} -eq 0 ]; then
	/sbin/zfs destroy ${MY_HOME_DATASET}@${MY_HOUR_OF_DAY} >/dev/null 2>&1
fi
/sbin/zfs snap ${MY_HOME_DATASET}@${MY_HOUR_OF_DAY} >/dev/null 2>&1
#
exit 0

...
Sounds like you have tuned your hardware very well and are relying on fast swap to an SSD to make up for lack of other resources. I Googled Gentoo... from what I can see you need to be a guru to manage that distro in that it has to almost completely compiled from source.
...

Yes, Gentoo is not for everyone.

...
Do you end up with work files - maybe proof of concept, downloads etc.? I find that I collect a lot of that type of stuff, that I don't want to get rid of immediately, it ends up collecting.

I keep a dumping ground in another file system. Then organize it somewhat;

DUMPING/hardware
DUMPING/software

Then documentation and or source code below as appropriate. For example, I try and keep all the firmware for my devices just in case I need or want to revert.

Since I have multiple computers, the DUMPING ground is actually the hostname as in;

/`uname -n`/

I make my ID the owner & group, but you could just as easily use 1777.

 

[NASbox]

Thanks @Arwen that helps a lot.

Something like below as a root cron. Thinking about it later, I realized that I did not have check if the dataset existed before running the destroy command. I already send the error output to NULL, so it would not mater. But, I have not gotten around to changing it, (it works).

Even though the error is suppressed, you likely save a mS or two by not calling zfs destroy unnecessarily.

I keep a dumping ground in another file system. Then organize it somewhat;

DUMPING/hardware
DUMPING/software

Great idea. Thanks for taking the time to share.

 

[averyfreeman]

No, you can not boot from ZFS snapshots. You CAN boot from a clone of a snapshot. (Okay, I am being nit picky :).

Yes, I have been doing ABEs, (Alternate Boot Environments), with OpenZFS for 3.5 years now. Before that I used BTRFS for 3 years, (with ABEs). And for several years before that, 2 different root FS partitions, alternating between them so I always had a backout plan and bootable system.

Here is roughly what I do for ZFS ABEs;

Code:

> df -h /
Filesystem                    Size  Used Avail Use% Mounted on
rpool/root/20171224            15G  5.9G  8.3G  42% /

> zfs list -t all -r rpool/root
NAME                    USED  AVAIL  REFER  MOUNTPOINT
rpool/root             10.2G  8.30G    96K  legacy
rpool/root/20170929    5.01M  8.30G  5.97G  legacy
rpool/root/20171028     892K  8.30G  5.52G  legacy
rpool/root/20171123     600K  8.30G  5.70G  legacy
rpool/root/20171224    10.1G  8.30G  5.88G  legacy
rpool/root/20171224@1  1.57G      -  5.97G  -
rpool/root/20171224@2   421M      -  5.52G  -
rpool/root/20171224@3   686M      -  5.70G  -

> head -19 /etc/grub.d/05_custom
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
####################
#
# Kernel 4.9.49-gentoo-r1 local version 2
# New grub entry for ZFS update via InitRD
#
menuentry 'Goblin linux-4.9.49-gentoo-r1.2a ZFS 20171224 resume' {
     load_video
     insmod gzio
     insmod part_gpt
     insmod zfs
     linux /linux-4.9.49-gentoo-r1.2 rootfstype=zfs root=ZFS=rpool/root/20171224 ro resume=/dev/sda2
     echo   'Loading initial ramdisk ...'
     initrd /linux-4.9.49-gentoo-r1.2a.img
}

Yes, I just updated yesterday.

I do have a script that lists, verifies, deletes and creates ABEs. It's mostly automated for the delete and create, except for the Grub2 stuff. I now know a way to automate that too. But, I am in no hurry. It's less than 1 minute to create a new ABE, even manually adding the new ABE's Grub2 entry.

My system does not use systemd, and does not require it for either the ABEs or the scripted create / delete part.

Hey, I was just thinking back to this conversation and I was wondering if you have any resources for me if I wanted to build a Gentoo-based systemd-free desktop for my Thinkpad on zfs root with ABEs as you call them :)

Thanks! Happy New year.

Edit: oh crap I guess there's a bunch of new posts I neglected to read :/

 

[Arwen]

@averyfreeman, If I remember correctly, I followed the normal EXT4 install procedure, (for systemd-free), to one partition. Installed the ZFS packages and made a root pool on the other mirror and copied my EXT4 to this root pool. Then booted to it. Whence I was happy, I simply mirrored over the old EXT4.

One word of caution. Always have a ZFS enabled boot media. Further, don't enable all the extra OpenZFS feature flags on your root pool, (data pools can be current). Stick with the feature flags your boot media uses.

I have misc. documentation that I wrote to perform a Gentoo update.

As for the ABEs, it's not hard, but does require thought and ZFS skill.
Like Grub2, I want to be able to boot from different ABEs, so my kernel lines in the Grub2 configuration file look something like this;

Code:

####################
#
# New grub entry for portage update
#
menuentry 'MyHost linux-4.14.83-gentoo.1    ZFS 20181222 resume' {
     linux  /linux-4.14.83-gentoo.1 rootfstype=zfs root=ZFS=rpool/root/20181222 ro resume=/dev/sdb2
     initrd /linux-4.14.83-gentoo.1.img
}
####################
#
# New grub entry for portage update
#
menuentry 'MyHost linux-4.14.83-gentoo.1    ZFS 20181209 resume' {
     linux  /linux-4.14.83-gentoo.1 rootfstype=zfs root=ZFS=rpool/root/20181209 ro resume=/dev/sdb2
     initrd /linux-4.14.83-gentoo.1.img
}

Stupid Grub2 wants to automatically generate the entries. It can't without a TON of help, so I don't bother. Takes me less than a minute to make my own, correct and functioning Grub2 entries.

Note that I keep other things on my root pool, separate from the OS. Like;

rpool/home
rpool/src
rpool/src/4.14.83-gentoo.1

And yes, resume from hibernation works fine with ZFS on Linux. Both desktop and laptop. Resume from suspend sometimes works on my laptop, but I think my problems are more with keeping the support software up to date than ZFS. Hibernate is less sensitive to keeping the support packages, (pm-utils), up to date.

I don't have a step by step guide to both install ZFS on Gentoo Linux, and use it for alternate boot environments, (aka ABEs). I probably should, but I've only really done it a hand full of times, years ago.

 

[averyfreeman]

1. Of course BTRFS is Linux only. It's very unlikely to be ported to any other OS, Unix like or other.

Thanks for these points, they are very useful

I only have enough knowledge to take issue with this one thing:

https://github.com/maharmstone/btrfs

Surprisingly, it actually works pretty well! Haven't taken a deep-dive, but it'll format / read / write a btrfs drive (personal experience).

 

[Ericloewe]

I remember that project, how're they doing? Since it's a reimplementation, I have some expectations of stability down the road, unlike the main project...

 

[averyfreeman]

I remember that project, how're they doing? Since it's a reimplementation, I have some expectations of stability down the road, unlike the main project...

I don't know anything about it personally other than it worked for me when I tried it, but maybe you could shoot the guy an email? or put a comment in an issue on github? His email from the github site is listed:

You are encouraged to play about with the source code as you will, and I'd appreciate a note (mark@harmstone.com) if you come up with anything nifty. On top of that, I'm open to relicensing the code if you've a burning desire to use it on a GPL or commercial project, or what have you - drop me a line and we'll talk.

 

[Arwen]

@averyfreeman, Actually, when I wrote that prior response on BTRFS, I forgot to mention ReactOS, a MS-Windows open source workalike. They may implement BTRFS on ReactOS.

Of course, OpenZFS now has an official MS-Windows implementation;

http://www.open-zfs.org/

It's early days, and may have less interest than BTRFS on ReactOS. But, it's interesting to see it.

 

[averyfreeman]

Actually, when I wrote that prior response on BTRFS, I forgot to mention ReactOS, a MS-Windows open source workalike. They may implement BTRFS on ReactOS.

Of course, OpenZFS now has an official MS-Windows implementation;

http://www.open-zfs.org/

It's early days, and may have less interest than BTRFS on ReactOS. But, it's interesting to see it.

No shit?!! ZFS on Windows? That's AMAZING! I'm so excited! I'm going to have to try that out.

I tried ReactOS, couldn't really get it to do much. Doesn't seem ready for prime-time. Nice idea, though, I hope it evolves into a real contender.

 

[NASbox]

No crap?!! ZFS on Windows? That's AMAZING! I'm so excited! I'm going to have to try that out.

I had a look at the website -- it appears to be somewhat "experimental" at this point -- or am I interpreting things incorrectly?

It will be great when it gets to the point that it is stable/reliable as it currently is on FreeBSD/Linux. Windows badly needs an easy way to snapshot a system so that botched updates, ransomware, malware can all be "rolled back" out of existance.

I guess they will never be directly compatible because of the very different permission/ACL systems on Unix/Windows -- or do I have that wrong as well?

 

[Arwen]

I had a look at the website -- it appears to be somewhat "experimental" at this point -- or am I interpreting things incorrectly?
...

No, you are right. Still experimental.

...
It will be great when it gets to the point that it is stable/reliable as it currently is on FreeBSD/Linux. Windows badly needs an easy way to snapshot a system so that botched updates, ransomware, malware can all be "rolled back" out of existance.
...

For the time being, it's data file systems only. I doubt booting is even on the radar. But, home directories and programs, maybe.

...
I guess they will never be directly compatible because of the very different permission/ACL systems on Unix/Windows -- or do I have that wrong as well?

You have it wrong. After all, FreeNAS supports MS-Windows clients. It's a bit tricky. I've followed the OpenZFS mailing lists for a year or so, including the ZFS on Windows. They have had to re-think how to store MS-Windows permissions in a more standard way. This will allow a ZFS pool to be used by *nix & Samba with the same results.

As I said, early days. But interesting never the less.

 

[averyfreeman]

I guess they will never be directly compatible because of the very different permission/ACL systems on Unix/Windows -- or do I have that wrong as well?

I seem to remember they're compatible, Solaris was great AFA having Windows domain integration support. I don't know about its ACLs in detail, though, so I found this for you:

https://www.ixsystems.com/community/threads/methods-for-fine-tuning-samba-permissions.50739/

And AFA the ZFS being stable/compatible with Windows, it's doubtful all the features work, but it's obviously a step in the right direction. I'm a snob after using OmniOS but I'm speaking from experience after having used ZFS on FreeBSD (vanilla), ZoL on several Linux distros, etc. and this is what I've learned:

The further you get away from the original iteration of OpenZFS on OpenSolaris the more features are missing or don't work right - IllumOS being best feature compatibility, FreeBSD being a somewhat distant second, and ZoL implementations being a total crapshoot - features like sharesmb, sharenfs, don't even bother.

That being said, stability of the FS itself, however, missing/corrupt files, bit flips, etc. haven't been an issue that I've ever noticed in regular use, so maybe you could move the drive to a more ZFS-compatible OS to implement missing features and then move it back to Windows and see if they still work ;)

I actually do that with OmniOS, I use it to create all my ZFS pools and then export->import because it formats them differently so they have the broadest compatibility if I start there.