SOLVED For those who love ZFS ...

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
# zpool status
pool: pfSenseMirror
state: ONLINE
scan: scrub in progress since Sun Dec 24 21:54:29 2017
252M scanned out of 620M at 84.1M/s, 0h0m to go
0 repaired, 40.70% done
config:

NAME STATE READ WRITE CKSUM
pfSenseMirror ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p4 ONLINE 0 0 0
ada1p4 ONLINE 0 0 0

errors: No known data errors

----

pfSense 2.4 ships with ZFS now! Is anyone else running this?

I just installed it on my J1800 board with dual Intel 24GB SLC mSata drives... been going a few days, seems to work great!

I've also been running Antergos on ZFS for a few months, it's the first Linux distro I know of that ships with ZFS in the installer to make installing on ZFS easy.

Lastly, I set up a KVM server using OpenSUSE Tumbleweed for testing other processor architectures like ARM64, running LXC containers, etc. and I have an 8TB mirror running ZFS. The packages for ZFS in the Munix repo work great, build DKMS using dracut automatically, all seems to work very well (been through quite a few upgrades no problem since it's a rolling distro...) should also mention that Samba config is dead easy on this distro, even for joining a domain.

Just thought I'd mention these few things since I know people on this board love ZFS.

What kind of stuff are you doing with ZFS these days?
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,177

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,470
pfSense 2.4 ships with ZFS now! Is anyone else running this?
Yes, I am. I was running 2.3, but when 2.4 released, I reinstalled using ZFS. I'm also using ZFS on my Proxmox hosts and my Nethserver box.
 

SlackerDude

Explorer
Joined
Feb 1, 2014
Messages
76
Is it really? I thought it was from Electric Sheep Fencing LLC ...

I believe Electric Sheep Fencing LLC was/is the "man behind the curtain", and now Netgate appears to have rebranded pfSense.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
I use ZFS with all my home Linux computers, desktop, laptop and miniture media server. All have Mirrored root pools, with the remaining space in a seperate pool without redundancy. For me, it was Gentoo Linux, which has had great ZFS support, (even for root pools!), since 2013.

Even use alternate boot evironments to make updates easier in case I have to back an update out.

PS - My NAS is of course FreeNAS.
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
I use ZFS with all my home Linux computers, desktop, laptop and miniture media server. All have Mirrored root pools, with the remaining space in a seperate pool without redundancy. For me, it was Gentoo Linux, which has had great ZFS support, (even for root pools!), since 2013.

Even use alternate boot evironments to make updates easier in case I have to back an update out.

PS - My NAS is of course FreeNAS.
I've been meaning to try Gentoo - having spent a good amount of time with FreeBSD, I love integrated PKGBUILD systems on Linux OS - reminds me of the port tree.

Have you heard of this program that can allow one to boot from ZFS snapshots? It integrates with SystemD boot (formerly Gummiboot). https://github.com/dasJ/sd-zfs

I have grub on my Arch ZFS laptop so I haven't taken the time to try and switch it over... seems like an awfully handy thing to have working in case of emergency, though.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
No, you can not boot from ZFS snapshots. You CAN boot from a clone of a snapshot. (Okay, I am being nit picky :).

Yes, I have been doing ABEs, (Alternate Boot Environments), with OpenZFS for 3.5 years now. Before that I used BTRFS for 3 years, (with ABEs). And for several years before that, 2 different root FS partitions, alternating between them so I always had a backout plan and bootable system.

Here is roughly what I do for ZFS ABEs;

Code:
> df -h /
Filesystem					Size  Used Avail Use% Mounted on
rpool/root/20171224			15G  5.9G  8.3G  42% /

> zfs list -t all -r rpool/root
NAME					USED  AVAIL  REFER  MOUNTPOINT
rpool/root			 10.2G  8.30G	96K  legacy
rpool/root/20170929	5.01M  8.30G  5.97G  legacy
rpool/root/20171028	 892K  8.30G  5.52G  legacy
rpool/root/20171123	 600K  8.30G  5.70G  legacy
rpool/root/20171224	10.1G  8.30G  5.88G  legacy
rpool/root/20171224@1  1.57G	  -  5.97G  -
rpool/root/20171224@2   421M	  -  5.52G  -
rpool/root/20171224@3   686M	  -  5.70G  -

> head -19 /etc/grub.d/05_custom
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
####################
#
# Kernel 4.9.49-gentoo-r1 local version 2
# New grub entry for ZFS update via InitRD
#
menuentry 'Goblin linux-4.9.49-gentoo-r1.2a ZFS 20171224 resume' {
	 load_video
	 insmod gzio
	 insmod part_gpt
	 insmod zfs
	 linux /linux-4.9.49-gentoo-r1.2 rootfstype=zfs root=ZFS=rpool/root/20171224 ro resume=/dev/sda2
	 echo   'Loading initial ramdisk ...'
	 initrd /linux-4.9.49-gentoo-r1.2a.img
}

Yes, I just updated yesterday.

I do have a script that lists, verifies, deletes and creates ABEs. It's mostly automated for the delete and create, except for the Grub2 stuff. I now know a way to automate that too. But, I am in no hurry. It's less than 1 minute to create a new ABE, even manually adding the new ABE's Grub2 entry.

My system does not use systemd, and does not require it for either the ABEs or the scripted create / delete part.
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
No, you can not boot from ZFS snapshots. You CAN boot from a clone of a snapshot. (Okay, I am being nit picky :).

Yes, I have been doing ABEs, (Alternate Boot Environments), with OpenZFS for 3.5 years now. Before that I used BTRFS for 3 years, (with ABEs). And for several years before that, 2 different root FS partitions, alternating between them so I always had a backout plan and bootable system.

...

My system does not use systemd, and does not require it for either the ABEs or the scripted create / delete part.

Sounds great, I'll have to look into how to doing that with Arch. I mostly just got interested in it because I saw the sd-ZFS package in the AUR which said it could boot from ZFS snapshots when I went to their github page, and I know OpenSUSE did something similar by default with BRTFS (and with grub).

I haven't done anything yet though because my ZFS-on-root system is my laptop and I can't have it dying on me because I use it for work. Will have to do some testing in a VM first to make sure I can fully wrap my head around it. Messing with the boot setup sketches me out unless I know exactly what's going on.

Thanks for the ideas, definitely very cool.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
@averyfreeman
One important thing to consider, is drive failure. For a laptop that has just one working drive, you have to have good backups.

My 1TB SSD died in my laptop recently. I have good backups, but they were 125 miles away :-(. As was my old 500GB SSD. Annoying, but I had my large tablet and work laptop. (Was going back home in less than a week...)

I now have a physically small, 32GB USB drive installed on my laptop. I keep it up to date as a backup boot media. (Only for my root pool, which includes my home directory.) I wish the SDXC slot was bootable on this laptop... I have a 128GB card in it.

Next, after having good backups, is a good restoration plan. I have throughly tested my backups and restores, even full restores. They work and work well, (but time consuming).
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
@averyfreeman
One important thing to consider, is drive failure. For a laptop that has just one working drive, you have to have good backups. Next, after having good backups, is a good restoration plan. I have throughly tested my backups and restores, even full restores. They work and work well, (but time consuming).

Yeah, I've had an OCZ SSD die on me before and I lost my music computer's Hackintosh configuration. Was the biggest PIA to setup, so it really killed me. I immediately bought two Intel SSDs to mirror for that computer after that happened, and sold my OCZ warranty replacement when it arrived. Now OCZ is dead to me ;)

I'm usually better about ensuring other people's backups restore properly - it's been a while since I tested my own, but I always test them for clients. I'm sure it's going to come back to bite me one day, but thankfully as far as data is concerned, my FreeNAS server is backed up to my KVM server (which also uses ZFS), and then I have another 8TB external drive I occasionally use to back those up manually, and all my work data is generally in the cloud so I can move from computer to computer if I want to...

So you had to fresh-install an OS on an SSD that you bought in a pinch since you weren't home? That sucks. I have old Thinkpads for laptops, 3 of them since they cost next to nothing - T520, T420 and X220 and they all have an mSATA drive and a 2.5" drive, so I have to OS installed on each one ;) The two bigger ones can use an additional 2.5 in their optical drive with a bay sled, too. Really works out good in a pinch!

You can't install an OS on your SD card? Is booting the issue? Maybe you could use a USB bootloader like Clover (Which is good for booting from NVMe adapters on computers that are too old to support NVMe from their BIOS, too!)

Heh, maybe it's taking me too long to get a decent OS restore image going since I'm so busy with new OS installations ;) The snake eats its tail...
 
Last edited:

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
...
So you had to fresh-install an OS on an SSD that you bought in a pinch since you weren't home? That sucks.
...
No, I simply did without my personal laptop for a week. At home I use a desktop, so my laptop can weeks or months without use.

Whence I got home, I had access to my old 500GB SSD and the backups. Plus, the physically small 32GB USB drive. So I put it all together before I went out on the road again.
...
You can't install an OS on your SD card? Is booting the issue? Maybe you could use a USB bootloader like Clover (Which is good for booting from NVMe adapters on computers that are too old to support NVMe from their BIOS, too!)
...
My laptop's BIOS does not support booting off the SDXC card slot. I can use a grub entry on either the SSD or USB to then boot off of it. But, when I lost my 1TB SSD, I did not have a bootable USB drive. Now I do.

All 3 my other computers support booting off SDXC cards, 2 using MicroSDXC and the desktop using full sized SDXC. My next laptop MUST support booting of of both USB and SDXC.

Note that my 1TB SSD was 3.5 years old at the time of the failure, and had been in heavy use at times.
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
All 3 my other computers support booting off SDXC cards, 2 using MicroSDXC and the desktop using full sized SDXC. My next laptop MUST support booting of of both USB and SDXC.

Note that my 1TB SSD was 3.5 years old at the time of the failure, and had been in heavy use at times.

Ouch. Well at least you were prepared! That sounds like a lot of data to go up in smoke ...

What kind of laptops boot of SD cards these days? I must be really out of the loop, all my laptops are ancient...
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
@averyfreeman, the 1TB drive had mostly copies of media, (movies and T.V. show episodes). The OS backup was not that old. Less than 3 weeks, so nothing was really lost, except time.

What I mean by booting my laptop off an SD card, is in addition to any other storage option, (like 2.5" drive or mSATA). With 128GB, (and newer 256GB), SDXC cards, leaving space for the OS that takes up 10GB or so, just makes sense.
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
But can / do you use the SD(XC) card for your OS in a pinch / as a backup? Would love to look at model numbers if you'd like to share ...
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
But can / do you use the SD(XC) card for your OS in a pinch / as a backup? Would love to look at model numbers if you'd like to share ...
Yes.

The model is an Asus X200CA bought early 2014. So it's getting old by laptop standards, (3.5 years). Since it still works, I will continue to use it.
 

averyfreeman

Contributor
Joined
Feb 8, 2015
Messages
164
Ah, looks like a cute little thing to put Linux on.

So my Antergos install got borked and I learned the hard way when I tried to roll back to an earlier snapshot that /boot was a separate EXT4 partition. So when I tried to boot into the restored snapshot the vmlinuz, initramfs, etc. were mismatched to the root partition.

Apparently the devs for Antergos don't know why making /boot a ZFS dataset would be important for rolling back to an earlier snapshot, or didn't know how to do it (explained in the Arch wiki pretty well, so...)

I feel for them, I tried to install Arch on ZFS for about a weekend before I gave up and just decided to install Antergos because it has ZFS in the installer. Long term, though, this has proven not to have been the best move.

I'm pretty allergic to using too much time for any one project so when it comes to OS installs I tend to like easy / mostly automated. OpenSUSE is my new fav because it defaults to BTRFS / with all subdevs and takes snapshots whenever the package manager is invoked. Grub can boot from snapshots, etc. Practically unbreakable. It reminds me a lot of the feature Solaris had built in eons ago.

Maybe some day I'll try and roll my own Arch or Gentoo on a ZFS root again, but for now this is working great for me.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,601
In my case, I have "/boot", (EXT4), swap and ZFS on separate partitions. I just make sure that "/boot" has all the necessary InitRDs and kernels for all the boot environments that I have available. (Well, some of the oldest boot environments may not be bootable without extra work... but then I generally don't need / try them.)

As for BTRFS, I used it before I switched to ZFS. BTRFS can be used to boot alternate file systems, which could have started life as a snapshot of an earlier release. It's got some clumsiness to it, in regards to how to setup alternate boot environments. For example, if you put the original OS on the top level in BTRFS, you can't easily manage boot environments. That's fixable, but more work.

BTRFS is starting to support some good features, like datadup, (similar to ZFS' "copies=2"). That's useful for laptops and desktops with only 1 drive. Except it HAS to apply to the entire BTRFS instance, (not like ZFS which can apply to specific datasets / zvols). Plus, there are some design flaws that can't be fixed easily. For example;
  1. BTRFS is not completely copy-on-write. A simple move of a file on top of another file can lead to un-predictable results: source file gone; destination file gone; or both gone. That can't happen with ZFS unless you have multiple hardware faults.
  2. Of course BTRFS is Linux only. It's very unlikely to be ported to any other OS, Unix like or other.
  3. Then there is space available per BTRFS sub-file system. It's not really available and quota groups are pretty bad. ZFS is not as fast as it could for this function, but it WORKS.
  4. Last, their is the flaw that the block checksums are stored with the blocks, not the inode / dir block. That means a mis-write by the disks / storage sub-system can lead BTRFS to think that data it reads is good when it's not. ZFS is specifically designed to account for storage writing to the wrong location.
All that said, I would not worry too much about using BTRFS. Just be aware of it's warts, moles and pimples. (ZFS has its own share of such, it's just 8 or more years older.)
 
Last edited by a moderator:
Top