OK, this *definitely* has to do with symlinks. I ran into the problem again today, and while in the process of whittling down this problem directory, I discovered that the system would kernel panic when removing the /home/user/bin directory inside the PC-BSD life-preserver backup directory.
Guess what's inside the bin dir? Lots of symlinks. (see listing below)
So there appears to be some bug with the way that FreeNAS/ZFS is handling file delete operations on those symlinks.
If there are any alternative tools for removing files or symbolic links other than the rm command, I'm all ears. It would be a huge hassle to have to back, delete and restore this dataset just to be able to delete a few dozen lousy symlinks.
[seth@nas] ~/life-preserver/sure/bin> ls
./ ebook-viewer@ launchy@ proxytrack@ webhttrack@
../ epub-fix@ libreoffice@ pwcsnap@ weechat-curses@
RetroShare@ evolution@ lobase@ pwcview@ widl@
bitcoin-qt@ evolution-settings@ localc@ rdiffdir@ wine@
calibre@ fetch-ebook-metadata@ lodraw@ regedit@ wine64-nvidiasetup@
calibre-complete@ filezilla@ loffice@ regsvr32@ wineboot@
calibre-customize@ firefox@ lofromtemplate@ remmina@ winebuild@
calibre-debug@ function_grep.pl@ loimpress@ rhythmbox@ winecfg@
calibre-mount-helper@ fzputtygen@ lomath@ rhythmbox-client@ wineconsole@
calibre-parallel@ fzsftp@ loweb@ skype@ winecpp@
calibre-server@ galculator@ lowriter@ soffice@ winedbg@
calibre-smtp@ globaltime@ lrf2lrs@ swine@ winedump@
calibre-uninstall@ gmplayer@ lrfviewer@ swinecli@ winefile@
calibredb@ gnc-fq-check@ lrs2lrf@ swinerun@ wineg++@
chrome@ gnc-fq-dump@ markdown-calibre@ tmux@ winegcc@
clipgrab@ gnc-fq-helper@ midori@ tor@ winemaker@
deluge@ gnc-fq-update@ mplayer@ tor-gencert@ winemine@
deluge-console@ gnome-mplayer@ mplayer.sh@ tor-resolve@ winepath@
deluge-gtk@ gnucash@ msiexec@ torify@ wineserver@
deluge-web@ gnucash-env@ msiexec.sh@ trafshow@ wmc@
deluged@ gnucash-make-guids@ mumble@ tz_convert@ wol@
duplicity@ gnucash-valgrind@ notepad@ uninstall-opera@ wol-bootptab@
dvx* htsserver@ opera@ unopkg@ wol-dhcpdconf@
ebook-convert@ httrack@ orage@ update-gnucash-gconf@ wrc@
ebook-device@ ikea@ pidgin@ vidalia@ xombrero@
ebook-meta@ ikec@ pinta@ web2disk@ xombrero-bin@
- - - Updated - - -
Update: I tried removing some symlinks one by one using the unlink command, and this worked. So I tried to automate the process using the find command and the system kernel panicked again.
Code:
find * -exec unlink {} \;
I tried again using the -ok switch to unlink them one by one. I pretty much pared it down to one problem symlink
lrwxr-xr-x 1 seth users 41 Feb 21 12:00 gnc-fq-check@ -> /usr/pbi/gnucash-amd64/.sbin/gnc-fq-check
I have no idea what is special about this symlink, but this is the one that consistently triggers the fatal trap kernel panic.
This FreeBSD forum thread might be the same issue.