Persistent Swap Drive (for pool recovery)

[Lurkr]

TL;DR:
- I dun f*#@d up, lol
- Use an entire drive (ada0) as a swap memory space temporarily
- Mount said swap across reboots temporarily
- Import pool without panics/reboots.
- un-dedup data
- export
- reinstall freeNAS, and rebuild OS-side.
- I am open to ideas
- Thank you all in advance.

Intro:
Hello, I've been using FreeNAS since about 2012, but without being successful in any repair attempts I hadn't posted on here until now.

Random tidbits:
I'll spare the deets as to why I even thought to deduplicate the thing until the second post on this thread, if anyone is interested. Likewise what I've already tried. I've searched for hours on google and here for tidbits of information I could combine to result in my end goal to little avail.
I've tried adding system tunables "swapfile=/dev/ada0p1", "swapfile=/dev/ada0", editing fstab, editing rc.conf, etc.

Hopefully (and forgive my train of thought) I will be able to make this into a guide for those with kernels which panic upon importing a pool with oversized/broken dedup/arc tables (applicable to anyone who didn't follow Cyberjock's constant reminder to use more than 5GB/TB).

Problem:
I had a pool running, with some dedup datasets, which I then tried to un-dedup (create new dataset with compression, mv command from Putty, delete original deduped dataset), and then things went wrong, and a plethora of wrong turns, and hardware limitations, bring me to the conclusion that I need way more than 16GB Ram to import the 6x3TB z1 array properly.

Proposed Solution:
Adding swap is my proposed solution, and forgive me on how 'insecure' my idea is, its not to be a permanent fix, just a data recovery hackjob solution. Adding the 40GB drive in to the previously unused IDE channel.

Main Hangup:
I think that adding a tunable "swapfile" valued "/dev/ada0" of type "RC" is on the right track, but having it actually use it (showing up in swapinfo command) seems to be the difficulty.

FreeNAS:
FreeNAS-9.10.2-U2 (e1497f2)

Hardware:
Motherboard: Asus TurboV Evo
CPU: AMD Athlon II x4 635
RAM: 16352 MB
Boot: 2x Lexar 8GB mirrored
HDD (Data): 6 * 3TB (WD or Seagate, can't remember) in a RaidZ1 config (each drive has 50 GB reserved for swap)
HDD (temporary swap drive): 40GB IDE ada0
HDD Controller: (onboard)
NICs: Onboard, and Dell Gb/s card.

I also have assorted old low-sized drives available to use if need be. I do NOT have more RAM, I do have better boards with better processors, and more SATA ports though.

 

[Lurkr]

**Reserved**

 

[Lurkr]

Zpool name: TPS

For the moment I've:
Installed Correll, 10, Stable
Set system variable debug.debugger_on_panic to 0 on Loader to disable the reboot on panic
Rebooted
Imported with -N flag (zpool import -N TPS) to disable auto-mounting on import. This seemed to import alright without kernel panics. (or at least they weren't reboot-causing.
Attached the 40 GB drive to the pool, "Zpool attach TPS cache ada0", this is only to be temporary to fix any import errors and pull data to backups.
I am now scrubbing the pool, and it should be done in about 14 days or so.... merp :/

Anyways, until then I can check the status using "zpool iostat -v TPS -t 1" (verbose io stats for the TPS zpool, at 1 second timed intervals)

I am still interested in anyone who knows how to add a drive as swap across reboots, in case this doesn't pan out.

Also, is there a way to selectively choose which datasets to mount?

 

[Lurkr]

Interestingly, using "zfs list", I've found that my TPS pool has a few hidden ".system" datasets. If one of said partitions were previously borked (I have a habit of breaking things), then mounting them automatically may have borked the current running system. Hence why using the "-N" flag on the "zpool import TPS -N" worked, because the -N flag means it will not mount the datasets on import.

here is the output of "zfs list | more":

Code:

# zfs list | more 
NAME USED AVAIL REFER MOUNTPOINT 
TPS 9.67T 1.19T 224K /mnt/TPS 
TPS/.system 95.6M 1.19T 208K legacy 
TPS/.system/configs-a75e42d926e4418a8e21357ce29c963c 7.76M 1.19T 7.76M legacy 
TPS/.system/cores 34.3M 1.19T 34.3M legacy 
TPS/.system/rrd-a75e42d926e4418a8e21357ce29c963c 39.1M 1.19T 39.1M legacy 
TPS/.system/samba4 7.31M 1.19T 7.31M legacy 
TPS/.system/syslog-a75e42d926e4418a8e21357ce29c963c 6.88M 1.19T 6.88M legacy 
TPS/Android 81.3G 1.19T 81.3G /mnt/TPS/Android 
TPS/Audiobooks 48.7G 1.19T 48.7G /mnt/TPS/Audiobooks 
TPS/Audiobooks2 132M 1.19T 132M /mnt/TPS/Audiobooks2 
TPS/Books 1.05M 1.19T 1.05M /mnt/TPS/Books 
TPS/Certificate backups 42.7M 1.19T 42.7M /mnt/TPS/Certificate backups 
TPS/Documents 511G 1.19T 511G /mnt/TPS/Documents 
TPS/Documents-Deduped 569G 1.19T 569G /mnt/TPS/Documents-Deduped 
TPS/Drivers 6.79G 1.19T 6.79G /mnt/TPS/Drivers 
TPS/FreeNAS_Info 344K 1.19T 344K /mnt/TPS/FreeNAS_Info 
TPS/Game-Installs 43.1G 1.19T 43.1G /mnt/TPS/Game-Installs 
TPS/Game-Saves 192K 1.19T 192K /mnt/TPS/Game-Saves 
TPS/ISO 498G 1.19T 498G /mnt/TPS/ISO 
TPS/ISO-Ded 896G 1.19T 896G /mnt/TPS/ISO-Ded 
TPS/Karaoke 116G 1.19T 116G /mnt/TPS/Karaoke 
TPS/KodiDatabase 192K 1.19T 192K /mnt/TPS/KodiDatabase 
--More--(byte 2086)																					  

TPS/Movies 1.08T 1.19T 1.08T /mnt/TPS/Movies 
TPS/Movies2Sort 9.87G 1.19T 9.87G /mnt/TPS/Movies2Sort 
TPS/Music 518G 1.19T 518G /mnt/TPS/Music 
TPS/Music2Sort 385G 1.19T 385G /mnt/TPS/Music2Sort 
TPS/NextCloud 3.69G 1.19T 3.69G /mnt/TPS/NextCloud 
TPS/Old 746G 1.19T 746G /mnt/TPS/Old 
TPS/OneNote 13.0G 1.19T 13.0G /mnt/TPS/OneNote 
TPS/Photos 528G 1.19T 528G /mnt/TPS/Photos 
TPS/Sites 16.0M 1.19T 16.0M /mnt/TPS/Sites 
TPS/Sites-deduped 101G 1.19T 101G /mnt/TPS/Sites-deduped 
TPS/Software 639G 1.19T 639G /mnt/TPS/Software 
TPS/TV 252G 1.19T 252G /mnt/TPS/TV 
TPS/TV-Duplicates 2.30T 1.19T 2.30T /mnt/TPS/TV-Duplicates 
TPS/TV-Ignore 50.9G 1.19T 50.9G /mnt/TPS/TV-Ignore 
TPS/TV2Sort 7.43G 1.19T 7.43G /mnt/TPS/TV2Sort 
TPS/Torrents_Old 92.5G 1.19T 92.5G /mnt/TPS/Torrents_Old 
TPS/Videos 207G 1.19T 207G /mnt/TPS/Videos 
TPS/WSUS 46.7G 1.19T 46.7G /mnt/TPS/WSUS 
TPS/jails 8.42G 1.19T 5.09M /mnt/TPS/jails 
TPS/jails/.warden-template-pluginjail 601M 1.19T 600M /mnt/TPS/jails/.warden-template
-pluginjail 
TPS/jails/couchpotato_1 310M 1.19T 902M /mnt/TPS/jails/couchpotato_1 
TPS/jails/emby_1 1.82G 1.19T 2.40G /mnt/TPS/jails/emby_1 
TPS/jails/firefly_1 115M 1.19T 709M /mnt/TPS/jails/firefly_1 
TPS/jails/headphones_1 569M 1.19T 1.13G /mnt/TPS/jails/headphones_1 
TPS/jails/lazylibrarian_1 295M 1.19T 887M /mnt/TPS/jails/lazylibrarian_1 
TPS/jails/madsonic_1 693M 1.19T 1.26G /mnt/TPS/jails/madsonic_1 
TPS/jails/nextcloud_1 2.21G 1.19T 2.79G /mnt/TPS/jails/nextcloud_1 
TPS/jails/owncloud_1 651M 1.19T 1.21G /mnt/TPS/jails/owncloud_1 
TPS/jails/plexmediaserver_1 358M 1.19T 953M /mnt/TPS/jails/plexmediaserver_
1 
TPS/jails/sickrage_1 788M 1.19T 1.35G /mnt/TPS/jails/sickrage_1 
TPS/jails/transmission_1 106M 1.19T 700M /mnt/TPS/jails/transmission_1 
freenas-boot 1.22G 5.98G 64K none 
freenas-boot/.system-da7cb443 104M 5.98G 104M /var/db/system 
freenas-boot/ROOT 1.11G 5.98G 29K none 
freenas-boot/ROOT/Initial-Install 1K 5.98G 1.11G legacy 
--More--(byte 5420)																					  

freenas-boot/ROOT/default 1.11G 5.98G 1.11G legacy 
freenas-boot/grub 4.88M 5.98G 4.88M legacy						

Boring side-notes:
Above you can see a lot of odd names for datasets, and several close-to-duplicate names, because I was in the middle of making new datasets, moving all data from old datasets to new ones, then deleting the old datasets, and then re-naming the new datasets to the old names, or vice-versa,

I hope I used the right 'code' flag while typing this in, so as not to create a super long post...

 

[Lurkr]

I'm going to try deleting the random legacy system datasets after a much needed rest. Until then, props to anyone who's made it this far through the epic conquest of understanding what I'm writing :)

 

[Lurkr]

I've destroyed the TPS/.system and TPS/jails datasets, and all recursive ones thereunder, with:

Code:

zfs destroy -r TPS/.system
zfs destroy -r TPS/jails

I don't need the jails atm, and I'll drop them onto a separate SSD later with an automated snapshot task of some sort for backups.

 

[Lurkr]

I've perhaps stumbled upon a limitation or a bug, or simply something I missed in the documentation;

When importing a pool, with more than one dataset, if the dataset name is the same as the first characters of another dataset, (specifically followed by a dash), it throws a kernel panic.

I do not have a machine to test this theory on, but maybe someone out there has one they can try.

It seems I was able to use:
zpool import TPS -N
# to import my pool without mounting anything
# then I renamed the "Documents", "ISO", "Sites", and "TV" each to "Documents-old", "ISO-old", "Sites-old", and "TV-old" respectively (or close to), and rebooted.
#Also I tried an additional thing to hopefully avoid the root dataset from being mounted, by setting the mount point of the TPS to "none".
zfs set mountpoint=none TPS

I was then able to successfully begin to import the pool using the GUI, which then began to scrub the pool automagically.

 

[Lurkr]

So, like, it rebooted and took a while to come up with this, maybe someone could help me decode it or point out some documentation on decoding messages like these, along with commands which may work at the debug prompt.



After this I just typed "reboot" and pressed enter.

 

[Lurkr]

Seems to happen on every reboot, no web online, but the numbers in the error change only ever so slightly each time

 

[Lurkr]

Right after the plugin "ISCSIInitiatorPlugin" is loaded, the motherboard's disk light remains constantly on (lots of IOs/s) for a few minutes before reaching that debug prompt you see above. So whatever is loaded next in the boot sequence after the ISCSIInitiator seems to be choking up.
It may also be noted that I have not turned on or touched any settings related to ISCSI after fresh install, and don't need to

 

[SweetAndLow]

Hardware could probably be unsupported, stop using corral when trying to debug things you might be hitting new bugs not problems with your system. Did this system ever work?

 

[Lurkr]

Yes, the system worked quite well for a while, was deduping things alright (yes, I realize now the implications of dedup vs compression). The problem seems to be the same independent of Corral vs 9.10, the kernel panics at the same points of time, doing the same sorts of thnigs, but in 9.10 it was rebooting rather than going to db> prompt.

Dedup was then affecting read/write speeds very poorly, and so I built secondary datasets with dedup off, then was using the mv command to move all the data from the deduped datasets over to the un-deduped new sets.

I don't have enough storage to back up everything externally unfortunately.

&&Thank you for your reply!!

 

[Lurkr]

There is a possibility that while using the mv command I moved around some symlinks which could be causing problems as well, I'm not entirely sure how to sniff them out properly, and a symlink loop could cause a kernel panic on import. It's yet another of a great deal of possible theories as to what is wrong. I'm fairly terrible at troubleshooting, so much so I have the keystrokes of windows installs nearly memorized I can install Windows blindly to computers now lol.

 

[Lurkr]

See next msg

 

[Lurkr]

The Instruction Pointer seems to be the same in every freeze, but the Stack Pointer and the Frame Pointer seem to be identical to each other but different (and ascending) in each picture I'm posting.

It could be that it is slowly repairing, and each error it needs a reboot, and then each time its rebooted it gets a little touch further.

Next time it happens I'm going to try to boot to previous evironment (where the pool was not imported yet) and import using -N flag, and try running a scrub. It'll take about two weeks, but maybe by then some suggestions will fly around as to what little things I should check, or where my attention should be focused on in order to troubleshoot it properly.

 

[Lurkr]

Well, I think I found a problem....

Somehow, my symbolic link loop theory must be true, because TPS/Documents-Deduped and TPS/Documents-Final ended up each inside themselves...

Observe:

Code:

[root@freenas] ~# zfs list
NAME													USED  AVAIL  REFER  MOUNTPOINT
TPS													9.66T  1.20T   224K  /mnt/TPS
TPS/Android											81.3G  1.20T  81.3G  /mnt/TPS/Android
TPS/Audiobooks										 48.7G  1.20T  48.7G  /mnt/TPS/Audiobooks
TPS/Audiobooks2										 132M  1.20T   132M  /mnt/TPS/Audiobooks2
TPS/Books											  1.05M  1.20T  1.05M  /mnt/TPS/Books
TPS/Certificate-Backups								42.7M  1.20T  42.7M  /mnt/TPS/Certificate-Backups
TPS/Documents-Deduped								   569G  1.20T   569G  /mnt/TPS/Documents-Deduped
TPS/Documents-Deduped/Documents-Deduped				 264K  1.20T   264K  /mnt/TPS/Documents-Deduped/Documents-
TPS/Documents-Final									 511G  1.20T   511G  /mnt/TPS/Documents-Final
TPS/Documents-Final/Documents-Final					 264K  1.20T   264K  /mnt/TPS/Documents-Final/Documents-Fi
TPS/Drivers											6.79G  1.20T  6.79G  /mnt/TPS/Drivers
TPS/FreeNAS_Info										344K  1.20T   344K  /mnt/TPS/FreeNAS_Info
TPS/Game-Installs									  43.1G  1.20T  43.1G  /mnt/TPS/Game-Installs
TPS/Game-Saves										  192K  1.20T   192K  /mnt/TPS/Game-Saves
TPS/ISO-Ded											 896G  1.20T   896G  /mnt/TPS/ISO-Ded
TPS/ISO-Other										   498G  1.20T   498G  /mnt/TPS/ISO-Other
TPS/Karaoke											 116G  1.20T   116G  /mnt/TPS/Karaoke
TPS/KodiDatabase										192K  1.20T   192K  /mnt/TPS/KodiDatabase
TPS/Movies											 1.08T  1.20T  1.08T  /mnt/TPS/Movies
TPS/Movies2Sort										9.87G  1.20T  9.87G  /mnt/TPS/Movies2Sort
TPS/Music											   518G  1.20T   518G  /mnt/TPS/Music
TPS/Music2Sort										  385G  1.20T   385G  /mnt/TPS/Music2Sort
TPS/NextCloud										  3.69G  1.20T  3.69G  /mnt/TPS/NextCloud
TPS/Old												 746G  1.20T   746G  /mnt/TPS/Old
TPS/OneNote											13.0G  1.20T  13.0G  /mnt/TPS/OneNote
TPS/Photos											  528G  1.20T   528G  /mnt/TPS/Photos
TPS/Sites-Other										16.0M  1.20T  16.0M  /mnt/TPS/Sites-Other
TPS/Sites-deduped									   101G  1.20T   101G  /mnt/TPS/Sites-deduped
TPS/Software											639G  1.20T   639G  /mnt/TPS/Software
TPS/TPS												 176K  1.20T   176K  /mnt/TPS/TPS
TPS/TV-Duplicates									  2.30T  1.20T  2.30T  /mnt/TPS/TV-Duplicates
TPS/TV-Ignore										  50.9G  1.20T  50.9G  /mnt/TPS/TV-Ignore
TPS/TV-Old											  252G  1.20T   252G  /mnt/TPS/TV-Old
TPS/TV2Sort											7.43G  1.20T  7.43G  /mnt/TPS/TV2Sort
TPS/Torrents_Old									   92.5G  1.20T  92.5G  /mnt/TPS/Torrents_Old
TPS/Videos											  207G  1.20T   207G  /mnt/TPS/Videos
TPS/WSUS											   46.7G  1.20T  46.7G  /mnt/TPS/WSUS
freenas-boot										   1.45G  5.76G	64K  none
freenas-boot/.system-775ecfd5						   105M  5.76G   105M  /var/db/system
freenas-boot/ROOT									  1.33G  5.76G	29K  none
freenas-boot/ROOT/Corral-RELEASE						316K  5.76G  1.11G  /
freenas-boot/ROOT/Corral-RELEASE-copy-1					0  5.76G  1.11G  /
freenas-boot/ROOT/Corral-RELEASE-copy-1-copy-1			 0  5.76G  1.11G  /
freenas-boot/ROOT/Corral-RELEASE-copy-1-copy-1-copy-1  1.33G  5.76G  1.11G  /
freenas-boot/ROOT/Initial-Install						 1K  5.76G  1.11G  legacy
freenas-boot/ROOT/default							   228K  5.76G  1.11G  legacy
freenas-boot/grub									  4.88M  5.76G  4.88M  legacy

I'm going to try mounting each dataset one by one and piecing the bits together properly, and destroying the nested things. But all after backing up everything

 

[Lurkr]

So, waiting for a scrub to complete has not repaired anything, here is the outputs of:

Code:

zpool status -v TPS
zfs list
zfs destroy TPS/ISO1

which then panicked the server.

Code:

[root@freenas] ~# zpool status -v TPS
  pool: TPS
 state: ONLINE
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 37h34m with 1 errors on Wed Mar 22 22:43:20 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		TPS											 ONLINE	   0	 0	 1
		  raidz2-0									  ONLINE	   0	 0	 6
			gptid/39e24424-a329-11e6-80ae-485b39ee36d5  ONLINE	   0	 0	 0
			gptid/3ad9269d-a329-11e6-80ae-485b39ee36d5  ONLINE	   0	 0	 0
			gptid/3bb2edd9-a329-11e6-80ae-485b39ee36d5  ONLINE	   0	 0	 0
			gptid/3ca5fb57-a329-11e6-80ae-485b39ee36d5  ONLINE	   0	 0	 0
			gptid/3d8be750-a329-11e6-80ae-485b39ee36d5  ONLINE	   0	 0	 0
			gptid/3e59b4d0-a329-11e6-80ae-485b39ee36d5  ONLINE	   0	 0	 0
		cache
		  ada0										  ONLINE	   0	 0	 0
		  ada1										  ONLINE	   0	 0	 0

errors: Permanent errors have been detected in the following files:

		<metadata>:<0x7d>
[root@freenas] ~# zfs list
NAME									  USED  AVAIL  REFER  MOUNTPOINT
TPS									  10.1T   730G   224K  /mnt/TPS
TPS/Android							  81.3G   730G  81.3G  /mnt/TPS/Android
TPS/Audiobooks1						  33.8G   730G  33.8G  /mnt/TPS/Audiobooks1
TPS/Audiobooks2						  48.8G   730G  48.8G  /mnt/TPS/Audiobooks2
TPS/Books								1.05M   730G  1.05M  /mnt/TPS/Books
TPS/Certificate-Backups				  42.7M   730G  42.7M  /mnt/TPS/Certificate-Backups
TPS/Documents-Deduped					 569G   730G   569G  /mnt/TPS/Documents-Deduped
TPS/Documents-Final					   511G   730G   511G  /mnt/TPS/Documents-Final
TPS/Drivers							  6.79G   730G  6.79G  /mnt/TPS/Drivers
TPS/FreeNAS_Info						  344K   730G   344K  /mnt/TPS/FreeNAS_Info
TPS/Game-Installs						43.1G   730G  43.1G  /mnt/TPS/Game-Installs
TPS/Game-Saves							192K   730G   192K  /mnt/TPS/Game-Saves
TPS/ISO1								  896G   730G   896G  /mnt/TPS/ISO1
TPS/ISO2								  959G   730G   959G  /mnt/TPS/ISO2
TPS/Karaoke							   116G   730G   116G  /mnt/TPS/Karaoke
TPS/KodiDatabase						  192K   730G   192K  /mnt/TPS/KodiDatabase
TPS/Movies							   1.08T   730G  1.08T  /mnt/TPS/Movies
TPS/Movies2Sort						  9.87G   730G  9.87G  /mnt/TPS/Movies2Sort
TPS/Music								 518G   730G   518G  /mnt/TPS/Music
TPS/Music2Sort							385G   730G   385G  /mnt/TPS/Music2Sort
TPS/NextCloud							3.69G   730G  3.69G  /mnt/TPS/NextCloud
TPS/Old								   746G   730G   746G  /mnt/TPS/Old
TPS/OneNote							  13.0G   730G  13.0G  /mnt/TPS/OneNote
TPS/Photos								528G   730G   528G  /mnt/TPS/Photos
TPS/Sites-Other						  16.0M   730G  16.0M  /mnt/TPS/Sites-Other
TPS/Sites-deduped						 101G   730G   101G  /mnt/TPS/Sites-deduped
TPS/Software							  639G   730G   639G  /mnt/TPS/Software
TPS/TV-Duplicates						2.30T   730G  2.30T  /mnt/TPS/TV-Duplicates
TPS/TV-Ignore							50.9G   730G  50.9G  /mnt/TPS/TV-Ignore
TPS/TV-Old								252G   730G   252G  /mnt/TPS/TV-Old
TPS/TV2Sort							  7.43G   730G  7.43G  /mnt/TPS/TV2Sort
TPS/Torrents_Old						 92.5G   730G  92.5G  /mnt/TPS/Torrents_Old
TPS/Videos								207G   730G   207G  /mnt/TPS/Videos
TPS/WSUS								 46.7G   730G  46.7G  /mnt/TPS/WSUS
freenas-boot							 2.04G  5.16G	64K  none
freenas-boot/.system-775ecfd5			 715M  5.16G   715M  /var/db/system
freenas-boot/ROOT						1.33G  5.16G	29K  none
freenas-boot/ROOT/CopyA-copy-1			  1K  5.16G  1.11G  /
freenas-boot/ROOT/Corral-RELEASE		  308K  5.16G  1.11G  /
freenas-boot/ROOT/Corral-RELEASE-copy-1  1.33G  5.16G  1.11G  /
freenas-boot/ROOT/Initial-Install		   1K  5.16G  1.11G  legacy
freenas-boot/ROOT/default				 228K  5.16G  1.11G  legacy
freenas-boot/grub						4.88M  5.16G  4.88M  legacy
[root@freenas] ~# zpool destroy TPS/ISO1
cannot open 'TPS/ISO1': invalid character '/' in pool name
use 'zfs destroy' to destroy a dataset
[root@freenas] ~# zfs destroy TPS/ISO1

Apparently I need to save up some dough and make another pool :/