SOLVED HELP! Data not accessible after FreeNAS USB stick died

[ChrisVL]

Hi Guys,

Reaching out to any resident FreeNAS experts on this forum for help. Would be greatly appreciated!

Don't ask why or how but the USB stick that my FreeNAS machine used to boot from was removed from the machine and erased. Sufficed to say someone thought it wasn't in use... :(

Now my FreeNAS installation was pretty old. It was installed late 2012 and I believe was version 8.2.0 or 8.3.0.
I've attempted to recreate a bootable USB stick with that version on it but for some reason that fails during boot. Perhaps my USB stick is too large (16GB) for that version. Not sure..

Deciding that surely the latest FreeNAS version should be able to import the ZFS pool I installed FreeNAS-11.1-U6 to a stick and that boots just fine.

In the GUI it sees the ZFS volume on my 4 x 3TB disks and I can import it. However it doesn't mount properly.
So I can see /mnt/volume1 but below that there is nothing. Whereas there should be 3 subvolumes (not sure if that is the right term).

My zpool status:

Code:

[root@freenas ~]# zpool status																									 
  pool: freenas-boot																												
 state: ONLINE																													 
  scan: none requested																											 
config:																															 
																																   
	   NAME		STATE	 READ WRITE CKSUM																					 
	   freenas-boot  ONLINE	   0	 0	 0																					
		 da0p2	 ONLINE	   0	 0	 0																					 
																																   
errors: No known data errors																										
																																   
  pool: volume1																													 
 state: ONLINE																													 
status: Some supported features are not enabled on the pool. The pool can														   
	   still be used, but some features are unavailable.																		   
action: Enable all features using 'zpool upgrade'. Once this is done,															   
	   the pool may no longer be accessible by software that does not support													 
	   the features. See zpool-features(7) for details.																			
  scan: scrub repaired 0 in 0 days 06:31:21 with 0 errors on Sat Jan 27 21:31:23 2018											   
config:																															 
																																   
	   NAME											STATE	 READ WRITE CKSUM												 
	   volume1										 ONLINE	   0	 0	 0												 
		 raidz1-0									  ONLINE	   0	 0	 0												 
		   gptid/047e1194-c808-11e5-9caa-bc5ff473e7aa  ONLINE	   0	 0	 0												 
		   gptid/4ff33153-9aee-11e7-8f0e-bc5ff473e7aa  ONLINE	   0	 0	 0												 
		   gptid/1808b34f-3a13-11e2-8f7e-bc5ff473e7aa  ONLINE	   0	 0	 0												 
		   gptid/18fef43b-3a13-11e2-8f7e-bc5ff473e7aa  ONLINE	   0	 0	 0												 
																																   
errors: No known data errors			   

and zfs list:

Code:

[root@freenas ~]# zpool list																										
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT												   
freenas-boot  3.78G   852M  2.95G		 -	  -	22%  1.00x  ONLINE  -														 
volume1	   10.9T  4.66T  6.21T		 -	 4%	42%  1.00x  ONLINE  /mnt													 
[root@freenas ~]# zfs  list																										 
NAME								USED  AVAIL  REFER  MOUNTPOINT																 
freenas-boot						851M  2.83G	64K  none																		
freenas-boot/ROOT				   842M  2.83G	29K  none																		
freenas-boot/ROOT/Initial-Install	 1K  2.83G   838M  legacy																	 
freenas-boot/ROOT/default		   842M  2.83G   838M  legacy																	 
freenas-boot/grub				  6.84M  2.83G  6.84M  legacy																	 
volume1							7.75T	  0   337K  /mnt/volume1																
volume1/set1						  3T  1.31T  1.69T  /mnt/volume1/set1														   
volume1/set2						  3T  1.30T  1.70T  /mnt/volume1/set2														   
volume1/set3					   1.75T  1.75T   209K  /mnt/volume1/set3														   
volume1/syslog					 1.65M	  0  1.65M  /mnt/volume1/syslog						   

During boot I see mountpoint errors for set1, set2 and set3: Eg/ Cannot mount '/mnt/volume1/set1' : failed to create mountpoint.

I am now VERY hesitant to try anything out of fear of damaging the pool and loosing my data.
Can anyone assist and recommend some steps to take in order to properly mount the volume and access the data?

Very much appreciated!

 

[Ericloewe]

Whereas there should be 3 subvolumes (not sure if that is the right term).

No, it is not.

For starters, the term "volume" is an unfortunate relic of the past that should be forgotten. The correct term is "pool".

Now, a ZFS pool just provides generic storage. ZFS datasets provide POSIX filesystems and every pool has a top-level dataset of the same name. You presumably meant that you had three datasets. And, indeed, your zfs list output shows a bunch of conspicuously-named datasets, set1, set2 and set3 and an additional syslog dataset, which seems to have a seriously misconfigured quota.

I am now VERY hesitant to try anything out of fear of damaging the pool and loosing my data.

Your pool is fine. I suspect you have some very poorly-chosen quotas and reservations that are screwing things up and preventing the mounting. Try removing all quotas and reservations on all the datasets, then export and import the pool again.

 

[Chris Moore]

Now my FreeNAS installation was pretty old. It was installed late 2012 and I believe was version 8.2.0 or 8.3.0.

The whole time since then, you never upgraded? That is very difficult to understand.

Perhaps my USB stick is too large (16GB) for that version. Not sure..

That is just silly. You could use a whole hard drive to install the boot pool on. Too large is not a thing.

In the GUI it sees the ZFS volume on my 4 x 3TB disks and I can import it. However it doesn't mount properly.

It isn't just a matter of importing it, you have to create a share before you can see it on the network.

You probably need to review the documentation to re-familiarize yourself with how to setup the system:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Uncle Fester's Basic FreeNAS Configuration Guide
https://www.familybrown.org/dokuwiki/doku.php?id=fester:intro

 

[ChrisVL]

Thanks for taking the time to help out guys!

Ok, check; got some terminology wrong. The problem indeed is in mounting the three datasets (set1, set2 & set3).

@Chris Moore; the problem isn't caused by a lack of network sharing. I get dataset mount errors on import and can't access them from the shell:

Code:

[root@freenas ~]# zpool import volume1																							
cannot mount '/volume1/set1': failed to create mountpoint																		  
cannot mount '/volume1/set2': failed to create mountpoint																		  
cannot mount '/volume1/set3': failed to create mountpoint																		  
cannot mount '/volume1/syslog': failed to create mountpoint

Quota's and reservations don't seem to be set?

Code:

[root@freenas ~]# zfs get quota																									
NAME										   PROPERTY  VALUE  SOURCE															
freenas-boot								   quota	 none   default															
freenas-boot/ROOT							  quota	 none   default															
freenas-boot/ROOT/Initial-Install			  quota	 none   default															
freenas-boot/ROOT/default					  quota	 none   default															
freenas-boot/ROOT/default@2018-09-14-09:34:36  quota	 -	  -																  
freenas-boot/grub							  quota	 none   default															
volume1										quota	 none   default															
volume1@manual-20130227						quota	 -	  -																  
volume1@manual-20160131						quota	 -	  -																  
volume1/set1								   quota	 none   local															  
volume1/set2								   quota	 none   local															  
volume1/set3								   quota	 none   local															  
volume1/syslog								 quota	 none   default  

[root@freenas ~]# zfs get reservation																							  
NAME										   PROPERTY	 VALUE   SOURCE														
freenas-boot								   reservation  none	default														
freenas-boot/ROOT							  reservation  none	default														
freenas-boot/ROOT/Initial-Install			  reservation  none	default														
freenas-boot/ROOT/default					  reservation  none	default														
freenas-boot/ROOT/default@2018-09-14-09:34:36  reservation  -	   -															  
freenas-boot/grub							  reservation  none	default														
volume1										reservation  none	default														
volume1@manual-20130227						reservation  -	   -															  
volume1@manual-20160131						reservation  -	   -															  
volume1/set1								   reservation  none	local														  
volume1/set2								   reservation  none	local														  
volume1/set3								   reservation  none	local														  
volume1/syslog								 reservation  none	default	  

Though in the GUI I do see the following:

(set 2 is 3TB and set3 is 1.75TB)

So, I'm not sure how to follow up the suggestion of removing the quota's and reservation..

 

[ChrisVL]

The whole time since then, you never upgraded? That is very difficult to understand.

Uhmm, yes, well. I might have been a bit lazy in my maintenance (understatement). It just worked pretty well and it's been busy years that just flew by.
Also I believe I once looked at upgrading years ago, but upgrading from v8.x wasn't as easily done as it is now?

That is just silly. You could use a whole hard drive to install the boot pool on. Too large is not a thing.

Ok, it was just a thought. These past few days I've made bootable USB sticks of every V8 version on download.freenas.org though and each version just stops halfway through boot with the error: "GEOM: da0: the primary GPT table is corrupt or invalid".
That just doesn't happen with any version from V9 and up (same USB stick, the 16GB one).

Thanks for the links, will be doing some more studying.

In the mean time though I'm still unable to mount my datasets and would appreciate any further thoughts and suggestions!

 

[Ericloewe]

Though in the GUI I do see the following:

(set 2 is 3TB and set3 is 1.75TB)

So, I'm not sure how to follow up the suggestion of removing the quota's and reservation..

Change those values to zero in the GUI and let's see what happens.

 

[ChrisVL]

Well, solved it! I've managed to mount the datasets but I'm not sure what was causing the problem.

The sets were configured to mount in: /mnt/volume1/ which should have been right.
After trying a lot different things (including changing the quotas in the GUI) nothing seemed to work. Each import resulted in the same errors (failed to create mountpoint).

I ended up changing the mount points to /mnt/ and that immediately worked. Then unmounted, changed the mount points back to /mnt/volume1/, exported and re-imported and that worked too. Huh?
So the mountpoints are exactly the same as before but somehow changing and the reverting seems to have resolved the issue.

Thanks for the help!