zpool import crashes FreeNAS

[Sefi]

Hello,

I am very new to FreeNAS and must admit have jumped in and set up a home NAS and Plex plugin without doing much research. Be that as it may I now have a problem which I'd appreciate some help on. The Plex plugin was used to rip all the DVDs we had in the house and this came to over 500 movies. Everything worked fine until I decided to upgrade the RAM with the finest China can produce via Ebay which caused all the problems. Please see summary below:

1- Hardware - Asus P5Q Premium MB, QX9650 Extreme processor, with 8GB RAM, DDR2 800Mhz, non ECC, unbuffered (yes I know :-(( ), 3 x 2 TB Seagate HD in raidz1 and 1 x 250GB HD for the OS

2- FreeNAS-9.3-STABLE-201605170422 installed on a 250GB HD

System worked fine until I replaced the RAM with 4 x Samsung (allegedly) 4GB sticks (The memory sticks are on the MB manufacturers QVL although they have a different CL timing of 6). The system booted and upon the Import, it would fail with more errors than I could copy off the screen and eventually ends up with a DB> prompt. I removed the 3 disks from the motherboard (detached the SATA cables), replaced the old GEIL memory sticks back in the motherboard and rebooted. The system comes back up but obviously does not see the 3 disks in the raidz1 and marks the zfs pool (called "plex") as status unknown. I have then re-attached the SATA cables of the disks to the motherboard and they all mount and are recognised by the system. However, this does not change the status of the zfs pool.

Using the GUI I can not see the zpool but via the CLI I can. So I have done the following:

Code:

[root@NAS ~]# zpool import																										
   pool: plex																													  
	 id: 10530614542600219746																									  
  state: ONLINE																													
action: The pool can be imported using its name or numeric identifier.															
config:																															
																																	
		plex											ONLINE																	
		  raidz1-0									  ONLINE																	
			gptid/96c31459-41dd-11e7-b217-00235407d497  ONLINE																	
			gptid/9760fc26-41dd-11e7-b217-00235407d497  ONLINE																	
			gptid/9844e79f-41dd-11e7-b217-00235407d497  ONLINE  

I have also checked the FreeNAS database with the following:

Code:

[root@NAS ~]# sqlite3 /data/freenas-v1.db "select * from storage_disk;"															
Disabled|Always On|2000398934016|Z4Z0SSY5||{serial_lunid}Z4Z0SSY5_5000c500795b03bb|1|1|Disabled|Auto|||ada||1|ada1				
Disabled|Always On|2000398934016|Z1E8CF89||{serial_lunid}Z1E8CF89_5000c5005d1053e0|1|2|Disabled|Auto|||ada||1|ada2				
Disabled|Always On|2000398934016|W4Z0L2K3||{serial_lunid}W4Z0L2K3_5000c5007c9940cb|1|3|Disabled|Auto|||ada||1|ada3

And

Code:

[root@NAS ~]# sqlite3 /data/freenas-v1.db "select * from storage_diskgroup;"														
Error: no such table: storage_diskgroup

Getting the zpool status I get the following:

Code:

[root@NAS ~]# zpool status																										
  pool: freenas-boot																												
state: ONLINE																													
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Jun 15 03:45:26 2017														
config:																															
																																	
		NAME		STATE	 READ WRITE CKSUM																					
		freenas-boot  ONLINE	   0	 0	 0																					
		  ada0p2	ONLINE	   0	 0	 0																					
																																	
errors: No known data errors

And attempting to import the zpool:

Code:

[root@NAS ~]# zpool import -f																									  
   pool: plex																													  
	 id: 10530614542600219746																									  
  state: ONLINE																													
action: The pool can be imported using its name or numeric identifier.															
config:																															
																																	
		plex											ONLINE																	
		  raidz1-0									  ONLINE																	
			gptid/96c31459-41dd-11e7-b217-00235407d497  ONLINE																	
			gptid/9760fc26-41dd-11e7-b217-00235407d497  ONLINE																	
			gptid/9844e79f-41dd-11e7-b217-00235407d497  ONLINE

And finally:

Code:

[root@NAS ~]# sqlite3 /data/freenas-v1.db "select * from storage_volume;"														  
10530614542600219746|plex||ZFS|0|1  

Trying to import the zpool causes FreeNAS to crash as mentioned above and we end up with a DB> prompt.

Could someone please guide me in the right direction as to whether the data is a lost cause or there is some hope of recovery. One thing that I must mention as I am not sure if it has a bearing on this fiasco, when I removed the SATA cables, initially, I may not have put them back in the same SATA port. Would this be an issue ordinarily?

many thanks
Sefi

 

[wblock]

FreeNAS 9.3 is a quite old version. The latest is 11.0, or use 9.10.2 if you avoid .0 releases.

 

[Sefi]

Sorry my mistake, the version is FreeNAS-9.10.2-U3 (e1497f269)

 

[rs225]

I would run a memtest on the hardware for at least 30 minutes to check for any serious memory problems.

After that, you can attempt to zpool import -o readonly=on plex and see if you can get access to the pool without a crash. If so, you would copy all data out of the pool, create a new pool, and then reload the data.

The SATA port location would not usually cause any problem, and in your case, it looks like it has located all the drives.

 

[Sefi]

Thanks for your suggestion. I ran the memtest for over an hour and it passed with no errors. Tried doing as you suggested, the screen was scrolling very rapidly and the last line I saw was "cpu reset" before the machine rebooted itself.

It gets past the loading of the ZFS pool and crashes to the debugger (db> prompt) when trying to import the zpool. The only way to get past the automatic loading of the zpool is to disconnect the SATA cables of the disk array, let the system boot and reconnect them again.

 

[rs225]

If you let it boot with the drives disconnected, then you would need to do a zpool export -f plex. Then reconnect the drives, and try the zpool import -o readonly=on plex

 

[Sefi]

Tried it and unfortunately it didnt work. With the pool drives disconnected the command zpool export -f plex returns with no such pool. After connecting the drives and the server recognising them coming online trying an import with readonly turned on has the same effect, the machine reboots itself and we are back to where we were.

 

[danb35]

There's a lot that's not up to scratch with your system, from the motherboard to the CPU to the DDR2 memory to the lack of ECC, but it's hard to say what role, if any, any of those issues would be playing in your current problems.

Try this: Do a clean install of FN11 to a different USB stick, and boot from that with the drives connected. Then see what happens when you do zpool import -f -F -n plex.

 

[Sefi]

Thank you Dan. I totally accept your points about the system set up etc. I did a fresh install of FN11 on a USB stick. I see exactly the same behaviour as before.

I tried what you suggested, again, a lot of output to the screen followed by the system rebooting it :-(

Code:

[root@freenas ~]# zpool import																									 
   pool: plex																													   
	 id: 10530614542600219746																									   
  state: ONLINE																													 
status: The pool was last accessed by another system.																			 
action: The pool can be imported using its name or numeric identifier and														 
		the '-f' flag.																											 
   see: http://illumos.org/msg/ZFS-8000-EY																						 
config:																															
																																	
		plex											ONLINE																	 
		  raidz1-0									  ONLINE																	 
			gptid/96c31459-41dd-11e7-b217-00235407d497  ONLINE																	 
			gptid/9760fc26-41dd-11e7-b217-00235407d497  ONLINE																	 
			gptid/9844e79f-41dd-11e7-b217-00235407d497  ONLINE   

Code:

[root@freenas ~]#  zpool status																									 
  pool: freenas-boot																												
state: ONLINE																													 
  scan: none requested																											 
config:																															 
																																	
		NAME		STATE	 READ WRITE CKSUM																					 
		freenas-boot  ONLINE	   0	 0	 0																					
		  da0p2	 ONLINE	   0	 0	 0																					 
																																	
errors: No known data errors   

Before running the zpool import, I checked the contents of the database and it was empty. After running the import as suggested which was followed by am automatic reboot, there appears to be data in the database as shown below.

Code:

[root@freenas ~]# sqlite3 /data/freenas-v1.db "select * from storage_disk;"														 
|Auto|Always On|Disabled|Disabled|1||{serial_lunid}Z1E8CF89_5000c5005d1053e0|ada0|1|Z1E8CF89|||ada|0|2000398934016				 
|Auto|Always On|Disabled|Disabled|1||{serial_lunid}Z4Z0SSY5_5000c500795b03bb|ada1|1|Z4Z0SSY5|||ada|1|2000398934016				 
|Auto|Always On|Disabled|Disabled|1||{serial_lunid}W4Z0L2K3_5000c5007c9940cb|ada2|1|W4Z0L2K3|||ada|2|2000398934016

Any help would be appreciated.

many thanks.

 

[danb35]

Do you have another machine you can try the import on? Perhaps a newer one, with more RAM? I'm kind of grasping at straws, unfortunately.

 

[Sefi]

Thank you Dan.

I have tried it with the following set up:

1- Hardware - MSI Z170-A PRO (MS-7971) MB, Intel Core i7 6700K processor running at 4GHz, with 32GB RAM, DDR4 1333mHz, non ECC, unbuffered, OS on 32GB, USB 3.0 stick

Exactly the same as before happens. I really do appreciate your comments. zpool import shows the details of the plex pool. Trying to import it causes a panic and machine reboots. There is however one difference. I can see the plex pool in the GUI whereas before I could only see it in the CLI. Trying to Import the Volume however causes the same behaviour in that the machine reboots itself.