SOLVED Stuck on ZFS import

Status
Not open for further replies.
zetoniak

zetoniak

Dabbler
Joined
May 14, 2015
Messages
27
Hello,

Three days ago i updated from FreeNAS 9.10 to FreeNAS 11, made a backup before upgrade.

Today, after see that my system was completely stuck, no services, (GUI, NFS, iSCSI,..) I decided to reboot it, after reboot, my system started to import the ZFS pools, and on the second one, it got stuck, when I hit ctrl+T it displays the following:

load 0.25 cmd: zpool 1061 [tx->tx_quiesce_done_cv] 2984.17r 0.00u 1.30s 0% 3140k

Already waited like 2 hours and its still on 0%

I have 3 pools, 2 for SAS disks and a SATA pool with a SAS ZIL.

I would like to give you more info about my setup, but i cannot access to CLI.

Any suggestions?

Thanks.
 
Last edited by a moderator:
IceBoosteR

IceBoosteR

Guru
Joined
Sep 27, 2016
Messages
503
Hi,
first of all, give us some information about your system as mentioned in the forum rules.
Are those drives encrypted?
You have full access to the GUI I suggest, why not via SSH?

I would start step by step importing the pools, but first of all, get proper access to your system and provide us some inforamtion so we can start to help you.

Edit found your setup and post it here, as people accessing with tapatalk cannot read signatures:
MB: X10SRI-F-O || CPU: E5-1650v3 || RAM: 2x - 16GB DDR4-2133 2R4 || HBA: M1015 IT Mode || SATA: 7x 2GB WD Red || SAS: 6x 15k 146GB || Case: SC846-R1200B
 
zetoniak

zetoniak

Dabbler
Joined
May 14, 2015
Messages
27
Hi again,

I'm sorry for my impatience, it just finished that process after finished this post writing, now i have my FreeNAS running again, as I could read around, this could be happened because of a server high-load, plus a pool with more than 80% used...

Now I would like to investigate this problem to know its roots and fix it to do not fail again, i almost get a heart attack this rainy morning.

Can you help me to diagnose this problem?

Thank you!


My FreeNAS setup is the following:

MB: X10SRI-F-O || CPU: E5-1650v3 || RAM: 2x - 16GB DDR4-2133 2R4 || HBA: M1015 IT Mode || Pool SATA: 7x 2GB WD Red (+1 spare + SAS ZIL) || Pool SAS: 6x 15k 146GB || Pool SAS75: 6x 7.5k 2TB || Case: SC846-R1200B
Code:
#zpool list

NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
SAS			808G   310G   498G		 -	58%	38%  1.00x  ONLINE  /mnt
SAS-75		5.44T  48.0G  5.39T		 -	 1%	 0%  1.00x  ONLINE  /mnt
Sata		  10.9T  6.26T  4.62T		 -	51%	57%  1.00x  ONLINE  /mnt
freenas-boot  59.5G  6.45G  53.1G		 -	  -	10%  1.00x  ONLINE  -

Code:
#zpool status

  pool: SAS
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
		still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
		the pool may no longer be accessible by software that does not support
		the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 09:17:11 with 0 errors on Sat Dec 30 09:17:14 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		SAS											 ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/602efac0-8acc-11e5-93cc-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/62141c93-8acc-11e5-93cc-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/6408e378-8acc-11e5-93cc-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/65ef5763-8acc-11e5-93cc-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/67d53e26-8acc-11e5-93cc-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/69bbc3de-8acc-11e5-93cc-0cc47a6c1e9c  ONLINE	   0	 0	 0

errors: No known data errors

  pool: SAS-75
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
		still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
		the pool may no longer be accessible by software that does not support
		the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 00:19:17 with 0 errors on Sun Nov 26 00:19:22 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		SAS-75										  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/3b4eb46f-b581-11e7-8f25-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/3f1211c0-b581-11e7-8f25-0cc47a6c1e9c  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/42c186e1-b581-11e7-8f25-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/4691c04f-b581-11e7-8f25-0cc47a6c1e9c  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/4a9d2cde-b581-11e7-8f25-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/4ea970f7-b581-11e7-8f25-0cc47a6c1e9c  ONLINE	   0	 0	 0

errors: No known data errors

  pool: Sata
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
		still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
		the pool may no longer be accessible by software that does not support
		the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 1 days 10:49:56 with 0 errors on Sun Dec 24 10:49:57 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		Sata											ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/9721d80a-8032-11e5-b085-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/92094721-8032-11e5-b085-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/93068547-8032-11e5-b085-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/9404cce4-8032-11e5-b085-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/95079824-8032-11e5-b085-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/9618b012-8032-11e5-b085-0cc47a6c1e9c  ONLINE	   0	 0	 0
		logs
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/c50b534a-17e1-11e7-87d1-0cc47a6c1e9c  ONLINE	   0	 0	 0
			gptid/c71958e1-17e1-11e7-87d1-0cc47a6c1e9c  ONLINE	   0	 0	 0
		spares
		  gptid/acc2211a-8973-11e5-b085-0cc47a6c1e9c	AVAIL

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:52 with 0 errors on Thu Jan  4 03:45:52 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  mirror-0  ONLINE	   0	 0	 0
			ada0p2  ONLINE	   0	 0	 0
			ada1p2  ONLINE	   0	 0	 0

errors: No known data errors

Code:
# zfs list
NAME														 USED  AVAIL  REFER  MOUNTPOINT
SAS														  384G   138G   192K  /mnt/SAS
SAS-75													  48.0G  5.22T	88K  /mnt/SAS-75
SAS-75/NFS-SAS75-StorageNET								 48.0G  5.22T  48.0G  /mnt/SAS-75/NFS-SAS75-StorageNET
SAS/NFS_Vmware											   312K   138G   312K  /mnt/SAS/NFS_Vmware
SAS/PVE-Freenas-SAS										  129G   138G   129G  /mnt/SAS/PVE-Freenas-SAS
SAS/iSCSI-SAS												254G   315G  76.5G  -
Sata														5.68T  1.34T  1.45M  /mnt/Sata
Sata/.system												9.87G  1.34T   208K  legacy
Sata/.system/configs-5ece5c906a8f4df886779fae5cade8a5	   91.1M  1.34T  89.6M  legacy
Sata/.system/cores										  7.09M  1.34T  5.57M  legacy
Sata/.system/rrd-5ece5c906a8f4df886779fae5cade8a5		   7.71G  1.34T  95.3M  legacy
Sata/.system/samba4										 30.0M  1.34T  1.98M  legacy
Sata/.system/syslog-5ece5c906a8f4df886779fae5cade8a5		67.7M  1.34T  16.9M  legacy
Sata/CIFS													208G   918G   208K  /mnt/Sata/CIFS
Sata/CIFS/Administracion									2.58G  77.4G  2.47G  /mnt/Sata/CIFS/Administracion
Sata/CIFS/Financiero										2.82G  47.2G  1.94G  /mnt/Sata/CIFS/Financiero
Sata/CIFS/ISP												192K  80.0G   192K  /mnt/Sata/CIFS/ISP
Sata/CIFS/IT												13.0G  67.0G  13.0G  /mnt/Sata/CIFS/IT
Sata/CIFS/Marketing										 2.68M  80.0G   192K  /mnt/Sata/CIFS/Marketing
Sata/CIFS/Postventa										 8.43G  71.6G  8.29G  /mnt/Sata/CIFS/Postventa
Sata/CIFS/Prevencion										 192K  80.0G   192K  /mnt/Sata/CIFS/Prevencion
Sata/CIFS/Preventa										  27.2G  52.8G  27.0G  /mnt/Sata/CIFS/Preventa
Sata/CIFS/Procedimientos									5.61M  80.0G   232K  /mnt/Sata/CIFS/Procedimientos
Sata/CIFS/Prueba											 176K   918G   176K  /mnt/Sata/CIFS/Prueba
Sata/CIFS/ReportingProyectos								 192K  80.0G   192K  /mnt/Sata/CIFS/ReportingProyectos
Sata/CIFS/Tecnico											154G  25.6G   154G  /mnt/Sata/CIFS/Tecnico
Sata/COMUN												   799K  50.0G   288K  /mnt/Sata/COMUN
Sata/FTP													23.9G   176G  23.9G  /mnt/Sata/FTP
Sata/NFSStorageNET										   910G   319G   807G  /mnt/Sata/NFSStorageNET
Sata/NFS_Backup_PVE										  600G	  0   365G  /mnt/Sata/NFS_Backup_PVE
Sata/NFS_PRUEBAS											 176K  99.8M   176K  /mnt/Sata/NFS_PRUEBAS
Sata/NFS_Vmware_SATA										11.7G  1.34T  11.7G  /mnt/Sata/NFS_Vmware_SATA
Sata/Personales											 6.18M  1024G   192K  /mnt/Sata/Personales
Sata/iSCSI												  1.52T  2.35T   520G  -
Sata/iSCSI-Proxmox1										  588G  1.83T  80.1G  -
Sata/jails												  1.15G   199G   208K  /mnt/Sata/jails
Sata/jails/.warden-template-pluginjail-10.3-x64			  600M   199G   600M  /mnt/Sata/jails/.warden-template-pluginjail-10.3-x64
Sata/jails/.warden-template-pluginjail-9.3-x64			   580M   199G   576M  /mnt/Sata/jails/.warden-template-pluginjail-9.3-x64
Sata/media												  1.85T  1.15T  1.85T  /mnt/Sata/media
freenas-boot												6.45G  51.2G	31K  none
freenas-boot/ROOT										   6.36G  51.2G	25K  none
freenas-boot/ROOT/11.1-RELEASE							  6.36G  51.2G   828M  /
freenas-boot/ROOT/9.10.1-U2								   64K  51.2G   621M  /
freenas-boot/ROOT/9.10.2-U1								   56K  51.2G   636M  /
freenas-boot/ROOT/9.10.2-U3								   53K  51.2G   638M  /
freenas-boot/ROOT/9.10.2-U4								 77.5K  51.2G   638M  /
freenas-boot/ROOT/9.10.2-U5								   71K  51.2G   638M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201510290351			 58K  51.2G   511M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201511040813			 42K  51.2G   511M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201511280648			 45K  51.2G   511M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201512121950		   63.5K  51.2G   511M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201601181840			 55K  51.2G   512M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011			114K  51.2G   512M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422			 55K  51.2G   526M  /
freenas-boot/ROOT/FreeNAS-ca82ba222c0be179a6983636c50732c3  67.5K  51.2G   613M  /
freenas-boot/ROOT/Initial-Install							  1K  51.2G   510M  legacy
freenas-boot/ROOT/default									 39K  51.2G   510M  legacy
freenas-boot/grub										   61.0M  51.2G  6.84M  legacy
 
Last edited by a moderator:
IceBoosteR

IceBoosteR

Guru
Joined
Sep 27, 2016
Messages
503
Hi,
from my point of view your pools and their usage looks excellent. I had a pool with more then 90% in use and could successfully import it (but on 9.10). But I fully understand your concerns - and your little heart attack ;)

What you can do, from the GUI, to upgrade your pools to the newest version, which will also lead to some more features, but you cannot import the pool to 9.10 again.
For better overview you can put in all the output into code brackets.
You can have a look into /var/log/messages and type dmesg on CLI, maybe we can see something unusual in the first place here to help you.

Do you have deduplication enabled?
 
Last edited by a moderator:
zetoniak

zetoniak

Dabbler
Joined
May 14, 2015
Messages
27
Hi,

I dont have dedup enabled on any pool since i found it a bit dangerous if i dont have enought RAM for it.

I've been looking at dmesg and messages files, and i couldn't find anything to worry about, just im missing logs from the date server got stuck

I found the process where my server was stuck after reboot under https://github.com/zfsonlinux/zfs/blob/master/module/zfs/txg.c wich makes me think that it has some kind of relation with txg, wich makes me think that it can be the process of concatenating the data from the txg on my ZIL disks to the pool, can someone confirm this?

I'm not a programmer and i dont understand good what txg.c does.

Thnx
 
IceBoosteR

IceBoosteR

Guru
Joined
Sep 27, 2016
Messages
503
Hi,
I am not a programmer too, and from the information you have posted here, I would try to involve the developer https://redmine.ixsystems.com/projects/freenas and oppen a bug. Attach a debug of your system (system->advanced-save debug). Make the ticket private if you think it may contains personal information.
You have HDDs as ZIL devices, right? If so, yes it could be the case that your import needed more time as the txg process needs to run through the ZIL to determine if the pool is in sync.
But thats from my opinion and might not be correct.
 
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
If you are on 11.1, try moving back to 11.0U4
 
Last edited by a moderator:
IceBoosteR

IceBoosteR

Guru
Joined
Sep 27, 2016
Messages
503
rs225 said:
If you are on 11.1, try moving back to 11.0U4
Click to expand...
I would agree. But he went straight from 9.10 to 11.1. He has no 11.0-U4 boot environment ot go back to I suggest.
But that can be changed^^
 
zetoniak

zetoniak

Dabbler
Joined
May 14, 2015
Messages
27
Hello,

I was thinking to go back to 9.10 tonight, since today i got the same problem (My FreeNAS froze), I tried to analyze the problem before reboot and if i hit ctrl+t while its stuck, it gaves me some python3.6 process, but i was unable to trace it, when I did a ps aux, I saw that it could be about the "autosnap.py" or some web service related.

Now getting ready for the downgrade and checking the FreeNAS server via WEB, I could see the next behavior on my graphs (interesting at least) and I want to share it with you, maybe this can guide us to the light.



I didn't update the ZFS filesystem, because this update prohibit to roll-back version option and in this case, the roll-back is one of the fixes options that I have for sure, also I don't have backup of all the data, and I didn't read in detail how much invasive, good or bad, a ZFS update can be for the data (and the possibility of losing it).

Thanks!
 
Last edited by a moderator:
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
zetoniak said:
i didn't read in detail how much invasive, good or bad, a ZFS update can be for the data (And the possiblity of loss it)
Click to expand...
There shouldn't be any risk to your data from doing a zfs upgrade, but it would prevent rolling back to an earlier version.
 
zetoniak

zetoniak

Dabbler
Joined
May 14, 2015
Messages
27
IceBoosteR

IceBoosteR

Guru
Joined
Sep 27, 2016
Messages
503
Looks like something is comsuming all the memory which leads to not having enough RAM for the system/ZFS...
This is now on 9.10? or 11?
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Stuck on ZFS import"

Similar threads

J
  • Locked
Stuck at Beginning ZFS volume imports
Replies
10
Views
4K
cyberjock
C
Bogd
  • Locked
Stuck on Auto Import Volume
Replies
5
Views
1K
Bogd
Bogd
T
  • Locked
How long should an import check take?
Replies
9
Views
4K
technopop
T
statik
  • Locked
FreeNAS crash, stuck on Auto Import Volume
2
Replies
27
Views
11K
cyberjock
C
pycvalade
Upgrade from BETA2 to RC1 stuck on ZFS Pool import
Replies
15
Views
2K
pycvalade
pycvalade
Top