zfs scrub stalled, zfs commands hang after upgrade to 11.1

[zuctronic]

I recently upgraded from 9.10 to 11.1-RELEASE. Everything seemed to be going fine until the first scrub began at 10:00 on Monday 12/25. Later that evening, I attempted to copy some files to the SMB share and after a few gigabytes were transferred, SMB went offline. The web interface was also inaccessible. I was not able to make a new SSH session to the FreeNAS server, but via the console I was able to get a shell and issue the reboot command. There was no response from the reboot command for over an hour, so I power cycled the system.

It appeared to come back up fine, but the scrub is not progressing. Also most zfs and zpool commands hang completely. Here is some supporting data:

Code:

root@freenas:~ # zpool status -v auto
  pool: auto
 state: ONLINE
  scan: scrub in progress since Mon Dec 25 10:00:01 2017
		69.2G scanned at 14.9M/s, 0 issued at 0/s, 10.1T total
		0 repaired, 0.00% done, no estimated completion time
config:

		NAME											STATE	 READ WRITE CKSUM
		auto											ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/5f0abe56-301a-11e7-ab6f-0cc47a68c6a8  ONLINE	   0	 0	 0
			gptid/b30e0f2a-310f-11e7-bea4-0cc47a68c6a8  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/4ce1574f-c092-11e7-a300-ac1f6b43aa8c  ONLINE	   0	 0	 0
			gptid/1d8ceaee-be52-11e7-967d-ac1f6b43aa8c  ONLINE	   0	 0	 0
		logs
		  gptid/ad4e6686-be08-11e5-8976-0cc47a68c6a8	ONLINE	   0	 0	 0
		cache
		  gptid/c02e1543-be08-11e5-8976-0cc47a68c6a8	ONLINE	   0	 0	 0

errors: No known data errors

Code:

root@freenas:~ # zpool list -v
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
auto									14.5T  10.1T  4.40T		 -	41%	69%  1.00x  ONLINE  /mnt
  mirror								7.27T  5.53T  1.74T		 -	43%	76%
	gptid/5f0abe56-301a-11e7-ab6f-0cc47a68c6a8	  -	  -	  -		 -	  -	  -
	gptid/b30e0f2a-310f-11e7-bea4-0cc47a68c6a8	  -	  -	  -		 -	  -	  -
  mirror								7.28T  4.61T  2.66T		 -	40%	63%
	gptid/4ce1574f-c092-11e7-a300-ac1f6b43aa8c	  -	  -	  -		 -	  -	  -
	gptid/1d8ceaee-be52-11e7-967d-ac1f6b43aa8c	  -	  -	  -		 -	  -	  -
log										 -	  -	  -		 -	  -	  -
  gptid/ad4e6686-be08-11e5-8976-0cc47a68c6a8   232G  1.04M   232G		 -	 0%	 0%
cache									   -	  -	  -		 -	  -	  -
  gptid/c02e1543-be08-11e5-8976-0cc47a68c6a8   224G   160G  63.3G		 -	 0%	71%
freenas-boot							14.3G  2.88G  11.4G		 -	  -	20%  1.00x  ONLINE  -
  mirror								14.3G  2.88G  11.4G		 -	  -	20%
	gptid/2b0530f2-7445-11e5-8fe2-0cc47a68c6a8	  -	  -	  -		 -	  -	  -
	da0p2								   -	  -	  -		 -	  -	  -
	da1p2								   -	  -	  -		 -	  -	  -

There isn't anything that appears relevant in /var/log/messages or /var/log/samba4/...

I do have my data backed up and I can still copy files FROM the zpool, just in case. But I'd really like to resolve the issue or at least understand what's gone wrong here.

Thanks in advance for any help provided!

 

[danb35]

https://forums.freenas.org/index.ph...very-slow-resilvering-extreme-slow-etc.60135/
https://forums.freenas.org/index.php?threads/problems-after-pool-scrub.60131/
https://redmine.ixsystems.com/issues/27437

 

[zuctronic]

That does sound like the issue I'm experiencing. Problem is, I upgraded from 9.10.2-U6 to 11.1-RELEASE so I don't have the 11.0-U4 boot environment to roll back to. I'm not sure how I should proceed.

 

[danb35]

You should be able to roll back to your previous version in any event.

 

[zuctronic]

Rolling back to 9.10.2-U6 has resolved the issue... I was able to zpool scrub -s auto and everything is back as it was before the upgrade.

Thanks for the assistance, danb35.