SOLVED It appears FreeNAS deleted my system dataset in the migration page

[BurntTech]

Hey guys,
I fear i'm on borrowed time as I have no system dataset currently. After reading 2 different forum posts about moving the system dataset I decided it was time. I used the Web GUI to select a new Volume for the system dataset. To my amazement I watched it not create the new one and delete the source one using zfs list. I feel like it should leave a copy and a few more checks and balances before it deletes but that is not the current topic (Might be a bug while running domain joined since samba doesn't like to restart while AD is enabled). The GUI doesn't load the system dataset page any more so I can't fix this from the GUI. It also appears smb is running but not serving. I still can see my data on disk at least. Looks like the log files were a link to the system dataset. Not sure where to start on missing system dataset.

Code:

lrwxr-xr-x   1 root	wheel	   58 Jan  1 23:44 log -> /var/db/system/syslog-76c11d7f8a944b3d8e42fe35420dbaa3/log

EDIT
Here was the error message I had on the dataset page. After a few minutes the page loads but i'm not going to touch it since I don't see a system dataset in zfs list still.

Code:

Request Method: POST

Request URL: http://nas.domain.com/admin/system/systemdataset/edit/1/?inline=true

Software Version: FreeNAS-11.1-RELEASE (dc7d195f4)

Exception Type: ClientException

Exception Value:

[Errno 66] Directory not empty: '/tmp/system.new'

Exception Location: /usr/local/lib/python3.6/site-packages/middlewared/client/client.py in call, line 394

Server time: Wed, 3 Jan 2018 09:46:11 -0600

Traceback



Environment:



Software Version: FreeNAS-11.1-RELEASE (dc7d195f4)

Request Method: POST

Request URL: http://nas.domain.com/admin/system/systemdataset/edit/1/?inline=true





Traceback:

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner

42.			 response = get_response(request)

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response

249.			 response = self._get_response(request)

File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response

178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)

File "./freenasUI/freeadmin/middleware.py" in process_view

162.		 return login_required(view_func)(request, *view_args, **view_kwargs)

File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view

23.				 return view_func(request, *args, **kwargs)

File "./freenasUI/freeadmin/options.py" in wrapper

210.				 return self._admin.admin_view(view)(*args, **kwargs)

File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view

149.					 response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func

57.		 response = view_func(request, *args, **kwargs)

File "./freenasUI/freeadmin/site.py" in inner

142.			 return view(request, *args, **kwargs)

File "./freenasUI/freeadmin/options.py" in edit

571.					 mf.save()

File "./freenasUI/system/forms.py" in save

1747.			 pk = c.call('systemdataset.update', data)

File "./freenasUI/system/forms.py" in save

1747.			 pk = c.call('systemdataset.update', data)

File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call

394.			 raise ClientException(c.error, c.errno, c.trace, c.extra)



Exception Type: ClientException at /admin/system/systemdataset/edit/1/

Exception Value: [Errno 66] Directory not empty: '/tmp/system.new'







Request information

GET

Variable Value

inline 'true'

POST

Variable Value

__all__ ''

sys_pool 'Plugins'

sys_syslog_usedataset 'on'

sys_rrd_usedataset 'on'

__form_id 'dialogForm_systemdataset'

FILES

No FILES data

 

[BurntTech]

sigh, it appears most items are linked to this system dataset. What a mess. It appears in the tmp folder its complaining about there is some config db files

Code:

[root@nas /tmp/system.new/configs-76c11d7f8a944b3d8e42fe35420dbaa3/FreeNAS-11.1-RELEASE (dc7d195f4)]# ls -l
total 1808
-rw-r-----  1 root  wheel  925696 Jan  2 03:45 20180102%s.db
-rw-r-----  1 root  wheel  925696 Jan  3 03:45 20180103%s.db

 

[BurntTech]

well this type of thing shouldn't happen which would explain not a lot of information about it. Here is my guide to fix if this happens to someone else.

NOTE:My Dir Hash will be different then yours i'm pretty sure

1. Check if you have a /tmp/system.new backup using "ls -l /tmp/system.new" Example below
2. [root@nas ~]# ls -l /tmp/system.new/
   total 1
   drwxr-xr-x 4 root wheel 128 Jan 2 03:45 configs-76c11d7f8a944b3d8e42fe35420dbaa3
   drwxrwxr-x 2 root wheel 448 Jan 1 14:44 cores
   -rw-r--r-- 1 root wheel 0 Oct 2 21:11 nfs-stablerestart
   -rw-r--r-- 1 root wheel 0 Oct 2 21:11 nfs-stablerestart.bak
   drwxr-xr-x 3 root wheel 128 Jan 3 09:34 rrd-76c11d7f8a944b3d8e42fe35420dbaa3
   drwxr-xr-x 4 root wheel 896 Jan 3 09:44 samba4
   drwxr-xr-x 3 root wheel 64 Oct 2 21:11 syslog-76c11d7f8a944b3d8e42fe35420dbaa3
   
   
3. copy them to non-temp location like "/root/system.new"
   
4. go back into the GUI under system then system dataset and click save even though it will still report the same volume but it will attempt to recreate the mount points
5. use "zfs list" to validate it created the correct datasets Example below
6. Plugins/.system 11.6M 2.55T 96K legacy
   Plugins/.system/configs-76c11d7f8a944b3d8e42fe35420dbaa3 88K 2.55T 88K legacy
   Plugins/.system/cores 88K 2.55T 88K legacy
   Plugins/.system/rrd-76c11d7f8a944b3d8e42fe35420dbaa3 11.2M 2.55T 11.2M legacy
   Plugins/.system/samba4 88K 2.55T 88K legacy
   Plugins/.system/syslog-76c11d7f8a944b3d8e42fe35420dbaa3 88K 2.55T 88K legacy
   
   
7. use "mount | grep system" Example below
8. Plugins/.system on /var/db/system (zfs, local, nfsv4acls)
   Plugins/.system/cores on /var/db/system/cores (zfs, local, nfsv4acls)
   Plugins/.system/samba4 on /var/db/system/samba4 (zfs, local, nfsv4acls)
   Plugins/.system/syslog-76c11d7f8a944b3d8e42fe35420dbaa3 on /var/db/system/syslog-76c11d7f8a944b3d8e42fe35420dbaa3 (zfs, local, nfsv4acls)
   Plugins/.system/rrd-76c11d7f8a944b3d8e42fe35420dbaa3 on /var/db/system/rrd-76c11d7f8a944b3d8e42fe35420dbaa3 (zfs, local, nfsv4acls)
   Plugins/.system/configs-76c11d7f8a944b3d8e42fe35420dbaa3 on /var/db/system/configs-76c11d7f8a944b3d8e42fe35420dbaa3 (zfs, local, nfsv4acls)
   
   
9. service samba_server stop
   
10. cp -a /root/system.new/configs-76c11d7f8a944b3d8e42fe35420dbaa3/ /var/db/system/configs-76c11d7f8a944b3d8e42fe35420dbaa3/
    
11. cp -a /root/system.new/rrd-76c11d7f8a944b3d8e42fe35420dbaa3/ /var/db/system/rrd-76c11d7f8a944b3d8e42fe35420dbaa3
    
12. cp -a /root/system.new/syslog-76c11d7f8a944b3d8e42fe35420dbaa3/ /var/db/system/syslog-76c11d7f8a944b3d8e42fe35420dbaa3
    
13. cp -a /root/system.new/samba4/ /var/db/system/samba4
    
14. cp -a /root/system.new/cores/ /var/db/system/cores
    
15. service samba_server start

Not sure if anything is needed with nfs-stablerestart yet

 

[dlavigne]

This should not happen. Please create a report at bugs.freenas.org and post the issue number here.

 

[BurntTech]

Bug Posted
https://redmine.ixsystems.com/issues/27619

 

[Kunzy]

BurntTech

I just had this exact thing happen to me just now. I changed the location of my system dataset and things went south. I got the same exact error message you got.

I made an account just now for the sole purpose of saying thanks for the recovery steps. Your steps worked perfectly. (Well, I did have to refresh the whole page to get the error to go away on the system dataset page so I could click save)

Anyway, you are the man.

 

[BurntTech]

I'm glad this information could help you solve your problem too. Take care

 

[FreeVel]

I have been facing this issue since upgrade to FreeNAS 11.1-U1.
I get this error whenever I try to change location of the system dataset.
To recover I always go back to the UI or refresh it, and just click the "save" button. No need for getting onto the console.

Is this a regression introduced in 11.1-U1 ?

Code:

Request Method:	POST
Request URL:	https://x.x.x.x/admin/system/systemdataset/edit/1/?inline=true
Software Version:	FreeNAS-11.1-U1 (f7e246b8f)
Exception Type:	ClientException
Exception Value:	
[Errno 66] Directory not empty: '/tmp/system.new'
Exception Location:	/usr/local/lib/python3.6/site-packages/middlewared/client/client.py in call, line 394
Server time:	Mon, 22 Jan 2018 15:06:34 +0000


Environment:

Software Version: FreeNAS-11.1-U1 (f7e246b8f)
Request Method: POST
Request URL: https://x.x.x.x/admin/system/systemdataset/edit/1/?inline=true


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.			 response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.			 response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.		 return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.				 return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
  210.				 return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  149.					 response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.		 response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
  142.			 return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in edit
  571.					 mf.save()
File "./freenasUI/system/forms.py" in save
  1775.			 pk = c.call('systemdataset.update', data)
File "./freenasUI/system/forms.py" in save
  1775.			 pk = c.call('systemdataset.update', data)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  394.			 raise ClientException(c.error, c.errno, c.trace, c.extra)

Exception Type: ClientException at /admin/system/systemdataset/edit/1/
Exception Value: [Errno 66] Directory not empty: '/tmp/system.new'

 

[BurntTech]

I would have to assume it was in before that as I got that error on Release version.

Software Version: FreeNAS-11.1-RELEASE (dc7d195f4)

 

[unwired]

So, before seeing this thread, I tried deleting /tmp/system.new to clear the POST error, but now I'm uncertain how to recover? I'm afraid to reboot the system at this point.