SOLVED FreeNAS the primary gpt table is corrupt or invalid when creating RaidZ2 with encryption (8 disks)

[devnullius]

Code:

Request Method: POST
Request URL: http://10.0.0.250/storage/volumemanager/
Software Version: FreeNAS-11.1-U1 (f7e246b8f)
Exception Type: MiddlewareError
Exception Value:
[MiddlewareError: Unable to GPT format the disk "da6": gpart: geom 'da6': File exists
]
Exception Location: ./freenasUI/middleware/notifier.py in __gpt_labeldisk, line 359
Server time: Sat, 3 Feb 2018 00:39:19 +0100

Traceback

Code:

Environment:

Software Version: FreeNAS-11.1-U1 (f7e246b8f)
Request Method: POST
Request URL: http://10.0.0.250/storage/volumemanager/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.			 response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.			 response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.		 return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.				 return view_func(request, *args, **kwargs)
File "./freenasUI/storage/views.py" in volumemanager
  148.		 if form.is_valid() and form.save():
File "./freenasUI/storage/forms.py" in save
  298.				 notifier().create_volume(volume, groups=grouped, init_rand=init_rand)
File "./freenasUI/middleware/notifier.py" in create_volume
  752.			 vdevs = self.__prepare_zfs_vdev(vgrp['disks'], vdev_swapsize, encrypt, volume)
File "./freenasUI/middleware/notifier.py" in __prepare_zfs_vdev
  687.								  swapsize=swapsize)
File "./freenasUI/middleware/notifier.py" in __gpt_labeldisk
  359.				 raise MiddlewareError(f'Unable to GPT format the disk "{devname}": {error}')

Exception Type: MiddlewareError at /storage/volumemanager/
Exception Value: [MiddlewareError: Unable to GPT format the disk "da6": gpart: geom 'da6': File exists
]

Code:

Request information
GET
No GET data

POST
Variable Value
__all__ ''
volume_name 'ZFS_8x_3TB_RAID2'
volume_add ''
encryption 'on'
layout-__all__ ''
layout-TOTAL_FORMS '2'
layout-INITIAL_FORMS '0'
layout-0-vdevtype 'raidz2'
layout-0-disks 'da3'
layout-1-vdevtype 'raidz2'
layout-1-disks 'da7'
__form_id 'dijit_form_Form_0'
FILES
No FILES data

COOKIES
Variable Value
fntreeSaveStateCookie 'root%2Croot%2F60%2F61%2Croot%2F60%2F135%2Croot%2F60%2F61%2F234%2Croot%2F60%2F243%2Croot%2F60%2F61%2F122%2F123%2Croot%2F60%2Croot%2F60%2F61%2F126%2Croot%2F60%2F61%2F146%2Croot%2F60%2F231'
csrftoken 'Q7X24Jy9evIr9YBolYCBOOZKLTWobpe65ovCz1MYaiayGxmLfkftnJxhCD5Fy8LC'
sessionid '7uxatn9x6sgd4613pesw5qjhxgwam073'
META
Variable Value

History: I created a ZFS Pool (RaidZ2, 8 disks of 3TB each). I then started a zfs import. I Ctrl-C'd that import. I then (with the GUI) removed the ZFS Pool again and tried to re-create it with encryption enabled. That doesn't work and generates the errors I'm showing above (all in the GUI).

Any idea what could be going on? The encryption is not the problem. I'll try to find some answers myself too. Here we go again - sigh :/

 

[devnullius]

PS: when the pool was destroyed, I re-arranged 3 disks - that couldn't cause any problems, right? Especially after a full reboot (which I did)?

Post-edit: the bootlogs (dmesg) show this error for 1 of the disks too. I don't think the hardware broke down, so if anyone has some ideas... I'm open for them :)

 

[devnullius]

Wiping each individual disk with the GUI.

1st disk da0: ok
2nd disk and following (da1 - da7) give these kind of errors:

Code:

Request Method: POST
Request URL: http://10.0.0.250/storage/disks/wipe/da1/
Software Version: FreeNAS-11.1-U1 (f7e246b8f)
Exception Type: MiddlewareError
Exception Value:
[MiddlewareError: Failed to wipe da1p1: dd: /dev/da1p1: Operation not permitted
]
Exception Location: ./freenasUI/middleware/notifier.py in _do_disk_wipe_quick, line 3584
Server time: Sat, 3 Feb 2018 01:05:38 +0100
Traceback

					Environment: Software Version: FreeNAS-11.1-U1 (f7e246b8f) Request Method: POST Request URL: http://10.0.0.1/storage/disks/wipe/da1/ Traceback: File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner 42.			 response = get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response 249.			 response = self._get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response 178.			 response = middleware_method(request, callback, callback_args, callback_kwargs) File "./freenasUI/freeadmin/middleware.py" in process_view 162.		 return login_required(view_func)(request, *view_args, **view_kwargs) File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view 23.				 return view_func(request, *args, **kwargs) File "./freenasUI/storage/views.py" in disk_wipe 850.				 notifier().disk_wipe(devname, form.cleaned_data['method']) File "./freenasUI/middleware/notifier.py" in disk_wipe 3607.				 self._do_disk_wipe_quick(part) File "./freenasUI/middleware/notifier.py" in _do_disk_wipe_quick 3584.				 "Failed to wipe %s: %s" % (devname, err) Exception Type: MiddlewareError at /storage/disks/wipe/da1/ Exception Value: [MiddlewareError: Failed to wipe da1p1: dd: /dev/da1p1: Operation not permitted ]

Request information
GET
No GET data

POST
Variable Value
__all__ ''
method 'quick'
__form_id 'form_DiskWipeForm'
FILES
No FILES data

COOKIES
Variable Value
fntreeSaveStateCookie 'root%2Croot%2F60%2F61%2Croot%2F60%2F135%2Croot%2F60%2F61%2F234%2Croot%2F60%2F243%2Croot%2F60%2F61%2F122%2F123%2Croot%2F60%2Croot%2F60%2F61%2F126%2Croot%2F60%2F61%2F146%2Croot%2F60%2F231'
csrftoken 'zYYznmOayao4CJYpyVy8FPR8gliVAM2PalyNdB93wExHfYjyCURb9LyqlzxtSM0p'
sessionid 'yauyoyvomcoses5m0vyt1tk7fqppbwsx'
META
Variable Value

I'll now try a full wipe :/

 

[devnullius]

Full wipe (random data) seems to be destroying da1. That'll probably take a long long time... So I'll continue this tomorrow, until I consider this case closed (though if anyone suspects why quick wipe fails here... Do let me know, I'm sure I'll learn something important here. See history mentioned above for a quick TL;DL)

 

[devnullius]

"fun" observation... If you open the GUI in several tabs, you can 'multitask' and wipe several disks at once...

Now the 'fun' part:
Do a full wipe on da0 (once started, I can Ctrl-C it even).
Done? Then you can do a quick wipe now on da1.
Try to quick wipe da2 next? Fail. It needs a full wipe (which you can cancel I think).
da2 busy with the full wipe? You can already quickly wipe da3. But not da4.
da4 needs to start a full wipe BEFORE you can QUICK wipe da5. This goes on (full wipe da6, quick wipe da7, full wipe da8).

I won't cancel the full wipes, it's a good test while I go to bed ;p But it's weird, no??

 

[devnullius]

Update: da0 still can't be quick wiped: probably Ctrl-C'ing the wipe task wasn't the smartest thing to do. Creating RAIDZ2 still fails, probably because of this. More in a few hours :)

Weird observation: while I'm wiping da0 (and that HDD's led is blinking as is expected), I have 3 more disks blinking (da2, da3 & da5)... da2 & da3 blink on and off (not as fast as da0) and da5 just remains 'on'. I really wonder wt frack is going on behind the scenes here!

 

[devnullius]

Not solved: full wipe of da0 gives up at about 17% progress?

Problem moved to: https://forums.freenas.org/index.php?threads/check-freenas_health-is-now-warning.61289/ - it also includes the console logs.