Boot Environments attach and replace fails: Operation not permitted

[Entex]

My boot device has "recently" been DEGRADED. So I've bought two new USBs and now I'm trying to replace the degraded device with two new.

This is my process:
System -> Boot Environments -> Status -> Attach
- Member Disk = da2
- Save

[EFAULT] 'newfs_msdos -F 16 /dev/da2p1' failed: newfs_msdos: /dev/da2p1: Operation not permitted

If i try it again i get

[EFAULT] 'gpart create -s gpt -f active /dev/da2' failed: gpart: geom 'da2': File exists

That can be fixed simply by

Code:

gpart destroy -F da2

System -> Boot Environments -> Status -> Replace (On the degraded device)
- Member Disk = da2

[EFAULT] 'newfs_msdos -F 16 /dev/da2p1' failed: newfs_msdos: /dev/da2p1: Operation not permitted

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 161, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1109, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/boot.py", line 194, in replace
    boottype = await self.format(dev)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/boot.py", line 94, in format
    raise CallError('%r failed:\n%s%s' % (" ".join(command), p.stdout.decode("utf-8"), p.stderr.decode("utf-8")))
middlewared.service_exception.CallError: [EFAULT] 'newfs_msdos -F 16 /dev/da2p1' failed:
newfs_msdos: /dev/da2p1: Operation not permitted

I can manuelly do "newfs_msdos -F 16 /dev/da2p1" in the shell and it works fine. But I don't know the rest of the steps to complete a new boot device.

Current System:
FreeNAS-11.2-RELEASE
Intel(R) Core(TM) i3-6100T CPU @ 3.20GHz (4 cores)
16 GB Memory

Current Boot Device:
Kingston USB 3.1 DataTraveler Micro 16GB

New Boot Devices:
2x Kingston USB 3.0 DataTraveler 50 16GB

 

[dlavigne]

For USB sticks it is easier (and a lot easier on the sticks) to do a fresh install of the same version (selecting the mirror option) then restoring a previously saved config.

 

[Entex]

For USB sticks it is easier (and a lot easier on the sticks) to do a fresh install of the same version (selecting the mirror option) then restoring a previously saved config.

Okay, I'll try that tomorrow night.

 

[FeepingCreature]

Just ran into the same issue, and no I'm not gonna go reinstall FreeNAS, I mean, is replacing disks supposed to work or not?

 

[FeepingCreature]

Manually followed the steps over at https://www.ixsystems.com/community/threads/boot-stick-mirror-fails-in-11-1-u3.62551/ (adjusting efi for size) and it seems to have worked just fine.

 

[Kramax]

I ran into this in 11.3 as well. I was running a "worst case" set of test scenarios, one of which was to pull one of the mirrored usb boot flash drives and wipe its zfs label and the partitions on another machine and then put it back in. The system booted fine with just the one drive which was great. I then plugged the now wiped usb flash drive in and tried to replace it with the webgui. Went to System/Boot and tried to replace the failed drive in the boot pool. But had the same issue with the newfs_msdos command failing. Tried various ways of re-wiping the drive and none fixed this issue. Finally tried the replace with the webgui again (and it failed the same way) and then went to the command line. First I looked at the partition table on the drive and noticed it had created 1 partition already (the EFI partition). So I started with the newfs_msdos command and then followed the steps linked above which for ease of use to follow on searchers was:

Code:

 # repeating the command the failed on command line worked fine:
 % newfs_msdos -F 16 /dev/da10p1
 # create root partition (part 2):
 % gpart add -b 532520 -s 62095360 -t freebsd-zfs da10
 # transfer boot code:
 % gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da10
 # find the uid of the "failed" drive:
 % zpool status freenas-boot
 # replace the failed drive in the pool:
 % zpool replace freenas-boot 5066036815731576530 /dev/da10p2

I then looked at the status of the pool and it was resilvering and then came online.

I then shut the system down and removed the other usb boot flash drive to see if would boot from the newly reconstructed drive. Sadly it did not. My system did not think it was "bootable". What happened? The above linked procedure probably works for a BIOS boot, but not for an UEFI boot. The gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da10 command does not restore the EFI partition.

I checked that this was the issue by trying to mount the EFI partition for each boot usb drive:

Code:

% mount_msdosfs /dev/da9p1 /mnt/efi/a
% ls /mnt/efi/a/efi
boot
% mount_msdosfs /dev/da10p1 /mnt/efi/b
mount_msdosfs: /dev/da10p1: Invalid argument

The disk on da9 (the working boot one) mounts fine and you can see the boot directory; the one on da10 does not mount.
I hit the web and found instructions to create an EFI partition but it did not work for me. I finally resorted to the hacky "dd" command: I just bit-copied the entire EFI partition from the working boot drive to the non-working one:

Code:

% dd if=/dev/da9p1 of=/dev/da10p1

It took forever (34 minutes for 260MB)--probably should have put a blocksize in that command. But it worked! Tried to mount the EFI partition and list it out:

Code:

% mount_msdosfs /dev/da10p1 /mnt/efi/b
% ls -R /mnt/efi/b
efi

/mnt/efi/b/efi:
boot

/mnt/efi/b/efi/boot:
BOOTx64.efi startup.nsh

Success! I tried booting with each drive individually and it booted on each! Would be nice if FreeNAS made this work -- I know the advice is to re-install, but obviously it is supposed to work and just does not. But considering what I paid for the software I really cannot complain :^)

 

[scurrier]

I use mirrored USB flash drives as my boot media. For whatever reason, in the last year or two they keep getting marked as failed even though they're not. I pull them out and run badblocks on them on another system and they are fine. So FreeNAS's heuristic for "disk has gone bad" isn't working.

When I try to replace the "failed" disk with itself in the GUI, the error messages mentioned above appeared. Using Kramax's post above, I was able to figure out the procedure to replace my "failed" disk with itself. I am sharing it here since it's a little easier to follow. Note that my USB disks are the same make and size and so the part about sizing partition 2 works well because of this. If yours aren't the same you may need to use some other method to determine the size or perhaps leave the size argument out entirely? Not sure. Probably it has to match the mirrored partition's size?

So here's all the commands that I did. Be sure to replace the disk id's to match your own healthy and failed drives. My healthy was da7 and failed/replacement was da8.

Code:

# Run the command that failed in the GUI
newfs_msdos -F 16 /dev/da8p1
# Get end and start offsets from healthy partition 2 (mine was da7p2) for use in next step
gpart list
# Create root partition (partition 2).  Use end - start + 1 formula from gpart list command above to determine -s argument:
gpart add -b 532520 -s 59539456 -t freebsd-zfs da8
# transfer boot code:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da8
# bit-copy partition 1 from healthy drive to failed drive:
dd if=/dev/da7p1 of=/dev/da8p1 bs=1M
# find the uid of the "failed" drive:
zpool status freenas-boot
# Replace the failed drive in the pool.  Replace the uid in this command with yours from above.
zpool replace freenas-boot 15122851356491095954  /dev/da8p2
# Check on resilver status
zpool status freenas-boot