How to change disks in freenas-boot pool?

[KevDog]

I originally had a pair of SATA-DOM SSDs purchased from SuperMicro as my mirrored boot pool. After years one of the SATA-DOMs finally broke and I ordered a replacement. Unfortunately the older boot disks were labeled (Gen 1) and the new DOMs were Gen 2 -- meaning the capacity of the disks were slightly different. Although they were sold and specified as 64Gb, the older DOMs were 59.63 Gb and the new DOM is 59 GB. I went ahead and installed FreeNAS core onto the newer slightly smaller DOM and then just used the configuration file from the old setup.

I'm having problems however getting the boot pool setup correctly. I need to manually boot into the new DOM however the configuration has this:

I want to specifiy ada0 as the sole disk in the freenas-boot pool, wipe ada1 and then setup ada1 as a mirror. The GUI options however aren't letting me replace ada1 with ada0 because it says it's too small, and I don't know how to manually configure the freenas-boot pool to switch members in the pool. Is there a guide on how to do this?

 

[Patrick M. Hausen]

Backup your config, reinstall to ada0 and ada1 simultaneously - the installer will do the right thing - reimport config.

 

[KevDog]

Thanks Patrick -- I'll definitely try today. Just some thoughts off the top of my head -- mostly focused around why I can't manually switch the a boot disk from one device to another without having to reinstall. Seems like this would be a far easier solution, and then I could add the second disk as a mirror. I'm aware the middlewares doesn't have this option, however honestly this seems like just a far easier solution, similar to how I can change a boot disk or partition on any system.

 

[Patrick M. Hausen]

You cannot have two pools named "boot-pool". ZFS boot works by name, not by device. And as you found out you cannot attach the smaller device to the larger one as a mirror. So the absolute easiest and fastest way is

- save config
- reinstall
- restore config

Lesst than 30 minutes, tops.

 

[KevDog]

Any idea why when I'm trying to install from USB device why I would be getting dd: /dev/ada0: operation not permitted? Says TrueNAS installation on ada0 and ada1 has failed.

Further investigation --- not sure how to get around this problem. Entered shell on installation disk.
Camcontrol devlist listed both ada0 and ada1

gpart show /dev/ada0
34 125045357 ada0 GPT (60G)
34 1024 1 freebsd-boot (512k)
1058 6 - free - (3.0K)
1064 125044320 2 freebsd-zfs (60G)
125045385 7 - free - (3.5K)

gpart destroy /dev/ada0
gpart: Device busy

Usually device busy means its mounted, however its not. This is from the installation media

 

[Patrick M. Hausen]

Sorry, no idea. Can you put the drives into a different system and try to erase them with dd?

 

[KevDog]

Sorry to keep on this topic -- but this entire experience hasn't been that great. I finally now have identical SATA DOM drives (needed to order and took awhile). Here are the disks specifically I'm referring to:

Code:

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 63350767616 (59G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: SuperMicro SSD
   lunid: 515d92221c000281
   ident: SMC0515D92221CN36281
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 63350767616 (59G)
   Sectorsize: 512
   Mode: r0w0e0
   descr: SuperMicro SSD
   lunid: 515d92221c000213
   ident: SMC0515D92221CN23213
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Currently my boot pool looks like the following which I believe is referencing the ada0 disk 2nd partition as when I installed FreeNAS onto the drive I chose the UEFI option:

So I'm electing to add the ada1 disk to the boot pool. I choose attach and see the disk as an option:

I select ada1 and have unfortunately gone through either selecting to Use all disk space or not use all disk space, but receive the following error:

The error reads as follows:
Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 2117, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 985, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 223, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 386, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1250, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1169, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks


So from what I'm reading in the documentation, this isn't really supposed to happen.

Here is some other information:

Code:

# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool  42.5G  1.31G  41.2G        -         -     0%     3%  1.00x    ONLINE  -
tank       43.5T  13.3T  30.2T        -         -    18%    30%  1.00x    ONLINE  /mnt

Code:

# zpool status boot-pool
  pool: boot-pool
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      ada0p2    ONLINE       0     0     0

errors: No known data errors

Do I add manually on command line?? I don't think this is usually recommended.

Code:

# camcontrol devlist
<ATA WDC WD60EFAX-68S 0A82>        at scbus0 target 0 lun 0 (pass0,da0)
<ATA WDC WD60EFRX-68L 0A82>        at scbus0 target 1 lun 0 (pass1,da1)
<ATA WDC WD60EFRX-68L 0A82>        at scbus0 target 2 lun 0 (pass2,da2)
<ATA WDC WD60EFAX-68S 0A82>        at scbus0 target 3 lun 0 (pass3,da3)
<ATA WDC WD60EFRX-68L 0A82>        at scbus0 target 4 lun 0 (pass4,da4)
<ATA WDC WD6003FFBX-6 0A83>        at scbus0 target 5 lun 0 (pass5,da5)
<ATA WDC WD60EFRX-68L 0A82>        at scbus0 target 6 lun 0 (pass6,da6)
<ATA WDC WD60EFRX-68L 0A82>        at scbus0 target 7 lun 0 (pass7,da7)
<SuperMicro SSD SOB20R>            at scbus1 target 0 lun 0 (ada0,pass8)
<SuperMicro SSD SOB20R>            at scbus2 target 0 lun 0 (ada1,pass9)
<Crucial CT525MX300SSD1 M0CR031>   at scbus5 target 0 lun 0 (ada2,pass10)
<WDC WD60EFRX-68L0BN1 82.00A82>    at scbus6 target 0 lun 0 (ada3,pass11)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus7 target 0 lun 0 (ses0,pass12)

 

[Patrick M. Hausen]

Why don't you
- export and save your configuration
- remove the old boot pool device(s)
- install to the new SATA DOMs
- import your saved configration?

If you have console or IPMI access that is a matter of less than 30 minutes - I don't see how going back and forth on the forum about this issue is in any way helping instead of a waste of time.

 

[KevDog]

I ended up doing what you exactly said, however not really trying to prove a point, the documentation would suggest the other way should have worked. Thank you for your time.

 

[Patrick M. Hausen]

You are right and have over the years done probably half a dozen of such replacements myself. What can cause them to fail are - apart from the obvious new device capacity being to small - different block sizes. Sometimes it's just not worth the effort - the config re-import of TrueNAS is very very very reliable.