Can't Mirror - Error: "can only attach to mirrors and top-level disks"

[Clinderw]

Context: I'm on Truenas 12.0
I've been a longtime user of using a good USB drive as my boot device. I'm trying to move off that to an SSD.
I was intending to mirror that boot pool over to the SSD but keep running into issues.

I keep trying to set up a mirror of my freenas-boot but it keeps giving me the error 'can only attach to mirrors and top-level disks'. I've tried via GUI and CLI. Ultimately trying to do this - https://www.truenas.com/community/t...-of-larger-ssds-for-boot-pool-and-data.81409/ but can't get past the mirroring part

CLI:

Via GUI:

Specific error in GUI:

Code:

Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 214, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 382, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 214, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 1986, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 217, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 361, in run await self.future File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 392, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1205, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1132, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks

Anyone have any guidance or experienced something similar?

 

[markshelbyperry]

I get the same error when trying to attach a mirror, but I'm not trying to do anything complicated--I'm literally just trying to mirror my plain-vanilla boot disk (no added partitions, no added uses, just booting the OS), which is a 60GB SSD, by attaching another, larger SSD.

 

[Clinderw]

Yea - what I’m doing is pretty vanilla too. I messed up a command and accidentally added my drive to the pool (can’t remove without borking the pool) so I’m reinstalling it from scratch now. Will see if that produces any different results. Seems like something might be messed up with mirroring the boot pool

 

[Patrick M. Hausen]

Use the CLI for low-level pool operations. Just my experience ...

So, e.g. your USB drive is /dev/da0 and you want to replace that with two SSD named /dev/ada4 and /dev/ada5 ...

Code:

# copy partition table - triple check not to pick the wrong target devices ...
gpart backup da0 | gpart restore -F /dev/ada4
gpart backup da0 | gpart restore -F /dev/ada5
# attach the newly created partitions to the boot-pool mirror
# name of the pool may be `freenas-boot` or `boot-pool` depending on the version
zpool attach boot-pool da0p2 ada4p2
zpool attach boot-pool da0p2 ada5p2
# wait for resilver to finish
# copy boot partition, works regardless of legacy or UEFI boot
dd if=/dev/da0p1 of=/dev/ada4p1 bs=1m
dd if=/dev/da0p1 of=/dev/ada5p1 bs=1m
# detach USB drive, can be pulled afterwards
zpool detach boot-pool da0p2

Works with only one new SSD, too, of course.

 

[Clinderw]

Still no dice - thanks for the feedback/advice. Any other suggestions?

Note - also tried zpool replace but that didn't work either

 

[Keynaan]

Hi,
just coming from over 8 years of nas4free, I have exactly the same problem as Clinderw.

I am on a fresh install of Truenas 12. USB Boot stick is /dev/da0 and I want to mirror it using a via USB connected SSD /dev/da1.

Code:

$ sudo gpart backup da0 | sudo gpart restore -F /dev/da1

$ zpool status
  pool: boot-pool
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      da0p2     ONLINE       0     0     0
      
errors: No known data errors

$ gpart show
=>      40  31436720  da0  GPT  (15G)
        40      1024    1  freebsd-boot  (512K)
      1064  31424512    2  freebsd-zfs  (15G)
  31425576     11184       - free -  (5.5M)

=>       40  234441568  da1  GPT  (112G)
         40       1024    1  freebsd-boot  (512K)
       1064   31424512    2  freebsd-zfs  (15G)
   31425576  203016032       - free -  (97G)
  
$ sudo zpool attach -f boot-pool da0p2 da1p2
cannot attach da1p2 to da0p2: can only attach to mirrors and top-level disks

Any ideas?

Thank you for your time.

 

[Patrick M. Hausen]

Sorry, I call "bug".

I have done that a dozen of times over various versions but not yet on any TrueNAS 12 ...

 

[Clinderw]

Submitted here - https://jira.ixsystems.com/browse/NAS-108568

 

[Keynaan]

Thank you for opening the bug. I have added some additional informations.

 

[Patrick M. Hausen]

OK, unfortunately my toy system is already at 12.0-U1, but anyway. This usually (!) works as intended:

Code:

root@freenas2[~]# zpool status boot-pool
  pool: boot-pool
state: ONLINE
  scan: scrub repaired 0B in 00:00:10 with 0 errors on Wed Dec  9 15:50:08 2020
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada6p2  ONLINE       0     0     0
        ada7p2  ONLINE       0     0     0

errors: No known data errors
root@freenas2[~]# zpool detach boot-pool ada7p2
root@freenas2[~]# zpool labelclear ada7p2
root@freenas2[~]# gpart delete -i 2 ada7
ada7p2 deleted
root@freenas2[~]# gpart delete -i 1 ada7
ada7p1 deleted
root@freenas2[~]# gpart destroy ada7
ada7 destroyed
root@freenas2[~]# gpart backup ada6 | gpart restore ada7
root@freenas2[~]# zpool attach boot-pool ada6p2 ada7p2
root@freenas2[~]# zpool status boot-pool
  pool: boot-pool
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Dec  9 16:09:27 2020
    2.22G scanned at 119M/s, 679M issued at 35.8M/s, 2.22G total
    692M resilvered, 29.94% done, 00:00:44 to go
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada6p2  ONLINE       0     0     0
        ada7p2  ONLINE       0     0     0  (resilvering)

errors: No known data errors
root@freenas2[~]# dd if=/dev/ada6p1 of=ada7p1 bs=1m
260+0 records in
260+0 records out
272629760 bytes transferred in 0.757671 secs (359825863 bytes/sec)

I had a quick chat with Allan Jude on Twitter. Both of us suspect an underlying problem hidden by an unhelpful error message, now. Is there any hint in the output of dmesg after you provoke the error?

 

[markshelbyperry]

Patrick, let me know whether I should start my own thread (I'm new here), but as mentioned above I am in a similar situation of being unable to add a mirror to boot and receiving the same error message.

I tried the suggestion above and received the same error.

 

[Keynaan]

Hi Patrick,

nope, there is no hint in dmesg. The following output was the same before I tried to attach it.

Code:

$ sudo zpool attach -f boot-pool da0p2 da1p2
cannot attach da1p2 to da0p2: can only attach to mirrors and top-level disks

$ dmesg | grep da1
da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
da1: <JMicron Generic 0508> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number <xxxxxxx>
da1: 400.000MB/s transfers
da1: 114473MB (234441648 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>

 

[Keynaan]

Oh, I forgot: da0 has no entry in dmesg.

 

[Patrick M. Hausen]

@Keynaan No entry in /var/run/dmesg.boot? The live dmesg buffer is limited. Stuff gets pushed out when new messages come in.
What's camcontrol devlist say?

 

[Keynaan]

@Patrick M. Hausen Yep, unfortunately no additional entry in /var/run/dmesg.boot after running sudo zpool attach -f boot-pool da0p2 da1p2.

camcontrol devlist shows nothing special:

Code:

$ camcontrol devlist
<  1100>                           at scbus7 target 0 lun 0 (pass9,da0)
<JMicron Generic 0508>             at scbus8 target 0 lun 0 (pass10,da1)

 

[Paravirtual]

I've been following Patrick's guide but have hit upon the same error. I am trying to take the new installation on a 16GB USB Stick and move across to 16GB of space at the start of two* 128GB SSDs, which are ada0 and ada1

Code:

root@alexandria[~]# zpool attach boot-pool da0p2 ada0p2
cannot attach ada0p2 to da0p2: can only attach to mirrors and top-level disks

(Edited below to update)

I did have some info in dmesg;

Code:

GEOM: ada0: the primary GPT table is corrupt or invalid.
GEOM: ada0: using the secondary instead -- recovery strongly advised.

But I fixed that, yet the main issue persists.



I've used Linux for Years but am completely new to BSD so am still trying to get to grips with this. Perhaps I will figure it out soon. Fortunately it's an entirely new build so there's no great loss if something goes wrong at this stage.

 

[Patrick M. Hausen]

Try gpart recover ada0

 

[Paravirtual]

Messages crossed because my account is new so my update wasn't approved yet, I've fixed the partition tables issue, no more errors about that, yet the main error (namesake of this entire thread) persists.



[Edit:]

I see now that this has been listed as a bug (closed as "expected behaviour") and the actual issue is with the mismatch in block size between SSD and USB stick;

https://jira.ixsystems.com/browse/NAS-108568 (Edit - 2 hours later I realised Clinderw, the OP, already said that, when he opened it - Sorry!)

This make some sense, and so the error message is slightly misleading and maybe a better error message will be the "fix" but the core issue of how to setup TrueNAS to boot from a mirror of Partitions on SSDs (without using the entire device) remains.

Any thoughts or suggestions, anyone?



The Installer clearly defaults to ashift=9 for installation to USB stick, I haven't tried it with an SSD to see if it makes any different (i.e. it applies an ashift=12)

I'm not sure how much of a big deal having the boot partition set to ashift=9 is anyway, it's not like a lots going to be written there, perhaps this is a non-issue, and I should just force the mirroring of da0p2 onto the ada0p2 etc. with ashift=9 and forget about it.


For reference, this is what it looks like now;

Code:

root@alexandria[~]# zpool attach -f boot-pool /dev/da0p2 /dev/ada0p2
cannot attach /dev/ada0p2 to /dev/da0p2: can only attach to mirrors and top-level disks

root@alexandria[~]# zpool attach -o ashift=9 -f boot-pool /dev/da0p2 /dev/ada0p2

root@alexandria[~]# zpool status
  pool: boot-pool
state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: resilvered 1.19G in 00:00:46 with 0 errors on Sun Dec 27 12:26:10 2020
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p2   ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0  block size: 512B configured, 4096B native

errors: No known data errors

 

[Paravirtual]

I have an idea but I need some help;

1. Install system to ~16GB USB Stick as normal [done]
2. boot once more using the installation USB stick, access Console
3. Create partitions on the target SSDs for boot-pool
4. Create new pool such as "boot-pool2" using mirror vdev of ada0p2+ada1p2 (+other ssds if desired) and correct ashift values
5. Import boot-pool from /dev/da0p2 (or whatever it's device designation is)
6. zfs send | zfs recv to get data from boot-pool to boot-pool2
7. Export boot-pool, export boot-pool2
8. import boot-pool2 as boot-pool, export it again (? - to rename the pool that exists on SSDs)
9. copy the 1st partition from da0 onto both the ssds to make them bootable
10. reboot, removing all USB sticks

I think the intent of the above is obvious but whether this will work or entirely makes sense I'm not sure. The End Goal is to end up with boot-pool on a mirror vdev of 16GB partitions on (larger) SSDs, and it be bootable, and work, etc.


The parts I'm not clear on is the zfs send & receive as the boot-pool contains numerous filesystems and I'm not sure how best to say "just copy EVERYTHING"
Alternatively, perhaps I could just "dd if=da0p2 of=ada0p2" but I suspect that will mess up zfs labels and break the mirror vdev config

I'm also not sure if simply exporting and re-importing the "boot-pool2" to rename it will work as I'm intending.



any thoughts, or a suggestion for how to obtain the end result in a simpler manner?





Separately. while playing around and trying to learn (no data on this system so it's effectively a sandbox for learning for the time being), I ended up like this;

Code:

root@alexandria[~]# zpool detach boot-pool da0p2
root@alexandria[~]# zpool status
  pool: boot-pool
state: ONLINE
  scan: resilvered 1.19G in 00:00:46 with 0 errors on Sun Dec 27 12:26:10 2020
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada1p2    ONLINE       0     0     0

errors: No known data errors

Wait, what? The non-native block size has gone!

Let's add another SSD partition to the mirror;

Code:

root@alexandria[~]# zpool attach boot-pool /dev/ada1p2 /dev/ada2p2
cannot attach /dev/ada2p2 to /dev/ada1p2: can only attach to mirrors and top-level disks

Hmm, no, doesn't like that, still. Let's force it to add by setting ashift=9 again...

Code:

root@alexandria[~]# zpool attach -o ashift=9 -f boot-pool /dev/ada1p2 /dev/ada2p2

root@alexandria[~]# zpool status
  pool: boot-pool
state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: resilvered 1.19G in 00:00:23 with 0 errors on Sun Dec 27 13:04:03 2020
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0  block size: 512B configured, 4096B native

errors: No known data errors

And now the warning is there again, but only for the second device? Well that's weird.

OK, let's add a 3rd ssd to the mirror;

Code:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0  block size: 512B configured, 4096B native
            ada3p2  ONLINE       0     0     0  block size: 512B configured, 4096B native

About what I expected...so why is that ada1 being different? let's remove it and re-add it...

Code:

 NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0  block size: 512B configured, 4096B native
            ada3p2  ONLINE       0     0     0  block size: 512B configured, 4096B native
            ada1p2  ONLINE       0     0     0  block size: 512B configured, 4096B native

yeah, I've no idea what's going on anymore.

 

[marcevan]

If I do zpool status freenas-boot it shows a gptid/long numbers.... ONLINE

it doesn't show my USB drive which is da6 (p1 = freebsd-boot, p2= freebsd-zfs

I have an SSD that does get the same error trying to add it but it does get a same sized p2 freebsd-zfs partition on it to match da6p2.

So no idea why freenas-boot doesn't explicitly show da6p2 in it.