GUI replace broken with libzfs.ZFSException: device is too small

[Bill McCormick]

Using the GUI to replace /dev/da3 with /dev/ada0:

Code:

root@freenas:/ # diskinfo -v /dev/ada0
/dev/ada0
        512             # sectorsize
        6001175126016   # mediasize in bytes (5.5T)
        11721045168     # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        11628021        # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        WDC WD6003FFBX-68MU3N0  # Disk descr.
        V9G7LM7L        # Disk ident.
        id1,enc@n3061686369656d30/type@0/slot@1/elmdesc@Slot_00 # Physical path
        No              # TRIM/UNMAP support
        7200            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

root@freenas:/ # diskinfo -v /dev/da3
/dev/da3
        512             # sectorsize
        6001175126016   # mediasize in bytes (5.5T)
        11721045168     # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        729601          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA WDC WD60EFRX-68M    # Disk descr.
        WD-WX11D35C8VA3 # Disk ident.
        No              # TRIM/UNMAP support
        5700            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Here's the full stack trace:

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 277, in replace
    target.replace(newvdev)
  File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 277, in replace
    target.replace(newvdev)
  File "libzfs.pyx", line 2060, in libzfs.ZFSVdev.replace
libzfs.ZFSException: device is too small

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 94, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 279, in replace
    raise CallError(str(e), e.code)
middlewared.service_exception.CallError: [EZFS_BADDEV] device is too small
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run
    await self.future
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 403, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/replace_disk.py", line 122, in replace
    raise e
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/replace_disk.py", line 102, in replace
    await self.middleware.call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EZFS_BADDEV] device is too small

I've done this before; many times since I build this system 5 years ago (starting with 1TB drives). I've also tried replacing this drive with another 6TB drive of the same model, and that didn't work either.

Any help appreciated.

 

[Bill McCormick]

I am able to replace via CLI, so I think the issue is that I built the pool (way back when) with CLI, and something is now missing for full GUI operation.

First, I had to use the GUI to offline the disk because zpool list does not include to which device /dev/da3 belongs. Then, zpool list show which device is offline, and finally the the replace command:

Code:

root@freenas:/ # zpool list -v -P
NAME                                                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                         55.5G  10.2G  45.3G        -         -     3%    18%  1.00x    ONLINE  -
  mirror                                             55.5G  10.2G  45.3G        -         -     3%  18.5%      -  ONLINE
    /dev/ada1p2                                          -      -      -        -         -      -      -      -  ONLINE
    /dev/ada2p2                                          -      -      -        -         -      -      -      -  ONLINE
z1                                                   43.6T  37.5T  6.17T        -         -    20%    85%  1.00x  DEGRADED  /mnt
  raidz2                                             43.6T  37.5T  6.17T        -         -    20%  85.8%      -  DEGRADED
    /dev/gptid/9db7daf4-9d28-11e9-8139-d050994770a0      -      -      -        -         -      -      -      -  OFFLINE
    /dev/gptid/e35f7e4e-dd2c-11e7-ba57-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
    /dev/gptid/50cf163e-f2a8-11e9-ae6b-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
    /dev/gptid/ec4ddd74-ff4d-11e9-a4c2-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
    /dev/gptid/71effbab-02ae-11ea-b7fa-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
    /dev/gptid/00afdd83-ddbe-11e7-b6a0-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
    /dev/gptid/2cc5c277-9ded-11e9-ba2f-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
    /dev/gptid/199791a5-c6a2-11e9-ab3e-d050994770a0      -      -      -        -         -      -      -      -  ONLINE
root@freenas:/ # zpool replace z1 /dev/gptid/9db7daf4-9d28-11e9-8139-d050994770a0 /dev/ada0

root@freenas:/ # zpool status z1
  pool: z1
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr 25 11:46:47 2021
        1.92T scanned at 1.62G/s, 929G issued at 784M/s, 37.5T total
        114G resilvered, 2.42% done, 13:34:42 to go
config:

        NAME                                              STATE     READ WRITE CKSUM
        z1                                                DEGRADED     0     0     0
          raidz2-0                                        DEGRADED     0     0     0
            replacing-0                                   DEGRADED     0     0     0
              gptid/9db7daf4-9d28-11e9-8139-d050994770a0  OFFLINE      0     0     0
              ada0                                        ONLINE       0     0     0  (resilvering)
            gptid/e35f7e4e-dd2c-11e7-ba57-d050994770a0    ONLINE       0     0     0
            gptid/50cf163e-f2a8-11e9-ae6b-d050994770a0    ONLINE       0     0     0
            gptid/ec4ddd74-ff4d-11e9-a4c2-d050994770a0    ONLINE       0     0     0
            gptid/71effbab-02ae-11ea-b7fa-d050994770a0    ONLINE       0     0     0
            gptid/00afdd83-ddbe-11e7-b6a0-d050994770a0    ONLINE       0     0     0
            gptid/2cc5c277-9ded-11e9-ba2f-d050994770a0    ONLINE       0     0     0
            gptid/199791a5-c6a2-11e9-ab3e-d050994770a0    ONLINE       0     0     0

errors: No known data errors

I think all the previous GUI replacement I've done were replacing smaller disks with larger disk. This is probably the fist time I've tried to replace identical sized disks.

Is there something I can do to restore full GUI functionality (short of starting over again)? Seems like there ought to be some flag or other such trivial thing to set or reset somewhere.

 

[sretalla]

What's probably happening there is that via the GUI, you're going to be trying to allocate a 2GB partition to swap and a second one to data (which must be at least the same number of bits as the other pool members).

If the data partition isn't big enough, you can't complete the add/replace (seems this was your case).

So, if you want to use the GUI, you have 2 options:
1. Find a disk that actually has the right amount of space available (seems your disk is just slightly smaller than the others... not sure if this is a manufacturer issue or whatever, but I can't see other explanations).
2. remove the swap partition from System | Advanced (set it to 0... or you could try something less than 2... neither option recommended, but would equate to what you have now).

At a minimum, you should re-do the partitioning on your disk and add it using the gptid (you can get it from glabel status)

 

[Bill McCormick]

So first off, as show in the first

Code:

Code

segment my original post, the new drive is the same size as the existing drive. There are difference is Strip Size, Cylinders, and Heads; not sure if that's an issue or not.

Where is the 2GB swap partition coming from? I mean, where did you get the 2GB number from? My swap size (System->Advanced) is set to 16GB. Why wouldn't be trying to partition a 16GB swap?

Code:

freenas% glabel status
                                      Name  Status  Components
gptid/95e15525-503f-11ea-adb8-d050994770a0     N/A  ada1p1
gptid/95f4ca90-503f-11ea-adb8-d050994770a0     N/A  ada2p1
gptid/00afdd83-ddbe-11e7-b6a0-d050994770a0     N/A  da0p2
gptid/ec4ddd74-ff4d-11e9-a4c2-d050994770a0     N/A  da1p2
gptid/e35f7e4e-dd2c-11e7-ba57-d050994770a0     N/A  da2p2
gptid/9db7daf4-9d28-11e9-8139-d050994770a0     N/A  da3p2
gptid/50cf163e-f2a8-11e9-ae6b-d050994770a0     N/A  da4p2
gptid/2cc5c277-9ded-11e9-ba2f-d050994770a0     N/A  da5p2
gptid/71effbab-02ae-11ea-b7fa-d050994770a0     N/A  da6p2
gptid/199791a5-c6a2-11e9-ab3e-d050994770a0     N/A  da7p2
gptid/9da5ecf6-9d28-11e9-8139-d050994770a0     N/A  da3p1
gptid/00a43672-ddbe-11e7-b6a0-d050994770a0     N/A  da0p1

I'm not sure what I should be getting from that (glabel status).

The replace works on the command line; seems like a bug that it doesn't work in the GUI.

 

[sretalla]

where did you get the 2GB number from?

That's the default out of the box... you must have changed it. That number is for each disk, not the total amount of swap for the box.

My swap size (System->Advanced) is set to 16GB. Why wouldn't be trying to partition a 16GB swap?

It would.

The replace works on the command line; seems like a bug that it doesn't work in the GUI.

Like I said, the GUI is trying to create a (now you shared 16GB) partition in addition to the data partition... if the swap partition isn't 16GB on the other disks, this explains the whole story. If it is 16GB on the other disks, then your "same size" claim is perhaps true at the summary level you shared, but is not the exact same number of bits (or you wouldn't be getting the error).

It's not a bug.

As I already mentioned, you shouldn't leave it like it is and you should change to using the gptid of the disk (preferably by re-doing the replace in the GUI, presumably after setting the swap size to 2GB)

 

[Bill McCormick]

More weirdness after completing the CLI replace. The GUI disk's view says that the new/replaced disk is not part of the pool.

However zpool status says that it is:

Code:

root@freenas:/ # zpool status z1
  pool: z1
state: ONLINE
  scan: resilvered 4.58T in 1 days 01:08:36 with 0 errors on Mon Apr 26 12:55:23 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        z1                                              ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            da3                                         ONLINE       0     0     0
            gptid/e35f7e4e-dd2c-11e7-ba57-d050994770a0  ONLINE       0     0     0
            gptid/50cf163e-f2a8-11e9-ae6b-d050994770a0  ONLINE       0     0     0
            gptid/ec4ddd74-ff4d-11e9-a4c2-d050994770a0  ONLINE       0     0     0
            gptid/71effbab-02ae-11ea-b7fa-d050994770a0  ONLINE       0     0     0
            gptid/00afdd83-ddbe-11e7-b6a0-d050994770a0  ONLINE       0     0     0
            gptid/2cc5c277-9ded-11e9-ba2f-d050994770a0  ONLINE       0     0     0
            gptid/199791a5-c6a2-11e9-ab3e-d050994770a0  ONLINE       0     0     0

errors: No known data errors

Also, the Dashboard says the pool has 8 disks:

What's going on here?

 

[sretalla]

More weirdness after completing the CLI replace. The GUI disk's view says that the new/replaced disk is not part of the pool.

However zpool status says...


Also, the Dashboard says the pool has 8 disks

What's going on here?

Do you remember when I said:

you shouldn't leave it like it is and you should change to using the gptid of the disk (preferably by re-doing the replace in the GUI, presumably after setting the swap size to 2GB)

...

That.

 

[Bill McCormick]

Right. I started the command line replace before I had a chance to change the swap size. I let it finish, but then I had more concerns.

Anyway, I finally did put the swap size back to 2Gb, re-did the replace, and now all is well.

Thanks!

 

[Zappo]

For what it's worth. Had the same problem in 12.0-U7 just now where I needed to replace a broken disk with a new, identical, completely wiped disk.
Running with the default 2GiB swap size (checked under "System -> Advanced).
Replace with GUI = always "disk too small". Replaced it on the CLI with no problems at all: "zpool replace Pool01 /dev/multipath/disk9 /dev/multipath/disk9" and all is good again. GUI showed "all normal" after resilvering finished.

I can't remember ever having this problem in previous versions on this box (although the last broken disk is quite a while back).
I think the GUI needs some love here ;-)

 

[Alibek]

TrueNAS-SCALE-22.02.4

Code:

Error: concurrent.futures.process._RemoteTraceback:

"""

Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 350, in replace

    target.replace(newvdev)

  File "libzfs.pyx", line 411, in libzfs.ZFS.__exit__

  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 350, in replace

    target.replace(newvdev)

  File "libzfs.pyx", line 2132, in libzfs.ZFSVdev.replace

libzfs.ZFSException: device is too small



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker

    r = call_item.fn(*call_item.args, **call_item.kwargs)

  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 114, in main_worker

    res = MIDDLEWARE._run(*call_args)

  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 45, in _run

    return self._call(name, serviceobj, methodobj, args, job=job)

  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 39, in _call

    return methodobj(*params)

  File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 39, in _call

    return methodobj(*params)

  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf

    return func(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 352, in replace

    raise CallError(str(e), e.code)

middlewared.service_exception.CallError: [EZFS_BADDEV] device is too small

"""



The above exception was the direct cause of the following exception:



Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run

    await self.future

  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 446, in __run_body

    rv = await self.method(*([self] + args))

  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf

    return await func(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf

    res = await f(*args, **kwargs)

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/replace_disk.py", line 101, in replace

    await self.middleware.call(

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1345, in call

    return await self._call(

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1302, in _call

    return await self._call_worker(name, *prepared_call.args)

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in _call_worker

    return await self.run_in_proc(main_worker, name, args, job)

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1223, in run_in_proc

    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1206, in run_in_executor

    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))

middlewared.service_exception.CallError: [EZFS_BADDEV] device is too small

Why this operation create swap on target drive in replace command?
I don't use swap.

I will try create zvol for swap in 2G and swapon it. But when I again tried replace device - new swap automaticaly created and mounted again

Code:

root@truenas01[~]# zfs create -s -V 2G boot-pool/swap1

root@truenas01[~]# ls -l /dev/zvol/boot-pool/swap1
lrwxrwxrwx 1 root root 9 Oct 27 19:24 /dev/zvol/boot-pool/swap1 -> ../../zd0

root@truenas01[~]# mkswap /dev/zd0
Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=65113d8b-d382-4efd-bfcb-dd141782803d

root@truenas01[~]# free -m
               total        used        free      shared  buff/cache   available
Mem:          257064       48630      206836         662        1597      206246
Swap:              0           0           0

root@truenas01[~]# swapon /dev/zd0
root@truenas01[~]# free -m       
               total        used        free      shared  buff/cache   available
Mem:          257064       48629      206837         662        1597      206247
Swap:           2047           0        2047

root@truenas01[~]# wipefs -a /dev/sdcv 
/dev/sdcv: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdcv: 8 bytes were erased at offset 0x105efffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdcv: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdcv: calling ioctl to re-read partition table: Success

root@truenas01[~]# fdisk -l /dev/sdcv                       
Disk /dev/sdcv: 16.37 TiB, 18000207937536 bytes, 35156656128 sectors
Disk model: WUH721818ALE6L4
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

# USE GUI TO REPLACE DEVICE

root@truenas01[~]# fdisk -l /dev/sdcv
Disk /dev/sdcv: 16.37 TiB, 18000207937536 bytes, 35156656128 sectors
Disk model: WUH721818ALE6L4
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 7BB8B31E-3429-40A8-9B6E-5E2A06C231BE

Device       Start         End     Sectors  Size Type
/dev/sdcv1     128     4194304     4194177    2G Linux swap
/dev/sdcv2 4194432 35156656094 35152461663 16.4T Solaris /usr & Apple ZFS

root@truenas01[~]# free -m
               total        used        free      shared  buff/cache   available
Mem:          257064       48651      206815         662        1597      206225
Swap:           4095           0        4095