Unable to Replace Failed Disk

[paul56]

Currently running Truenas SCALE 22.02-RC2 with 5x 4TB HDDs in ZRAID1.

On Monday one drive failed so the pool became degraded, so I ordered a new disk. Today the disk arrived so I took the old disk out and replaced with the new one and clicked "Replace" as per the Truenas website instructions.
Unfortunately I then got this error:

Error: concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 297, in replace target.replace(newvdev) File "libzfs.pyx", line 404, in libzfs.ZFS.__exit__ File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 297, in replace target.replace(newvdev) File "libzfs.pyx", line 2074, in libzfs.ZFSVdev.replace libzfs.ZFSException: device is too small During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 97, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1273, in nf return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 299, in replace raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADDEV] device is too small """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/job.py", line 409, in run await self.future File "/usr/lib/python3/dist-packages/middlewared/job.py", line 445, in __run_body rv = await self.method(*([self] + args)) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1269, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1137, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/replace_disk.py", line 137, in replace raise e File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/replace_disk.py", line 117, in replace await self.middleware.call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1324, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1289, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1295, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1218, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1192, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADDEV] device is too small


The failed drive was a WD40EFRX but those are no longer sold so the replacement is a WD40EFZX but before purchasing I checked online and apparently this is a direct replacement with no differences exept a bigger cache so should be the same size.

Can someone help me with this so I can restore redundancy by adding the new disk?

P.S. I have backups of the data elsewhere but want to avoid recreating the pool and restoring from backups if at all possible due to the time that will take and due to it meaning the system will need to go down for an extended period.

Thanks

 

[paul56]

I have always avoided using the CLI with Truenas as heard it can cause problems, but I couldn't see any way to get more information on the issue in this case so I ran fdisk --list to compare the old disks to the new one:


Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFZX-68A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 112358DD-F311-4A60-9399-122DC8A82465

Device Start End Sectors Size Type
/dev/sda1 128 4194304 4194177 2G Linux swap
/dev/sda2 4194432 7814037134 7809842703 3.6T Solaris /usr & Apple ZFS


Disk /dev/sdc: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68W
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8D3767E1-B2B0-CC4D-93E5-0D3F41E4A210

Device Start End Sectors Size Type
/dev/sdc1 2048 7814019071 7814017024 3.6T Solaris /usr & Apple ZFS
/dev/sdc9 7814019072 7814035455 16384 8M Solaris reserved 1


The first thing that I notice is that the disk size is exactly the same.
But then I see that Trunas GUI "Replace" action (presumably that as this is a brand new never touched disk) has partitioned the new one differently to the other drives and added a 2GB Linux swap partition for some reason.

Perhaps this is the issue, but I'm really struggling to understand why a swap partition would be placed on my data pool suddenly?
I'm thinking perhaps if I use fdisk to delete these partitions on the new disk and then let the GUI "Replace" again it might work?

But before I do anything by the CLI I'd like someone else's opinion as don't want to break things in the future.

 

[sretalla]

I don't know if some previous release has somehow avoided adding the standard swap partition, but it seems the current (and a few previous) versions have been doing it and I've seen a few folks fall into the same trap as you.

It can be worked around by setting the Swap partition size to 0:

Disable Swap Creation on New Pools

This advanced option was available in the 20.** release, but I am unable to locate in the 21.** release. Could someone please help with finding a system tunable that will disable swap creation on new pools? Thanks,

www.truenas.com

Then you can do the replace in the GUI.

 

[paul56]

I don't know if some previous release has somehow avoided ading the standard swap partition, but it seems the current (and a few previous) versions have been doing it and I've seen a few folks fall into the same trap as you.

It can be worked around by setting the Swap partition size to 0:

Disable Swap Creation on New Pools

This advanced option was available in the 20.** release, but I am unable to locate in the 21.** release. Could someone please help with finding a system tunable that will disable swap creation on new pools? Thanks,

www.truenas.com

Then you can do the replace in the GUI.

Thanks so much sretalla, I followed the link and ran "midclt call system.advanced.update '{"swapondrive": 0}'" and then Replaced again in the GUI and it's working now. Resilver is in progress! Perhaps it because I started off on the SCALE Beta?
Could I run into any other issues in future continuing to have no swap partition in my data pool though? What's the rational behind having a swap partition on a data pool anyway? Presumably there's a good reason as smarter people than me decided to add that by default at some point, but I've always only used swap on the OS/system drive/pool, not data one?

 

[dirtyfreebooter]

i too just ran into this issue. this pool is from my previous truenas core system, where i had swapondrive set to 0. trying to replace a disk now that is failing with the exact same model and getting this error.

swap on each drive is the so dumb. why they seem to now force it on SCALE i don't know why. i don't want swap on 16 drives, when i have 256gb of memory and super fast nvmes with swap already on them. i don't need TBs of swap on spindisks.. THANK YOU! lol

 

[Aken]

I've run into the same issue, only I'm not able to use the advice provided by sretalla. Trying to set swapondrive=0 using either midclt or the direct 'cli' results in the following error:

Code:

# midclt call system.advanced.update '{"swapondrive": 0}'
[EINVAL] advanced_settings_update.serialport: Serial port specified has not been identified by the system

I hoped I could just ignore this error, but the value doesn't stick.

 

[Aken]

Ah, it turns out I just had to set the serial port to 'false', that value was invalid and causing the config not to stick. Not sure why it set the serial port on in the first place.

(Side note: I tried editing my post but couldn't, I assume that's a new user restriction?)

 

[artlessknave]

the swap partition is used to have a placeholder of a few GB to prevent this error from occurring. by bypassing that, you bypass the TrueNAS attempt to help you by adding a bit of padding to disks for when they are slightly different in size.
ideally, what you should do is get a drive that's bigger, and let truenas manage your disks, the way it is designed to.

 

[Aken]

You don't need to allocate a swap partition to avoid this issue, OpenZFS by default will now allocate an 8MB partition at the end of the disk which avoids that issue entirely without having swap allocated on the pool.

 

[artlessknave]

openzfs and truenas are not exactly the same thing. TrueNAS is an appliance, and appliances often do more than the minimum to ensure functional operation, often despite users best attempts to break things by not following recomendations. in this case, TrueNAS assigns 2GB of swap on every disk by default. ignoring their recomendations tends to make people on the forums less inclined to help when things go wrong.

"By default, all data disks are created with the amount of swap specified. Changing the value does not affect the amount of swap on existing disks, only disks added after the change. Does not affect log or cache devices as they are created without swap. Setting to 0 disables swap creation completely. STRONGLY DISCOURAGED"

 

[Aken]

I know they're not the same thing. I'm saying that the version of OpenZFS currently used by TrueNAS will create a trailing 8MB partition if you do a zpool replace or zpool create using the entire disk (from the shell). This solves the concern regarding adding slightly smaller disks later on. This is imo a far more sensible approach than what TrueNAS does through the GUI, which is to create a 2GB swap partition and then use the second partition as the argument for zpool create/replace.

Swap on array drives was always a dumb idea, and was at least moderately defensible when everyone was booting FreeNAS/TrueNAS from USB sticks on systems with 4GB of RAM and then complaining their flash was dying due to excessive paging. It also as a side-effect solved the problem of being unable to replace disks with different sizes.

It's no longer defensible in 2022 when people are booting from SATA or NVMe SSDs and wasting hundreds of GBs of perfectly good, fast storage purely because TrueNAS insists on occupying the entire boot volume and not allowing it to be partitioned at install. (Ironically this would make more sense if it was booting from a USB.)

 

[jgreco]

Swap on array drives was always a dumb idea, and was at least moderately defensible when everyone was booting FreeNAS/TrueNAS from USB sticks on systems with 4GB of RAM and then complaining their flash was dying due to excessive paging.

The iX product known as FreeNAS *never* swapped to USB sticks, by the way, though I wouldn't be shocked if some users of its predecessor, prior to the iX acquisition more than a decade ago, did such silliness. Those of us who were here in the early days spent a lot of time fixing misconceptions and really bad ideas from the pre-iXsystems era.

In any case:

Not having swap on array drives is a tragic idea, and can result in really bad things happening.

There are conditions that can occur, when you're importing a pool, that can result in the ZFS import process using lots of memory, gobs of it in fact. If that happens, you will not be able to import the pool until you physically add more RAM to the system, or give it something to swap onto.

You should also trust that the developers have just a tiny bit of experience with all of this stuff, and are engineering for cases that you did not expect. You run the risk of bad and unexpected things happening when you decide to force the system to do things it doesn't want to do.

Further, placing swap on the SSD boot device is potentially foolish. If you take 2GB of space from a 1TB HDD, that's 0.5% of your HDD's space, which is nothing in the grand scheme of things. Since most HDD's are much larger, it's an unnoticeable blip. However, HDD has nearly unlimited endurance, while SSD has limited endurance, so if you were to develop a condition where there was swapping going on, swapping to SSD is literally eating away at the device that boots your NAS. To me, that seems like a moronic strategy. Better to eat a small fraction of a percent of the HDD capacity and not have to worry about it.

So the only thing dumber than swap on array drives is NOT having swap on array drives.

It also as a side-effect solved the problem of being unable to replace disks with different sizes.

The hard drive manufacturers worked out that problem many years ago; most drives of a given size in the past decade, even from different manufacturers, have an identical number of sectors.

 

[Aken]

There are conditions that can occur, when you're importing a pool, that can result in the ZFS import process using lots of memory, gobs of it in fact. If that happens, you will not be able to import the pool until you physically add more RAM to the system, or give it something to swap onto.

I've heard you mention this before, and I admittedly have only used comparatively small pools, but I've never heard of this being an issue elsewhere. Swap on the pool means you also have the issue of losing a disk and crashing the system, which was only solved recently by mirroring the swap partitions (which just mitigates the problem, it doesn't eliminate it). I'm very skeptical that swap-on-pool is a good trade-off for SME or home user cases.

Either way it's ix's choice how they prioritise these things.

Further, placing swap on the SSD boot device is potentially foolish. If you take 2GB of space from a 1TB HDD, that's 0.5% of your HDD's space, which is nothing in the grand scheme of things. Since most HDD's are much larger, it's an unnoticeable blip. However, HDD has nearly unlimited endurance, while SSD has limited endurance, so if you were to develop a condition where there was swapping going on, swapping to SSD is literally eating away at the device that boots your NAS.

This is a non-issue with modern SSDs. Perhaps 10 years ago this might have been a relevant point?

In a different thread you noted that "you're never supposed to run out of RAM", so this shouldn't be a problem, right?

 

[jgreco]

I've heard you mention this before, and I admittedly have only used comparatively small pools, but I've never heard of this being an issue elsewhere.

There are lots of things like this. You hopefully do not have personal experience with an automotive seat belt performing its job, yet do you need that experience to understand the value of not being thrown face-first through a pane of glass and into whatever catastrophe has caused your car to stop suddenly? Or, for a more ZFS-centric example, people obsess over SLOG all the time, but a large percentage of them do not actually understand that the ONLY time the SLOG is ever read from is during pool import. Many, even most (I'd guess), of them will ever experience a situation where critical data is actually saved due to the SLOG. Yet we still design for the issue.

wap on the pool means you also have the issue of losing a disk and crashing the system, which was only solved recently by mirroring the swap partitions (which just mitigates the problem, it doesn't eliminate it).

The mirroring thing has been going on since 2017, so, "for some value of recently" I guess. I'm happy to agree that it was a potential issue before then. However, even there, it would only be crashing userland processes, and a reboot would fix it, because a failed disk would not swapon.

I'm very skeptical that swap-on-pool is a good trade-off for SME or home user cases.

Okay, so, suggest an alternative that is suitable to an appliance device. We know the HDD's will be there. We don't know that the boot device will be an SSD, or that it will be large enough for swap. The choice has to be reasonably clear for beginners at install time, if a decision needs to be made at that time. Entertain us with a practical suggestion that lets a SME or home user who has decided to put way too much HDD on his way undersized-RAM NAS host, has encountered a power outage, and now has a pool import operation that is experiencing difficulty unwinding the ZIL.

Telling him that he's $#!+ outta luck and he's gotta Amazon Prime himself some more RAM is the only one that comes to mind.

This is a non-issue with modern SSDs. Perhaps 10 years ago this might have been a relevant point?

It's a relevant point TODAY. We've discussed QVO failures just recently in these forums somewhere. And there are still users who are using boot devices with crappy controllers that aren't "SSD-grade" for wear leveling etc. You can SAY it's a non-issue, but it's an issue.

In a different thread you noted that "you're never supposed to run out of RAM", so this shouldn't be a problem, right?

Yes. However, the thing that isn't supposed to happen... often does.

Bear in mind that I spent some time architecting UNIX (Solaris) OS to run on embedded medical devices back in the '90's, and one of the things that you don't want to happen is for your monitoring device to crash and fail to reboot quickly when the surgeon's got his hands around the patient's heart.

Lots of computer people optimistically write code and they're happy when it compiles and runs. But running isn't success. Success is when it does the correct thing under adverse conditions. Bounds checking, input validation, etc.

We know that users do not properly resource their ZFS filers. It seems prudent to plan for that, and for the adverse conditions that can hit even a properly sized ZFS filer. But that's just decades of my paranoia talking.

 

[Aken]

To be frank, I'm not invested enough in this issue to keep at it, and I can see where you're coming from. Though your attitude towards SSD wear seems inconsistent given that in other threads you've told people not to worry about it since it'll never swap anyway - or that the swap space is only important for corner-case pool imports (where wearing out the boot drive isn't going to happen unless they reboot once an hour).

Okay, so, suggest an alternative that is suitable to an appliance device. We know the HDD's will be there. We don't know that the boot device will be an SSD, or that it will be large enough for swap. The choice has to be reasonably clear for beginners at install time, if a decision needs to be made at that time. Entertain us with a practical suggestion that lets a SME or home user who has decided to put way too much HDD on his way undersized-RAM NAS host, has encountered a power outage, and now has a pool import operation that is experiencing difficulty unwinding the ZIL.

Telling him that he's $#!+ outta luck and he's gotta Amazon Prime himself some more RAM is the only one that comes to mind.

If users wanted a true storage appliance, they'd buy a Synology. It's not an appliance if you're installing your own OS. Besides, this is what minimum specs are for. When users try to run TrueNAS with less than 8GB of RAM, the forum (rightly) tells them they're below the minimum spec, and SOL.

If 2GB per disk in the pool is always likely to be sufficient (as the default would suggest), tell users that at install and let them put this much space on the boot drive or a separate SSD without ridiculous workarounds. SCALE already defaults to this if you're installing onto a large enough device (though it still sets swapondisk=2GB). Ditto for the application pool (from the other thread).

 

[jgreco]

Though your attitude towards SSD wear seems inconsistent given that in other threads you've told people not to worry about it since it'll never swap anyway - or that the swap space is only important for corner-case pool imports (where wearing out the boot drive isn't going to happen unless they reboot once an hour).

I've told people not to worry about it since it'll never swap anyways?

Perhaps you misunderstood something. I can see a case for saying something that could be mistaken for that on a really large RAM system, but historically, there've been daemons with memory leaks and occasional bouts of spike demand that tend to push things out to swap. This isn't really a bad thing.

Regardless, that's talking about swapping to HDD, which is generally considered to be unlimited endurance.

It's not an appliance if you're installing your own OS.

You're mistaken, but I understand your confusion. TrueNAS is indeed an appliance. When you order your TrueNAS system from iXsystems, it comes with TrueNAS preinstalled -- you don't even need to do the annoying Synology Assistant/DSM install thing that the Syno's do. But since you're probably using the free version of TrueNAS that iXsystems gives away for free, you may have failed to remember that little fact. Your converted PC is not really the target platform for TrueNAS. It just happens that you're allowed to load TrueNAS on your PC if you so desire. It is absolutely intended to be an appliance though, it is developed to be an appliance, it has official hardware that it runs on as an appliance.

When users try to run TrueNAS with less than 8GB of RAM,

That's, uh, (*cough*) 16GB now, per the download page. And you're currently talking with the person who originally defined the 8GB minimum RAM size. So I know a little bit about it.

If 2GB per disk in the pool is always likely to be sufficient (as the default would suggest),

It's not entirely clear. I suspect part of this is driven by the problem reports and system debugs actually received.

tell users that at install and let them put this much space on the boot drive or a separate SSD

Telling users stuff that they don't absolutely have to know at install time is onerous. Users want, thrive on, and often need simplicity. If they wanted to be making all the decisions, they'd probably just install FreeBSD and configure all their services by hand. That's what the appliance is for. It makes the complicated thing simple by making reasonable decisions and selections on their behalf.

without ridiculous workarounds.

There is no need for ridiculous workarounds. You can just let it do what it wants and things will work fine.

 

[Aken]

Hang on, users are already asked at install time whether or not they'd like to create a swap partition on the boot device (I assume equal to the amount of system memory, though that's a sample size of 1). There's no change to this process. If currently they 'need' both this much swap *and* 2GB per disk in their pool, then increase the prompted amount to match. Double it, for all I care. I just don't want 240 of my 256GB NVMe SSD going completely wasted (again, ditto for the application pool).

There's is a middle ground between Synology and full DIY, and TrueNAS doesn't always get the balance right.

 

[Aken]

By the way, the concern re: OOM issues when importing a pool, is that related to this Github issue? Or is this a different problem? Like I said, I'm relatively new to ZFS and keen to learn more about these sort of edge-cases, so I'd appreciate any more detail you can give.

 

[jgreco]

TrueNAS doesn't always get the balance right

And if you were the target market for TrueNAS, then perhaps that opinion would even be relevant. However, since you're putting this on hardware that you didn't buy from iXsystems, the fact that you designed your hardware in a manner that doesn't work out ideally is probably not causing anyone over at iXsystems to lose much sleep, or doing much to convince them that the ""balance"" is wrong. Because, once again, from the point of view of iXsystems,

There's is a middle ground between Synology and full DIY,

They aren't trying to cater to DIY or to some middle ground. They are writing an appliance. They're just nice enough to make it also usable on other hardware, and they give it away for free. Complaining about how their free gift to you isn't just the way you think it should be seems kinda rude.

By the way, the concern re: OOM issues when importing a pool, is that related to this Github issue? Or is this a different problem? Like I said, I'm relatively new to ZFS and keen to learn more about these sort of edge-cases, so I'd appreciate any more detail you can give.

They far predate 2016, so, who knows. It could well be related, but you'd have to ask the developers or look through Jira to see if you can identify relevant clues. A lot of stuff with ZFS doesn't always have pleasantly concrete answers, though.

As an example, the 8GB minimum RAM you mentioned earlier came from me. It didn't have any sort of super scientific equation that showed that when ARC plus kernel size plus process size did some specific thing then the system tanked. It was, instead, a lifetime of syseng experience, where I noticed LOTS of 4GB AMD APU users coming in with trashed unimportable pools, and some 6GB ones too, but no 8GB ones. And these weren't instant 4GB-kills-your-pool-the-moment-you-boot-it dealies, some of these people had been running a year or more. The pragmatic choice, when faced with a difficult to replicate issue like this, is to do the pragmatic thing and raise the minimum. So that's what I did.

We have absolutely seen discussions of people experiencing process growth (memory leaks) and other things that make swap a reasonable thing to have at least some of. We also know that there seem to be some ZFS import operations that take RAM scaled in some way to the size of the pool, and there's been some more specific discussions of that on the forum you're free to search for. This isn't really an interesting topic to me except insofar as I always want to be able to import my pools, but I'm pretty sure jpaetzel discussed it in some depth at one point.

 

[Aken]

They aren't trying to cater to DIY or to some middle ground. They are writing an appliance. They're just nice enough to make it also usable on other hardware, and they give it away for free. Complaining about how their free gift to you isn't just the way you think it should be seems kinda rude.

On the one hand - fair cop. On the other hand, giving something away for free doesn't invalidate opinions others may have about my design choices. It just means I can ignore them, because they're not paying me.

If your position is that posters on these forums should never provide negative feedback on decisions made by iX, then... ok?