Error when trying to attach new SSD to BOOT mirror

[1koolcat]

I am receiving an error when attempting to introduce a new SSD drive to a boot mirror configuration.

I have been using FREENAS successfully for approx. four years and upgrade to TrueNAS 12 earlier this year (2021).

My NAS server platform is a DELL Precision T20 mini tower. It has 16GB of RAM and four WD RED 4TB drives for storage. Originally I was using two 16GB USB thumb drives for a mirrored boot, but decided to upgrade to a spinning platter HDD as suggested in the TRUENAS installation notes. I successfully migrated from the two USB drives to a WD Velociraptor 80GB HDD a few months ago. More recently I added a SAMSUNG EVO 500GB SSD as a boot mirror. I didn't have any issues or errors with this setup.

Today I removed the SAMSUNG 500GB SSD and replaced it with a SAMSUNG PRO 128GB SSD and was unable to attach it as a mirrored boot drive to the WD 80GB drive. The SAMSUNG PRO is recognized in the DRIVES list in the GUI, but I receive a cryptic error (cryptic to me!) when I try to attach it using the GUI interface for the BOOT POOL. I have reformatted a couple times (both as MBR and GPT) and I continue to get this error in the GUI:
Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 214, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 214, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 2030, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 94, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 977, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 217, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 367, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 398, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1219, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1146, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1120, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks

I have a background in supporting WINDOWS computers and I've played with various LINUX distros over the years, but I can't make sense of this message. I'm comfortable running commands in the SHELL if need be. Thanks for any assistance.

 

[Alecmascot]

From NAS-108568 :

"Looking into the code I see that "can only attach to mirrors and top-level disks" message is in fact reported on many other reasons when disk can't be added to the pool (ENOTSUP). In this particular case I bet the problem is in physical sector size of devices, reported as 4KB by the SSDs and not reported by the USB stick and probably defaulted to 512. You can not add into mirror with existing vdev with ashift=9 vdev with ashift=12. It should probably be possible to force ashift=9 for the added vdev, since 512e disks support smaller I/Os, but they will work suboptimal. So instead of the new SSDs addition into the mirror I would recommend you to backup config and reinstall from scratch onto the new devices. "

Can I suggest that you use CODE tags when you post tracebacks and command output to improve readability.

 

[1koolcat]

Thanks for your input. I will try your suggestion of backing up my config and re-installing to the new drives.

Sorry about the error code mess. I'm not familiar with CODE tags, but I'll research them so my post is more legible next time.

 

[Redcoat]

I'm not familiar with CODE tags, but I'll research them so my post is more legible next time.

Here's an easy way:

 

[Spearfoot]

My suggestion is: don't worry about having mirrored boot drives for TrueNAS.

I used to boot from a mirrored pair of SSDs on a Dell H200 in IR mode. But over time I realized this was overkill because it's very easy to re-install TrueNAS/FreeNAS as long as you back up your configuration regularly. So nowadays I just boot from a single, smallish (80GB, 100GB) Intel DC S3500-series SSD.

An alternative is to configure mirroring when you install TrueNAS, i.e., install a pair of small boot SSDs in your system and select both when you install TrueNAS. I think this is more-or-less what you're trying to achieve, "after the fact", and matching up an SSD with a HDD, which isn't optimal. It would be much easier to start clean: install matching SSDs; re-install TrueNAS; restore your configuration.

 

[1koolcat]

Thanks for the suggestions everyone. I backed up my configuration and installed TRUENAS from scratch on my SSD drives. Then restored my configuration. This turned out to be much easier than I expected.

Peter W.