how do i replace a failed drive in TNC (latest stable)

[jcizzo]

upon inserting another drive, it still has it offline but there seems to be small amounts of disk activity..

does it start the rebuild process automatically? and ix just never updated the documentation?

 

[jcizzo]

of COURSE i looked through the documentation, but TNC refuses to behave...
i went Pools-> clicked the gear ->status, clicked the 3 ... next to the failed drive, offlined it, inserted the new drive (after verifying it was completely blank), clicked replace, chose the drive from the memberdisk dropdown (with Force checked), and...
what gives? this shouldn't be this difficult


Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
await self.future
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 981, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool_/replace_disk.py", line 78, in replace
await self.middleware.call('pool.format_disks', job, disks)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1283, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1240, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool_/format_disks.py", line 28, in format_disks
await asyncio_map(format_disk, disks.items(), limit=16)
File "/usr/local/lib/python3.9/site-packages/middlewared/utils/asyncio_.py", line 16, in asyncio_map
return await asyncio.gather(*futures)
File "/usr/local/lib/python3.9/site-packages/middlewared/utils/asyncio_.py", line 13, in func
return await real_func(arg)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool_/format_disks.py", line 24, in format_disk
await self.middleware.call('disk.format', disk, swapgb if config['create_swap'] else 0, False)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1283, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1251, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1156, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/disk_/format.py", line 21, in format
raise CallError(f'Failed to wipe disk {disk}: {job.error}')
middlewared.service_exception.CallError: [EFAULT] Failed to wipe disk da4: [Errno 6] Device not configured

 

[Davvo]

How are you connecting the drive?
Output of camcontrol devlist please.
Are you able to smartctl -a /dev/da4?
Did you try rebooting power cycling the system and trying a second time?

 

[Ericloewe]

While you're at it, a full description of your storage setup would be useful to understand what's going on.

 

[jcizzo]

How are you connecting the drive?
Output of camcontrol devlist please.
Are you able to smartctl -a /dev/da4?
Did you try rebooting power cycling the system and trying a second time?

the drives are in one of those trayless 5 x.3.5"HDD drive bays/caddies that occupies 3x 5.25 bays on the front of an atx chassis. lift the lever and they pop out, it's all supposed to be hot-swappable.

the caddy has 5 sata connectors that attach to an LSI 9200 series hba card in IT mode.

i rebooted/powercycled the system a few times but no dice. i verified that the drives were clear (no partitions) from the company i bought them from. they are the same model and size as the others that are in there (hgst 4TB). i bought 2 and after the 1st didn't work i tried the 2nd, thinking maybe i just received a bad drive, but same thing happened. fortunately its set up as raidz2, however the drive died about a little over a week ago so since then its been humming along on 4 drives.

i'll run that command when i get home, i figured i'd just reply now to give you folks something to ponder.

 

[jcizzo]

While you're at it, a full description of your storage setup would be useful to understand what's going on.

full description of setup: it is all about as outta the box and simple as TNC can be.

supermicro x11ssh-f motherboard flashed with latest stable bios
2 x ssd's for TNC (mirrored, default install) - uses sata connectors on motherboard
2 x ssd's for VERY important data (legal docs and smaller things that aren't frequently accessed), - uses sata connectors on motherboard.
32GB ram (ecc).
xeon E3-1240L v5 cpu.

LSI 9211-8i HBA card in IT mode cooled with a dedicated fan.

and the aforementioned 5 x 3.5" HDD drive bay.

the drive pool layouts/configs are as simple and straightforward as can be.

thanks!

 

[Evertb1]

If I were you I would pull out that caddy for the time being and connect your drives directly to the HBA and the power. That way you can determine if the caddy plays a role in your difficulties.

 

[jcizzo]

How are you connecting the drive?
Output of camcontrol devlist please.
Are you able to smartctl -a /dev/da4?
Did you try rebooting power cycling the system and trying a second time?

"camcontrol devlist" results in the following:


as for the other command "smartctl -a /dev/da4"
it ran and it printed a whole ton of stuff.. however i'm guessing you're most interested in the overall health, in which it states at the top:
"SMART overall-health self-assessment teset result passed"

previous self-test yadda yadda completely without error...

let me know if that satisfies your query

thanks!!

 

[Ericloewe]

it ran and it printed a whole ton of stuff.. however i'm guessing you're most interested in the overall health, in which it states at the top:

Actually no, it's the whole bunch of data that's interesting. That headline never says anything other than Healthy unless the disk is a paperweight.

 

[jcizzo]

If I were you I would pull out that caddy for the time being and connect your drives directly to the HBA and the power. That way you can determine if the caddy plays a role in your difficulties.

i'll be damned....

i followed your suggestion and it seems as though that was it...!!?? all drives are green and it's coming up as 'resilvering'.

if that's how it all works, i have to wonder if the drive was truly dead or is it the drive enclosure that's failing and causing the issues..

Thanks!!

i'll keep you posted when it's done!