Pool is offline. i/o failure. don't use sata to pci 1x cards.. Drives are fine.

[danb35]

Are you sure there weren't any other devices that were part of Rabbit2? A SLOG, L2ARC, or special vdev? Because the zpool import output you shared indicates that all four disks were present and accounted for, and ZFS should be able to import that pool even if one (or two) of them were missing.

 

[CookieMonster1]

I'll send a picture.

 

[CookieMonster1]

Are you sure there weren't any other devices that were part of Rabbit2? A SLOG, L2ARC, or special vdev? Because the zpool import output you shared indicates that all four disks were present and accounted for, and ZFS should be able to import that pool even if one (or two) of them were missing.

I am 100% sure there is no other names that's why I did Rabbit0 Rabbit1 Rabbit2.

 

[CookieMonster1]

No judging, I was broke and this all I had to work with.. Not my best work.. I use these female connectors to male all the hard drives. I only plug into the main power supply with them. No daisy chains. 500 watt power supply. There are 3 chains for all the drives.
12 Drives total.

 

[CookieMonster1]

Are you sure there weren't any other devices that were part of Rabbit2? A SLOG, L2ARC, or special vdev? Because the zpool import output you shared indicates that all four disks were present and accounted for, and ZFS should be able to import that pool even if one (or two) of them were missing.

I am 100% sure there is no other names that's why I did Rabbit0 Rabbit1 Rabbit2.
What should I do? :(

 

[CookieMonster1]

So my two cents are:
1) You are not clear on describing the errors other than your pool is offline.
2) All the drive data you mentioned really is out of context. Slow down and if you desire to post the entire SMART output of all your drives, make sure you use CODE brackets, but I highly doubt your drives are the issue.
3) Have you rebooted?
4) Do you have a backup of your TrueNAS configuration file?
5) Do not upgrade your ZFS Pool !!! Not until after you recover your pool.
6) What is your boot drive? A Flash Drive, SSD, Hard Drive?
7) If you can, in the GUI, can you select the previous Boot Environment?
8) You could also try to recreate a new boot drive if you must but I'd save that for last. I dislike working with encrypted drives so you should ensure you get some good advice on this aspect.
9) Do you think you could provide some information about your system and it's setup/configuration per the forum rules? It will help us from just guessing. Computer (CPU/RAM/Boot Drive), Hard Drives, HBA if you have one.

We might have a bigger problem then we thought.

 

[danb35]

I am 100% sure there is no other names that's why I did Rabbit0 Rabbit1 Rabbit2.

You've posted this twice, but in neither case does it answer the question I asked:

Are you sure there weren't any other devices that were part of Rabbit2?

 

[CookieMonster1]

You've posted this twice, but in neither case does it answer the question I asked:

Yes I am 100% sure only hdd's for the pool. No SLOG, L2ARC, or special vdev. I was thinking about it but never wanted to do it. So no.

 

[HoneyBadger]

No judging, I was broke and this all I had to work with.. Not my best work.. I use these female connectors to male all the hard drives. I only plug into the main power supply with them. No daisy chains. 500 watt power supply. There are 3 chains for all the drives.
12 Drives total.

I can't tell what exact model of SATA cards those are, but they definitely aren't LSI HBAs, and they appear to be only PCIe x1. You're also chaining SATA power cables and there is definitely no support for staggered spinup on that platform so it's potentially close to having the inrush current of trying to spin up 12 HDD's causing your PSU some grief on startup.

I would recommend removing the 4x16T drives and putting them into your other machine (or conversely, removing all but the 4x16T from this one) and connect them directly to the motherboard SATA ports in either case. Try to import just this pool, and see what results.

 

[CookieMonster1]

I can't tell what exact model of SATA cards those are, but they definitely aren't LSI HBAs, and they appear to be only PCIe x1. You're also chaining SATA power cables and there is definitely no support for staggered spinup on that platform so it's potentially close to having the inrush current of trying to spin up 12 HDD's causing your PSU some grief on startup.

I would recommend removing the 4x16T drives and putting them into your other machine (or conversely, removing all but the 4x16T from this one) and connect them directly to the motherboard SATA ports in either case. Try to import just this pool, and see what results.

I will have to try it. Give me about 1 day, will give an update then.

 

[CookieMonster1]

Code:

 Error: concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 352, in import_pool
    self.logger.error(
  File "libzfs.pyx", line 392, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 346, in import_pool
    zfs.import_pool(found, new_name or found.name, options, any_host=any_host)
  File "libzfs.pyx", line 1151, in libzfs.ZFS.import_pool
  File "libzfs.pyx", line 1179, in libzfs.ZFS.__import_pool
libzfs.ZFSException: I/O error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
    await self.future
  File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1464, in import_pool
    await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1278, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1243, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1249, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1168, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1151, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('I/O error',)

 

[CookieMonster1]

 

[CookieMonster1]

Wow can this even happen. The drives are fine.

I installed a clean truenas on another 7010. Samething. I am so worried right now. I know I lost everything.. :(

I own a tool called Spinrite / paid. It repairs hdd's because its an i/o this might not be fixable? Going to get started on each drive will take weeks because they are 16TB..

Can someone explain why this i/o failure happened in the first place?? I thought the smart tests would show this..

This pool is only 10 months old with brand new drives.

 

[joeschmuck]

On the little bit of research I've done, it sounds like you have severe data corruption likely caused from a cable issue, but this is new stuff for me so don't bank on my information. I doubt the problem is the hard drives. I hope you are able to recover your data.

 

[CookieMonster1]

On the little bit of research I've done, it sounds like you have severe data corruption likely caused from a cable issue, but this is new stuff for me so don't bank on my information. I doubt the problem is the hard drives. I hope you are able to recover your data.

I shut down the storage server until I can come up with a plan for the other pools. The other 2 free computers right now are running spinrite. Takes 8 days because of the size of disks. Will be 16 days until it finishes it.. I have already had to restart it because it found noncorrectable error. If it does it again I am I guess I lost it all. If you have any other tools that are paid that might able to help please let me know. Thank You for all your help!

What do you think is best for the other 2 pools that are fine? Same thing might happen right?

When you say cable issues power or sata?

 

[Davvo]

What do you think is best for the other 2 pools that are fine? Same thing might happen right?

When you say cable issues power or sata?

If the cause is not found yes it can happen again.

Generally a bad or incorrectly seated one (both types, power and data, can suffer from this issue).

About paid tools, iirc there was a zfs recovery service but it costed a ton of money (in the order of 10000 dollars I think?) Plus if you were using encryption... Rip.

 

[CookieMonster1]

If the cause is not found yes it can happen again.

Generally a bad or incorrectly seated one (both can be affected).

About paid tools, iirc there was a zfs recovery service but it costed a ton of money (in the order of 10000 dollars I think?) Plus if you were using encryption... Rip.

My drives were encrypted, any other good tools for encryption?

 

[Davvo]

My drives were encrypted, any other good tools for encryption?

I fear you might have met a dead end, I'm sorry. I don't know of anything or anyone that can recover data from an encrypted and corrupted ZFS pool.
Also, if something/someone as such exists it should be at an enterprise-level pricing.

 

[CookieMonster1]

I fear you might have met a dead end, I'm sorry. I don't know of anything or anyone that can recover data from an encrypted and corrupted ZFS pool.

Funny thing is that the drives are fine, i/o failure so yep..

 

[Davvo]

Funny thing is that the drives are fine, i/o failure so yep..

ZFS as a system doesn't fail.
When ZFS fails, it does so spectacularly.

Edit: anyway, check the cables.