Missing zpool

[fazzical]

Hello all,

I have recently switch over from Unraid to Truenas Scale, and after a brief learning curve, got everthing up and running.

Simple Server, 15 disks. NVME for Boot/OS, 4 x 6TB 7200 RAID-Z1 (Apollo), 10 x 800GB SSD RAID-Z3 (Ares). Apps located on SSD Pool.

I got all my data restored to the Apollo and created all my apps on the Ares and everything was running just fine. It then looks like I had something glitch causing failure and when the server rebooted it is unable to bring Ares pool back online.

In the console I have
View attachment 54488

View attachment 54487

I have tried zpool import Ares and I get:
cannot import 'Ares': I/O error
Destroy and re-create the pool from
a backup source.

I've tried
zpool import -F Ares
zpool import -FfmX Ares
both fail.

I've also tried
zpool status Ares
zpool destroy Ares
zpool clear Ares
and I get "cannot open 'Ares': no such pool"

I even tried

zpool import -d /dev/disk/by-partuuid/0dfdb7df-b7ff-49ad-85cc-3f71143ca1b8

pool: Ares
id: 15639148256716711440
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

Ares ONLINE
raidz3-0 ONLINE
0dfdb7df-b7ff-49ad-85cc-3f71143ca1b8 ONLINE
d0681454-5640-4e38-8b83-044fb7cf19f5 ONLINE
e283ed89-3a6e-490c-be0f-12adfb0a553a ONLINE
e2494bb2-8926-4e5d-ab22-bbfd0b57d9f0 ONLINE
547d276e-0d5a-4eae-bc34-a1d17d5b7bc3 ONLINE
641c9a14-dcae-492f-a490-d6d875f4c0de ONLINE
b2237eb9-fe0c-4dd9-b6b7-9399e6deb1ef ONLINE
79e711c3-6892-4e82-9390-67670360e060 ONLINE
4e198db8-b75b-49f7-a7bf-841c75ab06f2 ONLINE
18127aa4-7efc-4252-b639-57a54af1bc2d ONLINE

and

zdb -e -C Ares

MOS Configuration:
version: 5000
name: 'Ares'
state: 0
txg: 4
pool_guid: 15639148256716711440
errata: 0
hostid: 736759888
hostname: 'zeus.local'
com.delphix:has_per_vdev_zaps
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 15639148256716711440
create_txg: 4
children[0]:
type: 'raidz'
id: 0
guid: 804297253874833685
nparity: 3
metaslab_array: 256
metaslab_shift: 34
ashift: 12
asize: 7980137840640
is_log: 0
create_txg: 4
com.delphix:vdev_zap_top: 129
children[0]:
type: 'disk'
id: 0
guid: 11153446576616311539
path: '/dev/disk/by-partuuid/0dfdb7df-b7ff-49ad-85cc-3f71143ca1b8'
create_txg: 4
com.delphix:vdev_zap_leaf: 130
children[1]:
type: 'disk'
id: 1
guid: 8322636522202523637
path: '/dev/disk/by-partuuid/d0681454-5640-4e38-8b83-044fb7cf19f5'
create_txg: 4
com.delphix:vdev_zap_leaf: 131
children[2]:
type: 'disk'
id: 2
guid: 8748942591705755313
path: '/dev/disk/by-partuuid/e283ed89-3a6e-490c-be0f-12adfb0a553a'
create_txg: 4
com.delphix:vdev_zap_leaf: 132
children[3]:
type: 'disk'
id: 3
guid: 3068813605273205228
path: '/dev/disk/by-partuuid/e2494bb2-8926-4e5d-ab22-bbfd0b57d9f0'
create_txg: 4
com.delphix:vdev_zap_leaf: 133
children[4]:
type: 'disk'
id: 4
guid: 10904326080831814824
path: '/dev/disk/by-partuuid/547d276e-0d5a-4eae-bc34-a1d17d5b7bc3'
create_txg: 4
com.delphix:vdev_zap_leaf: 134
children[5]:
type: 'disk'
id: 5
guid: 5878246634067620028
path: '/dev/disk/by-partuuid/641c9a14-dcae-492f-a490-d6d875f4c0de'
create_txg: 4
com.delphix:vdev_zap_leaf: 135
children[6]:
type: 'disk'
id: 6
guid: 5648342104744974838
path: '/dev/disk/by-partuuid/b2237eb9-fe0c-4dd9-b6b7-9399e6deb1ef'
create_txg: 4
com.delphix:vdev_zap_leaf: 136
children[7]:
type: 'disk'
id: 7
guid: 9874749319818608655
path: '/dev/disk/by-partuuid/79e711c3-6892-4e82-9390-67670360e060'
create_txg: 4
com.delphix:vdev_zap_leaf: 137
children[8]:
type: 'disk'
id: 8
guid: 8415984191253077936
path: '/dev/disk/by-partuuid/4e198db8-b75b-49f7-a7bf-841c75ab06f2'
create_txg: 4
com.delphix:vdev_zap_leaf: 138
children[9]:
type: 'disk'
id: 9
guid: 14660595219738274327
path: '/dev/disk/by-partuuid/18127aa4-7efc-4252-b639-57a54af1bc2d'
create_txg: 4
com.delphix:vdev_zap_leaf: 139
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data

I am a real loss with this. I know there is something simple I am missing. I really dont want to have to recreate all my Apps again, but if thats the only way then any pointers would be good.

Thanks for reading.

 

[jgreco]

Welcome to the forums!

Sorry to hear you're having problems.

Please help us help you by posting detailed information about your system, especially including, but not limited to, detailed information about your storage controller setup. The Forum Rules, helpfully linked at the top of every page in red, includes some guidance as to how to build a problem report and hardware description that is more likely to result in some useful responses.

Particularly apropos to this discussion would be an explanation of what storage controller you're using, and whether you've perhaps errantly used a RAID controller or non-LSI HBA, which often end in misadventures resembling what you're presenting here. See

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[fazzical]

Hi, Thanks for the reply.

MB: MSI X470 GAMING PLUS MAX
Processor: AMD Ryzen 5 1600
RAM: 2 x 8GB DDR4 nonECC
Storage Controllers: LSI SAS2008 running the correct FW & onboard AMD SATA (AHCI)
10 x Intel DC S3500 800GB SSD
4 x Toshiba X300 HDWE160 6TB 7200 HDD
1 x Samsung SM951 NVMe 512GB
Emulex OneConnect 10GbE NIC
GeForce GTX970

The 4 HDD are connected to one channel of the SAS2008 as are 4 of the SSD. The other 6 SSD are connected to the 6 onboard SATA.

 

[fazzical]

While waiting for a reply, I though I would do a MEMTEST check. Turns out I have a (very) bad stick of RAM. Time to invest in some ECC and rebuild my array (again). Atleast I still have all my backups on other disks.

 

[NugentS]

I would say that the 2008 is not 100% ideal for an SSD array. Not that it would have caused this issue - but it does lack somewhat in IO performance when dealing with SSD's as I understand it.

I honestly don't know how much of an effect (if any) this will have on your Z3 array as its partially on the ACHI and partially on the HBA so would only effect the HBA part