TrueNAS System Dataset pool went offline

[harshadsane]

the configuration database and will be reset on reboot.

Code:

root@freenas[~]# zpool status -v
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:26 with 0 errors on Sat Aug 14 03:45:26 2021
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2      ONLINE       0     0     0

errors: No known data errors

I have a 4TB data drive that I dont see here. I have a SSD for dataset which is what I see below I think

root@freenas[~]# gpart show
=> 40 468862048 ada0 GPT (224G)
40 1024 1 freebsd-boot (512K)
1064 30441472 2 freebsd-zfs (15G)
30442536 438419552 - free - (209G)

root@freenas[~]#


The machine is booting up - I see two critical errors

error
CRITICAL
Pool NAS4TB state is OFFLINE: None
2021-08-15 21:51:05 (America/Los_Angeles)Dismiss

notifications_active
CRITICAL
Device: /dev/ada1, 16 Currently unreadable (pending) sectors.
2021-07-17 08:06:27 (America/Los_Angeles)Dismiss

Please help to recover ada1.

 

[harshadsane]

Sorry for the incomplete post above - Basically - my pool is showing as OFFLINE and I am unable to access my data. Let me know if there is a way of recovering what was on /dev/ada1. Thats all my data.

 

[Kris Moore]

Did your NAS4TB pool have any redundancy, or was it just a single drive?

 

[harshadsane]

No @Kris Moore - I live pretty close to iXsystems (in Santa Clara) - is there a service that I can avail that can look at the machine if I bring it over?

Also is there something I can run (diag) from the TrueNAS webinterface that can tell me whats hosed?

 

[Kris Moore]

No redundancy isn't exactly best practice here, since this is exactly the type of situation ZFS redundancy is supposed to prevent :/

You can try creating a debug file (System -> Advanced -> Save Debug) and looking through the logs, but I'm thinking the disk has had too many fatal errors to import the pool.

If you want to send the log files here of output from running "dmesg" and "zpool import" from the CLI that could be helpful as well.

 

[harshadsane]

@Kris Moore - thank you! Luckily - after being shutdown for a day - when I booted up the machine again - the drive came back up. I am backing up all the data. I have another 4TB drive that I will add. Please point me to setting up ZFS redundancy properly.

 

[harshadsane]

zpool import gives:

Code:

root@freenas[~]# zpool import
no pools available to import

And dmesg output is attached. @Kris Moore

 

[john60]

I think I have the same problem.

root@truenas[~]# zpool status -v
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:36 with 0 errors on Fri Jul 29 03:45:39 2022
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sda ONLINE 0 0 0

errors: No known data errors

The gui at /ui/storage report

jjg-14TB-z2(System Dataset Pool)OFFLINE

I would appreciate suggestions on what I need to do.

 

[john60]

Got a lot of alerts.
What do the following mean?

Reporting database used size 3.59 GiB is larger than 1.01 GiB.​

2022-07-30 11:39:43 (America/Los_Angeles)

and

Pool jjg-14TB-z2 is offline, not running scrub.
2022-07-31 00:00:01 (America/Los_Angeles)

and

Failed to configure kubernetes cluster for Applications: Missing "jjg-14TB-z2/ix-applications/k3s, jjg-14TB-z2/ix-applications/releases, jjg-14TB-z2/ix-applications/docker" dataset(s) required for starting kubernetes.​

2022-07-30 11:39:45 (America/Los_Angeles)

 

[sarieri]

Did you find a way to resolve this. I met similar problem as yours

 

[john60]

I can avoid the issue.

My problem was caused by running very long /data protection/rsync. see NAS-117410
After I initiate a long GUI RSYNC, the system gets progressively worst over a couple of hours with finally the pools go offline.
100% reproducible. A restart fixes the issue.

I turn RSYC off in the GUI, run RSYNC the command line, and the system is stable.
Problem avoided.