SOLVED Can't import full, encrypted pool. vdev state changed and my physical pool disks are really, really hard at work

[devnullius]

top -m io mostly shows zfs & uwsgi on top. I also see python3.6 pop up once in a while and syslog-ng with nginx and smbd. I didn't find another way then top -m io to see what's active.

No plugins are enabled, so those are out as a possible culprit too.

Code:

root@Freenas:~ # top -m io
last pid: 65596;  load averages:  0.28,  0.25,  0.28                                                         up 1+17:02:48  14:52:07
75 processes:  1 running, 70 sleeping, 4 zombie
CPU:  0.1% user,  0.0% nice,  0.6% system,  0.0% interrupt, 99.3% idle
Mem: 218M Active, 910M Inact, 3756M Wired, 34G Free
ARC: 525M Total, 19M MFU, 432M MRU, 64K Anon, 3803K Header, 70M Other
     126M Compressed, 338M Uncompressed, 2.68:1 Ratio
Swap: 6144M Total, 6144M Free

  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
65582 root          412      0      0      0      0      0   0.00% zpool
6311 root           47      1      0      0      0      0   0.00% uwsgi
4190 root          355      0      0      0      0      0   0.00% consul
  230 root           18      0      0      0      0      0   0.00% python3.6
88232 www            75      0      0      0      0      0   0.00% nginx
2533 root           98      0      0      0      0      0   0.00% syslog-ng
53905 root            2      0      0      0      0      0   0.00% top
2264 root            9      0      0      0      0      0   0.00% devd
3271 root            3      0      0      0      0      0   0.00% nmbd
2955 root            7      0      0      0      0      0   0.00% powerd
5819 root            2      0      0      0      0      0   0.00% python2.7
30240 root            2      0      0      0      0      0   0.00% sshd
2952 root            2      0      0      0      0      0   0.00% ntpd
3280 root            1      0      0      0      0      0   0.00% winbindd
4163 root            1      0      0      0      0      0   0.00% uwsgi
6532 root            0      0      0      0      0      0   0.00% zfsd
4267 root            0      0      0      0      0      0   0.00% collectd
60374 root            0      0      0      0      0      0   0.00% smbd
4152 root            0      0      0      0      0      0   0.00% python3.6
3275 root            0      0      0      0      0      0   0.00% smbd
4183 root            0      0      0      0      0      0   0.00% consul-alerts
7186 root            0      0      0      0      0      0   0.00% consul
7187 root            0      0      0      0      0      0   0.00% consul
65526 root            0      0      0      0      0      0   0.00% python3.6
3821 root            0      0      0      0      0      0   0.00% smartd
7109 root            0      0      0      0      0      0   0.00% python3.6

Code:

root@Freenas:~ # zpool iostat -v
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
VOLU10TB                                3.34T  5.73T      0      0    249  3.45K
  gptid/6baca59b-0553-11e8-a1db-0025901159d4  3.34T  5.73T      0      0    249  3.45K
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                             756M  13.6G      0      0    724    122
  da8p2                                  756M  13.6G      0      0    724    122
--------------------------------------  -----  -----  -----  -----  -----  -----

 

[PhiloEpisteme]

SMART tests?

 

[devnullius]

smartctl -a <device>

What do I look for in the output? As far as I can see that gives a very nice detailed overview of the SMART status, but that's about it, no? :)

Post-edit: OOH, you probably mean if the SMART logs give an ALL OK for each of my 8 disks, correct?

 

[PhiloEpisteme]

What do I look for in the output?

I would suggest you either Google how to determine if a SMART test is running, or post the output of the command you ran on the drives which are active. Both would be better as it'll help folks help you faster and in your searching you may find something we hadn't thought of. :)

 

[devnullius]

Fair enough - I'll edit this post later (I'll be back, as they say)

Edit: I found the SMART tests under TASKS in the GUI (my bad!) I only have a short one scheduled, for every 11 hours. Could a short self-test run that long? Next step: investigate if I can shut any self-tests off.

I'll be back :)


I'm back: https://linustechtips.com/main/topic/942073-how-do-you-run-a-smart-test-with-freenas/
As far as I can tell, the SMART self test is ran by smartctl. When I ask for any running processes with that name, it turns up empty.

Code:

root@Freenas:~ # ps -aux | grep -i smartctl
root       70997    0.0  0.0   6696   2612  0  S+   15:13       0:00.00 grep -i smartctl

 

[PhiloEpisteme]

Post-edit: OOH, you probably mean if the SMART logs give an ALL OK for each of my 8 disks, correct?

No, I mean to determine if SMART tests are running on your drives. It seems unlikely given how long the drives have been active for, but possible.

 

[devnullius]

It doesn't seem to be the case! See my last reply above.

With any other computer, I'd just reboot and start fresh. With FreeNAS… sigh : )

 

[devnullius]

I'm gonna download FreeNAS 11.1-U6 and start over; I'll give it a reboot this evening to see if anyone has any ideas as to what to do :|

Philo! Can't say thank you enough! : )

 

[Redcoat]

if anyone has any ideas as to what to do

Well, OP, I did give you what I thought was the safest suggestion earlier - replace all your discs with larger ones and expand your pool. That may still be your best option - hopefully nothing done since you discovered your problem has munged your data ...

 

[devnullius]

But how can I expand a pool that I can't open...? Let alone the costs involved (8x 3TB disks). Let's first re-start the initial dance that started this all. See if it goes different this time. I can't imagine anything got corrupted; at least not due to something we did.

 

[Redcoat]

Are you sure it ever decrypted given your shortage of space? Was that not the initial issue?

 

[PhiloEpisteme]

Well, OP, I did give you what I thought was the safest suggestion earlier - replace all your discs with larger ones and expand your pool. That may still be your best option - hopefully nothing done since you discovered your problem has munged your data ...

I think that is a reasonable option so long as budget allows. To be clear though, decrypting those drives will be required, no? And then imported as well so as to perform the disk replacement and resilvering process.

 

[Redcoat]

To be clear though, decrypting those drives will be required, no? And then imported as well so as to perform the disk replacement and resilvering process.

No, I don't believe so. The premise is that the current shortage of space is inhibiting the decryption.

 

[devnullius]

No worries: I'll be back :) I just have no clue as to what it's doing atm. I hope a reboot and/or fresh install will enable me to decrypt the disks again (that part was successful by doing it for each disk separately: some long string names were involved :) )

xx

 

[devnullius]

I think that is a reasonable option so long as budget allows. To be clear though, decrypting those drives will be required, no? And then imported as well so as to perform the disk replacement and resilvering process.

That's what I thought too...

 

[Redcoat]

That's what I thought too...

I believe the process would be - increase all drive sizes to make more space - then decrypt - then if pool not already imported, import pool.

 

[devnullius]

But in the GUI I could not select the pool te expand it's size: it simply wasn't shown as a valid option (presumably because it was either encrypted or locked)… :|

 

[PhiloEpisteme]

I believe the process would be - increase all drive sizes to make more space - then decrypt - then if pool not already imported, import pool.

By what method can OP replace the drives with larger drives to decrypt? Are you proposing using dd to copy the drives from one to the other?

 

[Redcoat]

No, a full set of larger drives - the https://ixsystems.com/documentation/freenas/11.2/storage.html#disks 9.5.2 method to expand a pool. I am sure that I've seen a post here on this forum of someone doing just this in the OP's situation, with success, but I have not been able to find it.

 

[PhiloEpisteme]

No, a full set of larger drives - the https://ixsystems.com/documentation/freenas/11.2/storage.html#disks 9.5.2 method to expand a pool. I am sure that I've seen a post here on this forum of someone doing just this in the OP's situation, with success, but I have not been able to find it.

Ah yeah, I'm aware of replacing each drive one by one with larger disks and letting them resilver. What I haven't seen is resilvering encrypted disks in a pool that is not imported. If you find something I'd definitely be interested in checking it out. :)