Citadel - Build Plan and Log

[ctag]

11.3 has email alerts now when rsync tasks fail, which is an awesome and welcome feature. So I discovered that my rsync tasks are returning [EFAULT] rsync command returned 24. Check logs for further information.

Code 24 corresponds to vanished files, which is expected since I'm backing up user home directories, and cache files change. I added some of the obvious cache directories to an exclude list, but 1) certain directories seem to still be pulled (but not all of them?!) and 2) I'd like to not get an email for vanished files, but would like an email for other errors.

 

[ctag]

Rsync finally appears to be working. The issue with exclusion is that full paths are ignored, excluded paths have to match from the root of the rsync operation. That's pretty garbage, but at least now it works.

 

[ctag]

It has now been several hours since restarting my FreeNAS machine and unlocking the main pool. My auto-start enabled jails still have not started, so I'm launching them manually.

 

[ctag]

Tonight I'm playing around with installing a linux VM and can't get it working.

I create a UEFI VM with debian 10 install ISO and start it. And then the VNC image is junk and the serial connection doesn't seem to work...

Seems to be the same issue as here: https://www.ixsystems.com/community/threads/glitch-vnc-bhyve-11-1-u1.61372/

 

[marunjar]

Using debian 10 in a bhyve vm is a little bit tricky ;)
You have to set the resolution of the VNC device of your VM to 800x600

And after installing debian you have to copy debians grubx64.efi file to a location where bhyve expects it to be.
Else you won't be able to stop and start your vm:

Code:

mkdir -p /boot/efi/EFI/BOOT
cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi

And to be safe you have to copy this everytime grubx64.efi changes...

 

[ctag]

Using debian 10 in a bhyve vm is a little bit tricky ;)
You have to set the resolution of the VNC device of your VM to 800x600

And after installing debian you have to copy debians grubx64.efi file to a location where bhyve expects it to be.
Else you won't be able to stop and start your vm:

Code:

mkdir -p /boot/efi/EFI/BOOT
cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi

And to be safe you have to copy this everytime grubx64.efi changes...

Thanks marunjar!

I made a thread detailing the trouble: https://www.ixsystems.com/community/threads/display-scrambled-vnc-to-bhyve-linux-vm.83332/
And eventually gave up and found Ubuntu to work pretty well instead. I'm now running Folding@Home on my FreeNAS box :)

 

[ctag]

Just want to make sure I have this saved somewhere:

iocage helper thread

Hi, since I love jails and already love the new iocage I just wanted to share some hints since the UI doesn't really help me. I expect a new jail to be there with a name and a release set, nothing else. set jail hostname iocage set host_hostname="yourhostname" enable DHCP iocage set vnet=on...

www.ixsystems.com

Set up SMART Reporting via email

This is a simple way to get SMART monitoring to report daily the status of your hard drives. The purpose of this is to have all your SMART enabled drives report how they are doing daily. I wanted to see how many times the drives spun up and down, see if there were any flaky transmission...

www.ixsystems.com

Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config

I wanted to share my scripts because I think I'm not the only one who wants to monitor his FreeNAS server a bit more closely and in a more personalized way than the automatic nightly emails would do. Please note that you're the only one responsible of what you do in the CLI so don't blame me if...

www.ixsystems.com

Chapter 14. Configuration, Services, Logging and Power Management

This chapter explains much of the FreeBSD configuration files, how to enable or disable a service, how to configure the logging system and the power management area.

www.freebsd.org

 

[ctag]

Upgraded to 11.3-U2 and now I can't log into my Bhyve Ubuntu VM.

 

[ctag]

Woke up to some alert emails:

FreeNAS @ bns-citadel.local

New alert:
* Boot pool status is DEGRADED: None.

The following alert has been cleared:
* Boot pool status is DEGRADED: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state..

Current alerts:
* Scrub of pool 'freenas-boot' started.
* Boot pool status is DEGRADED: None.

I'm unable to reach the Web UI.

 

[ctag]

I can't get the web UI to load, and I also can't log in over SSH. I get the MOTD but no shell.

Should I attempt a reboot via ACPI?

 

[ctag]

My jails appear to still be working. Odd.

 

[ctag]

ACPI power off did not work, so I held the button.

System came back up and confirmed that there is a problem:

...
New alerts:
* Boot pool status is ONLINE: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state..

FreeNAS @ bns-citadel.local

New alert:
* Boot pool status is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected..

The following alert has been cleared:
* Boot pool status is ONLINE: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state..

FreeNAS @

New alerts:
* bns-citadel.local had an unscheduled system reboot.
The operating system successfully came back online at Sat Apr 18 11:04:30 2020.

I guess I should attempt replacing the flash drive, but so far I cannot find where it tells me which one needs replacement...

 

[ctag]

I read this thread:https://www.ixsystems.com/community...-experienced-an-unre-mirrored-boot-usb.57118/

And have the output of the commands:

bns-citadel# comcontrol devlist
comcontrol: couldn't open file devlist: No such file or directory
bns-citadel# camcontrol devlist
<AHCI SGPIO Enclosure 2.00 0001> at scbus4 target 0 lun 0 (pass0,ses0)
<ATA WDC WD80EFAX-68L 0A83> at scbus5 target 0 lun 0 (pass1,da0)
<ATA WDC WD80EFAX-68L 0A83> at scbus5 target 1 lun 0 (pass2,da1)
<ATA WDC WD80EFAX-68L 0A83> at scbus5 target 2 lun 0 (pass3,da2)
<ATA WDC WD80EFAX-68L 0A83> at scbus5 target 3 lun 0 (pass4,da3)
<ATA WDC WD80EFAX-68L 0A83> at scbus5 target 4 lun 0 (pass5,da4)
<ATA WDC WD80EFAX-68L 0A83> at scbus5 target 5 lun 0 (pass6,da5)
<SanDisk Cruzer Fit 1.00> at scbus7 target 0 lun 0 (pass7,da6)
<SanDisk Cruzer Fit 1.00> at scbus8 target 0 lun 0 (pass8,da7)
bns-citadel# glabel status
Name Status Components
gptid/0f763719-0e81-11e8-b9d2-f04da2301444 N/A da0p2
gptid/10119b97-0e81-11e8-b9d2-f04da2301444 N/A da1p2
gptid/10a5f4d8-0e81-11e8-b9d2-f04da2301444 N/A da2p2
gptid/1145e65f-0e81-11e8-b9d2-f04da2301444 N/A da3p2
gptid/11f0d1c6-0e81-11e8-b9d2-f04da2301444 N/A da4p2
gptid/1294e8b1-0e81-11e8-b9d2-f04da2301444 N/A da5p2
gptid/8840add5-0e79-11e8-a8ef-f04da2301444 N/A da6p1
gptid/881f5230-0e79-11e8-a8ef-f04da2301444 N/A da7p1
bns-citadel# zpool status
pool: freenas-boot
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://illumos.org/msg/ZFS-8000-9P
scan: resilvered 527M in 0 days 00:06:14 with 0 errors on Sat Apr 18 11:10:36 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da7p2 ONLINE 0 0 2
da6p2 ONLINE 0 0 0

errors: No known data errors

I can see the checksum error on da7p2. I'm going to try clearing the error and resilvering.

 

[ctag]

bns-citadel# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:24:11 with 0 errors on Sat Apr 18 12:26:42 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da7p2 ONLINE 0 0 0
da6p2 ONLINE 0 0 0

errors: No known data errors

As far as I can tell, that appears to have worked.

 

[ctag]

Well I made the mistake of trying to update my jails.

First I used `iocage` from the command line to upgrade jails to the same FreeBSD release. I couldn't find how to do this from the web UI, although I'd be pretty surprised if upgrading jails couldn't be done there...

Code:

iocage upgrade bns-xwing -r 11.3-RELEASE

Then I updated the patch level from the web UI.

That all went pretty smoothly and the jails restarted without any services failing (that I noticed). Next up I went to upgrade the packages on the system, but ran into an error with `pkg`. Resolving that entailed:

Code:

pkg remove -f pkg
pkg bootstrap -f

After which I could update and upgrade system packages.

During the package upgrade on one jail postgresql went from 9.5.12 to 9.5.21. After that `psql` stopped working, but running pkg upgrade again installed postgresql 11.7 and then psql worked again(??!). To make things even weirder, somewhere in that upgrade path the default postgresql user changed from "pgsql" to "postgres" which meant that when I re-imported my sql dump to the newer server, it got some sort of split-brain injury where half of the users are wrong:

Code:

postgres=# \l
                             List of databases
   Name    |  Owner   | Encoding | Collate | Ctype |   Access privileges  
-----------+----------+----------+---------+-------+-----------------------
pgsql     | pgsql    | UTF8     | C       | C     |
postgres  | postgres | UTF8     | C       | C     |
template0 | postgres | UTF8     | C       | C     | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
template1 | postgres | UTF8     | C       | C     | postgres=CTc/postgres+
           |          |          |         |       | pgsql=CTc/postgres   +
           |          |          |         |       | =c/postgres
ttrss     | pgsql    | UTF8     | C       | C     | pgsql=CTc/pgsql      +
           |          |          |         |       | =Tc/pgsql            +
           |          |          |         |       | ttrss=CTc/pgsql
(5 rows)

I'm not really sure what I can even do about that besides tearing into it manually and trying to remove the old role again. I suppose this upgrade was planned on the postgres side of things with `pg_upgrade` in mind, but that tool doesn't appear to work on FreeBSD since only one instance of the server can be installed at a time.

 

[ctag]

I'm trying to set up an rsync task to backup a remote server, but now the lines in /etc/cronttab don't contain the rsync command. Instead it's:

0 0 * * sun root PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" midclt call rsynctask.run 7 > /dev/null 2> /dev/null

There's shockingly little information on Google for midclt and I can't seem to find where the rsync commands are squirreled away :(

 

[ctag]

OK, so it looks like `midclt` is a python front-end for `middlewared` which is FreeNAS specific. Digging through client.py I don't see any methods that would let me print the command and arguments to verify them..

 

[ctag]

Finally got the sync to work by removing compression. I guess there was a compatibility mismatch in the rsync versions. I still don't know how to get at the actual rsync output to debug issues though.

This involves to a broader UI oddity I'd like to explore sometime. When an rsync command fails, I can click the "Failed" button and get... the stacktrace from the python middleware? It does give me the rsync exit code, but that's a minimum of helpfulness. I'd like to see a lot more metrics present in many of FreeNAS's UI. When did the last task complete? How much data was transferred? What's the result of the SMART tests? How long did the last scrub take? What was the CLI output? As far as I can tell none of that is available from the web UI, and I really think it should be.

And that's not even mentioning how clunky the UI is with accordion expansion panels everywhere that adds more clicks to get to actions I want to take.

 

[ctag]

Trying to get mariadb-server104 installed in a jail and it wouldn't start. After pulling my hair out for a few days I finally found a blog post that explains the default my.cnf has the PID file set to a bad location, and sure enough changing it led to the server starting.

https://www.samueldowling.com/2020/06/26/install-nextcloud-on-freenas-iocage-jail-with-hardened-security/

This experience has really solidified for me how much I don't know how to fish for myself. I checked the output of /var/log/messages and log files in /var/db/mysql/ and couldn't find anything at all, much less errors that would lead me to fixing the pidfile location on my own.

And so I don't lose the link, I'm also using this blog post: https://www.pavlovs.ky/2020/02/16/freenas-11-3-jail-wordpress-nginx/

 

[ctag]

Previously I've had the opinion that losing the past decade of home videos and cellphone pictures would be miserable, but survivable. To that end having a NAS I felt was dependable was "good enough." This year, however, I did taxes on my own for the first time, without my dad's help (I'm a big kid now!) and I'm ready to start having a backup of this freenas server (and tax documents) in an offsite location.

Becoming a remote sysadmin for a server running in a family member or friend's house seems like a bad idea, so I am mostly interested in an offline solution that I can update once a year. To that end I bought (yet another) Easystore 8TB external drive. Unfortunately despite healthy levels of conversation on this forum, using a USB drive to backup a freenas box seems far from plug-and-play. I'm still trying to read through the threads and figure out what my plan is going to be.

In the meantime the drive is here and I've run a badblocks and SMART tests on it to burn it in. The first few SMART attempts failed because the USB port went to sleep on me, so I wound up using gnu-watch to keep pinging the drive and keep it awake.

Here's the guide I'm using: https://www.reddit.com/r/DataHoarder/comments/alh22g/burning_in_hard_drives/efemr7k/

And here's the forum version: https://www.ixsystems.com/community/resources/hard-drive-burn-in-testing.92/