iocage fetch still not finished after 8 hours, now everything is slow with jails

[JamesT42]

Hi!
Is this bug known to someone? I tried searching it but no luck so far.
I'm trying to upgrade my jails from 11.2 to 11.3. But the iocage fetch is taking forever to finish, is this normal? Also it seems the whole system slows down while this is running, after a while the webinterface is not responding, samba service down, I got an email saying smartd is not running, console commands don't execute in the ssh session I had opened before, I can not login in the hardware console because after entering the username nothing happens. I had a top running and there was not much happening. Only thing I can do is reset. Also after this disaster whenever i start jails the system slows down massively, samba acces takes severals seconds for every folder, webinterface is slow. With no jails everything is ok speed-wise. I don't understand what is going on.
This happened on 11.3-U2.1 and is repeatable on 11.3-U3.2 after update.

Code:

root@freenas1:~ # iocage upgrade -r 11.3-RELEASE qbittorrent
Snapshotting qbittorrent...
Snapshot already exists
Updating plugin INDEX...
Running upgrade...
11.3-RELEASE is missing 'src.txz', please refetch!
root@freenas1:~ # iocage fetch 11.3-RELEASE
[0] 11.2-RELEASE (EOL)
[1] 11.3-RELEASE
[2] 11.4-RELEASE
[3] 12.0-RELEASE (EOL)
[4] 12.1-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection: (11.3-RELEASE)
Type EXIT to quit: 1
Fetching: 11.3-RELEASE

Extracting: base.txz...
Extracting: lib32.txz...
src.txz missing, will try to redownload!
Downloading: src.txz [####################] 100%
Extracting: src.txz...

I do not think this is hardwar related, but here are the system specs:
Motherboard make and model - HP Microserver Gen8 CPU make and model - Intel Xeon E3-1265L RAM quantity - 16GB ECC Hard drives, quantity, model numbers, and RAID configuration, including boot drives Boot drive: Some WD Green SSD in USB enclosure Jails: 1 ST1000LM014 (no raid level) Mass storage: 4 WDC WD120EFAX in RaidZ1 Hard disk controllers - integrated in mainboard Network cards - Mellanox ConnectX 10G

What can I do? Can I rebuild iocage from scratch somehow and keep my old jails?

Cheers
James

 

[Samuel Tai]

Unfortunately, your jail pool is running on a hybrid SMR drive, which explains the very slow write performance.

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.ixsystems.com

You'll need to switch to a different drive for your jail pool. You'll sadly need to wait for this to complete, so as not to corrupt iocage. Once it completes, you may want to move everything over to your RAIDZ1 pool.

 

[JamesT42]

Unfortunately, your jail pool is running on a hybrid SMR drive, which explains the very slow write performance.

List of known SMR drives

Hard drives that write data in overlapping, "shingled" tracks, have greater areal density than ones that do not. For cost and capacity reasons, manufacturers are increasingly moving to SMR, Shingled Magnetic Recording. SMR is a form of PMR...

www.ixsystems.com

You'll need to switch to a different drive for your jail pool. You'll sadly need to wait for this to complete, so as not to corrupt iocage. Once it completes, you may want to move everything over to your RAIDZ1 pool.

Its a CMR drive with SSD cache and I have been using it for 5 years now for my jails, no smart errors, no failed scrub ever, so I do not see this as an explanation.

 

[Samuel Tai]

During those 5 years your usage pattern was mostly reads. Scrubs and SMART tests don't change that pattern. Now that you're updating, you're writing a lot to all jails, and seeing dismal performance writing. Smells like SMR to me.

 

[JamesT42]

During those 5 years your usage pattern was mostly reads. Scrubs and SMART tests don't change that pattern. Now that you're updating, you're writing a lot to all jails, and seeing dismal performance writing. Smells like SMR to me.

There is also a zvol on the drive for a virtual machine that is used for compiling stuff, never had a problem, sometimes 20GB of very small files are written in 1-2 hours. It is not a SMR drive. It does not smell like an SMR drive. iocage fetch installs 1.2GB, that is nothing. SMR does not explain anything in regards to my problems. Please stop suggesting it.

 

[Samuel Tai]

Ok, if it's not SMR, then the next most likely scenario is the flash side of the drive has failed, and you're seeing the bare performance of the hard drive side of the drive.

 

[JamesT42]

Ok, if it's not SMR, then the next most likely scenario is the flash side of the drive has failed, and you're seeing the bare performance of the hard drive side of the drive.

Did you read my post? Why would slow write performance crash everything? Also the SSD cache is used to cache reads, not for writes. I've just written about 30GB with approx. 100MB/s as a test, no problems there.

 

[Samuel Tai]

Forgive me if I've been unhelpful. I really am trying to help. Please provide zpool status -v.

 

[JamesT42]

Nothing interesting there. Could we go to how to repair iocage now?

Code:

  pool: SSHD
 state: ONLINE
  scan: scrub repaired 0 in 0 days 08:29:42 with 0 errors on Sun Jun 28 08:31:49 2020
config:

        NAME                                              STATE     READ WRITE CKSUM
        SSHD                                              ONLINE       0     0     0
          gptid/56472412-4da2-11e5-92b9-3ca82a9fe208.eli  ONLINE       0     0     0

errors: No known data errors

 

[Samuel Tai]

OK, grep iocage /var/log/middlewared.log. Also, please verify your jail pool has been properly designated as the iocage pool: zfs get org.freebsd.ioc:active <name of jail pool>. Make sure this property isn't set on your other pools, using the same ZFS get command.