iocage list is very slow after upgrade to FN-11.3-RC1

[ornias]

Ok but I'm running 11.3-U1 and still see the dramatic slowdowns if I have a large number of snapshots on my system.

You call that a dramatic slowdown? Meet 11.3 (non-u), which could be 3-4 MINUTES based on snapshots.

 

[Patrick M. Hausen]

Well there's still room for improvement, isn't it?

Code:

root@freenas-pmh[~]# time iocage list
+-----+-----------+-------+--------------+---------------+
| JID |   NAME    | STATE |   RELEASE    |      IP4      |
+=====+===========+=======+==============+===============+
| 5   | gitea     | up    | 11.3-RELEASE | 192.168.93.51 |
+-----+-----------+-------+--------------+---------------+
| 6   | grafana   | up    | 11.3-RELEASE | 192.168.93.56 |
+-----+-----------+-------+--------------+---------------+
| 4   | guacamole | up    | 11.3-RELEASE | 192.168.93.52 |
+-----+-----------+-------+--------------+---------------+
| 7   | monit     | up    | 11.3-RELEASE | 192.168.93.55 |
+-----+-----------+-------+--------------+---------------+
| 3   | nextcloud | up    | 11.3-RELEASE | 192.168.93.53 |
+-----+-----------+-------+--------------+---------------+
| 1   | openvpn   | up    | 11.3-RELEASE | 192.168.93.54 |
+-----+-----------+-------+--------------+---------------+
| 2   | proxy     | up    | 11.3-RELEASE | 192.168.93.50 |
+-----+-----------+-------+--------------+---------------+
iocage list  1.62s user 8.86s system 92% cpu 11.347 total
root@freenas-pmh[~]# zfs list -t snap -r hdd/iocage/jails | wc -l
    5026

But performance in real production environments is an ongoing iocage topic it seems ...

Patrick

 

[ornias]

But performance in real production environments is an ongoing iocage topic it seems ...

Patrick

There primarily seem to be a lot of issues based on how iocage also tries to deal with snapshotting a lot...
This 11.3non-u performance issues where due to iocage also trying to read ALL snapshots for ALL jails every list command and to be frankfully honest, they where quite arogant about it...

So I share your doubts about it being being ready for production performance wise...

 

[Patrick M. Hausen]

We based our "proServer" hosting environment on iocage and have been struggling with performance ever since, unfortunately. OTOH the tool is quite capable and we have yet to find a suitable replacement. Creating an ansible module that understands jails and using just the base system would be one option. py-libiocage is another one.

Sometimes it was just ecosystem/middleware problems, for example we first diagnosed the problem with python rather stupidly closing all fd's from 3 to MAXFDS indivdually ... https://reviews.freebsd.org/D21627

And of course a system running hundreds of jails will have tens of thousands of snapshots - that's the point ;)

Patrick

 

[ornias]

We based our "proServer" hosting environment

Happen to have a link to check out? :)

 

[Patrick M. Hausen]

My architecture talk in Paris 2017:

FreeBSD as a Hosting Platform, Revisited,by Patrick M. Hausen (EuroBSDcon 2017)

www.youtube.com

Our product website:

Server-Hosting für Entwickler: Root-Access und High-Performance.

Der Server, den sich Entwickler immer gewünscht haben: Root-Access ohne Update-Sorgen.

infrastructure.punkt.de

English still being worked on, sorry.

Patrick

 

[ornias]

My architecture talk in Paris 2017:

FreeBSD as a Hosting Platform, Revisited,by Patrick M. Hausen (EuroBSDcon 2017)

www.youtube.com

Our product website:

Server-Hosting für Entwickler: Root-Access und High-Performance.

Der Server, den sich Entwickler immer gewünscht haben: Root-Access ohne Update-Sorgen.

infrastructure.punkt.de

English still being worked on, sorry.

Patrick

Skimmed through it a bit, going to watch the whole talk, seems interesting indeed...

Don't worry about english, if it was in german it would only be a good training to get my german freshed-up a bit ;)

 

[hylyah]

Same, very very slow to list jails...

 

[BetYourBottom]

Same.
Iocage list is relatively fast taking only a second or two before displaying
Jails in the GUI takes much much longer 15 seconds or more.