Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

iocage list is very slow after upgrade to FN-11.3-RC1

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Niels Erik

Junior Member
Joined
Aug 9, 2015
Messages
16
Also posted to: https://github.com/iocage/iocage/issues/1114
iocage list is very slow after upgrade to FN-11.3-RC1
RC 1 contains iocage 1.2
The command 'iocage list' takes 45 seconds to complete on my system.
After a lot of digging I have found out that the time is consumed in
iocage-lib/zfs.py line 8 run(command,**kwargs)
The function performs the command "zfs get -H -o name,property,value all" , that lists all attributes, for all snapshots, for all data-sets.
On my system it generates an output that is 256960 lines long.
I have 183 data-sets on my system, and 2 months daily snapshots.

The main part of my data-sets is not even in the path of iocage.. and should not be of interest to iocage.
There is only 3 jails on my system.
Other iocage commands like console have the same behavior.
Have others experienced this slow behavior?
Should this be reported as a bug?
 
Last edited:

Yorick

Dedicated Sage
Joined
Nov 4, 2018
Messages
1,781
Have others experienced this slow behavior?
Should this be reported as a bug?
No, and yes.

I have only one jail, a plex plugin. iocage list returns results immediately.

And yes, that should be reported as a bug. There's some interplay between your setup and the 11.3 release that's causing this, I am thinking. And with a bug, the devs can figure out what it is and account for it / fix it.
 

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
I have same Thing going on its just not the J list doing it. When I edit a jail it takes like 1min to load the setting's.
Also my jails are on a Raidz1 array of SSD'S I mean speed of pool is not a problem. Read & wright is in the 1.8Gb/s both way's
I do not have the amount of snapshots you do. I do a 3 week and 2 months snapshot. The 3 week one is Weekly.

Unlike this problem witch I am not having.
My problem in detail.

Plugin also take About 20 secs to load not sure if your also have the same problem.
 
Last edited:

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
I Removed all my snapshots to see if this would help it has not. Still take 60sec just to load the jails in the gui.
I am pretty sure this is a bug if more then One person is having this problem.
 
Last edited:

Newfoundland.Republic

Neophyte Sage
Joined
Jul 2, 2019
Messages
637
I have some slowness with four jails - 3 Emby jails (different subnets) and 1 for the UniFi Controller. It seems that the jail GUI response takes about twice as long to load. Maybe if I have time this weekend I will delete the four jails and reinstall them to see if a "native" installation makes it quicker. I do have all my jails upgraded to 11.3.
 

beedix

Junior Member
Joined
Aug 11, 2017
Messages
20
I also upgraded from 11.2 (latest) to 11.3 today. After upgrade, I have the same issue with extremely slow iocage commands. I have 13 jails. As others describe, the GUI even has a seemingly consistent delay until the jails are even displayed.

Any iocage command seems to have this delay.

Any help on where to go is appreciated.
 

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
I do not have all my jails upgraded its about half and half. 11.3/11.2.
 

Newfoundland.Republic

Neophyte Sage
Joined
Jul 2, 2019
Messages
637
I wonder, could it be where it is reaching out to the iXsystems' official plugins and the new community plugins site?
 

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
Here is what happening for me, go to About 00:25 Even deleting jails takes a long time which in 11.2 it was fast. Right after that I add a jail and it takes like 20 secs to just load the setup page.

Well ones a jail is created It should not have to talk to the ixsystems any more, unless you are updating a plugin I think.
 
Last edited:

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
Has any one created a bug issues for this yet ?
 

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
when you do iocage list is it very slow or fast for you?
 

pkropf

Neophyte
Joined
Jul 15, 2019
Messages
11
I'm seeing the same slowdown after upgrading to 11.3. From the CLI, iocage list shows:

b0210520-fs01% time iocage list
+-----+------------+-------+--------------+-----------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+============+=======+==============+=================+
| 12 | bacula | up | 11.3-RELEASE | 192.168.1.94/24 |
+-----+------------+-------+--------------+-----------------+
| 13 | git | up | 11.3-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
| 2 | influxdb | up | 11.3-RELEASE | 192.168.1.80 |
+-----+------------+-------+--------------+-----------------+
| 3 | logging | up | 11.2-RELEASE | 192.168.1.16 |
+-----+------------+-------+--------------+-----------------+
| 9 | mqtt | up | 11.2-RELEASE | 192.168.1.99 |
+-----+------------+-------+--------------+-----------------+
| 5 | mysql | up | 11.3-RELEASE | 192.168.1.55 |
+-----+------------+-------+--------------+-----------------+
| 6 | netbox | up | 11.3-RELEASE | 192.168.1.82 |
+-----+------------+-------+--------------+-----------------+
| 7 | postgres | up | 11.3-RELEASE | 192.168.1.40 |
+-----+------------+-------+--------------+-----------------+
| 8 | zoneminder | up | 11.2-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
18.13 real 3.07 user 15.24 sys
b0210520-fs01%


The GUI is around the same time lag to display.
 

Snow

Senior Member
Joined
Aug 1, 2014
Messages
284
iocage is fast for me, but the Jails in the GUI take about 30-45secs.
 

pkropf

Neophyte
Joined
Jul 15, 2019
Messages
11
Looks like this may be related to the number of snapshots of the iocage dataset:

root@b0210520-fs01[~]# time zfs get -H -o name,property,value -r all space/iocage > /dev/null
zfs get -H -o name,property,value -r all space/iocage > /dev/null 2.54s user 43.88s system 95% cpu 48.354 total
root@b0210520-fs01[~]#
 

pkropf

Neophyte
Joined
Jul 15, 2019
Messages
11
(Forgot this part)

root@b0210520-fs01[~]# zfs list -t snapshot | grep iocage | wc
10143 50715 1085301
root@b0210520-fs01[~]#
 

pkropf

Neophyte
Joined
Jul 15, 2019
Messages
11
I've pruned down to 40 snapshots on this server and response time from the CLI has dropped. However,, it's still in the 20 - 30 second range from the GUI.


root@b0210520-fs01[~]# time iocage list
+-----+------------+-------+--------------+-----------------+
| JID | NAME | STATE | RELEASE | IP4 |
+=====+============+=======+==============+=================+
| 1 | bacula | up | 11.3-RELEASE | 192.168.1.94/24 |
+-----+------------+-------+--------------+-----------------+
| 2 | git | up | 11.3-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
| 3 | influxdb | up | 11.3-RELEASE | 192.168.1.80 |
+-----+------------+-------+--------------+-----------------+
| 4 | logging | up | 11.2-RELEASE | 192.168.1.16 |
+-----+------------+-------+--------------+-----------------+
| 5 | mqtt | up | 11.2-RELEASE | 192.168.1.99 |
+-----+------------+-------+--------------+-----------------+
| 8 | mysql | up | 11.3-RELEASE | 192.168.1.55 |
+-----+------------+-------+--------------+-----------------+
| 6 | netbox | up | 11.3-RELEASE | 192.168.1.82 |
+-----+------------+-------+--------------+-----------------+
| 9 | postgres | up | 11.3-RELEASE | 192.168.1.40 |
+-----+------------+-------+--------------+-----------------+
| 7 | zoneminder | up | 11.2-RELEASE | DHCP |
+-----+------------+-------+--------------+-----------------+
iocage list 1.59s user 2.09s system 94% cpu 3.913 total
root@b0210520-fs01[~]# zfs list -t snapshot | grep iocage | wc
40 200 4120
root@b0210520-fs01[~]#
 

ornias

Neophyte Sage
Joined
Mar 6, 2020
Messages
1,037
Looks like this may be related to the number of snapshots of the iocage dataset:
This has already been known for about a month now (had a chat with iocage folks over on github about it and scolded them about not doing enough research on it, because it slipped past them and thats how it got into FreeNAS 11.3 in the first place.

Most of it has been fixed in 11.3U1.
It's still a tad slow at times, but the minute+ long loadtimes from 11.3 are gone :)
 

pkropf

Neophyte
Joined
Jul 15, 2019
Messages
11
Ok but I'm running 11.3-U1 and still see the dramatic slowdowns if I have a large number of snapshots on my system.
 
Top