Jails won't start after upgrade

W

Wolfeman0101

Patron
Joined
Jun 14, 2012
Messages
428
In the GUI Jails are empty. This is the error I'm seeing.

Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1198, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1102, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/jail_freebsd.py", line 181, in query
    resource_list = iocage.list('all', plugin=True, plugin_data=True)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 1344, in list
    return ioc_list.IOCList(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_list.py", line 76, in list_datasets
    ds = list(ds)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 84, in get_dependents
    if ds.locked:
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 90, in locked
    return not self.mounted or (
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 79, in mounted
    return self.properties['mounted'] == 'yes'
KeyError: 'mounted'
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
Upgrade from before 11.3 to 11.3?

Were your jails Warden-based?

You need to reboot to the older version and do the migration from Warden to iocage before upgrading... just search for migrate warden jails to iocage, there are plenty of tips on how you can do it.
 
W

Wolfeman0101

Patron
Joined
Jun 14, 2012
Messages
428
sretalla said:
Upgrade from before 11.3 to 11.3?

Were your jails Warden-based?

You need to reboot to the older version and do the migration from Warden to iocage before upgrading... just search for migrate warden jails to iocage, there are plenty of tips on how you can do it.
Click to expand...
It was post 11.3 iocage jails.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
OK, so what does zfs list show... the error indicates perhaps your iocage mount isn't there.

Also zpool status -v might have some clues.
 
W

Wolfeman0101

Patron
Joined
Jun 14, 2012
Messages
428
sretalla said:
OK, so what does zfs list show... the error indicates perhaps your iocage mount isn't there.

Also zpool status -v might have some clues.
Click to expand...
Code:
bkwolfe@Packers:~ % zfs list
NAME                                                    USED  AVAIL     REFER  MOUNTPOINT
freenas-boot                                           2.71G  11.1G       31K  none
freenas-boot/ROOT                                      2.57G  11.1G       25K  none
freenas-boot/ROOT/11.3-U3.2                            6.51M  11.1G     1.01G  /
freenas-boot/ROOT/FreeNAS-12.0-BETA                    2.57G  11.1G     1.06G  /
freenas-boot/ROOT/Initial-Install                         1K  11.1G      510M  legacy
freenas-boot/ROOT/default                               219K  11.1G      510M  legacy
freenas-boot/grub                                       131M  11.1G     6.83M  legacy
vol1                                                   15.2T  1.68T     14.8T  /mnt/vol1
vol1/.system                                            484M  1.68T      984K  legacy
vol1/.system/configs-5ece5c906a8f4df886779fae5cade8a5   359M  1.68T      359M  legacy
vol1/.system/cores                                     35.3M   989M     35.3M  legacy
vol1/.system/rrd-5ece5c906a8f4df886779fae5cade8a5      69.8M  1.68T     69.8M  legacy
vol1/.system/samba4                                    7.51M  1.68T     5.64M  legacy
vol1/.system/services                                   153K  1.68T      153K  legacy
vol1/.system/syslog-5ece5c906a8f4df886779fae5cade8a5   11.7M  1.68T     11.7M  legacy
vol1/.system/webui                                      141K  1.68T      141K  legacy
vol1/iocage                                             362G  1.68T     13.3M  /mnt/vol1/iocage
vol1/iocage/download                                    971M  1.68T      141K  /mnt/vol1/iocage/download
vol1/iocage/download/11.1-RELEASE                       260M  1.68T      260M  /mnt/vol1/iocage/download/11.1-RELEASE
vol1/iocage/download/11.2-RELEASE                       272M  1.68T      272M  /mnt/vol1/iocage/download/11.2-RELEASE
vol1/iocage/download/11.3-RELEASE                       288M  1.68T      288M  /mnt/vol1/iocage/download/11.3-RELEASE
vol1/iocage/download/11.4-RELEASE                       152M  1.68T      152M  /mnt/vol1/iocage/download/11.4-RELEASE
vol1/iocage/images                                      141K  1.68T      141K  /mnt/vol1/iocage/images
vol1/iocage/jails                                       356G  1.68T      153K  /mnt/vol1/iocage/jails
vol1/iocage/jails/CentOS-paoqys                         202G  1.87T     4.45G  -
vol1/iocage/jails/deluge                               24.7G  1.68T      160K  /mnt/vol1/iocage/jails/deluge
vol1/iocage/jails/deluge/root                          24.7G  1.68T     19.3G  /mnt/vol1/iocage/jails/deluge/root
vol1/iocage/jails/nzbget                               4.01G  1.68T      160K  /mnt/vol1/iocage/jails/nzbget
vol1/iocage/jails/nzbget/root                          4.01G  1.68T     3.50G  /mnt/vol1/iocage/jails/nzbget/root
vol1/iocage/jails/organizr                             1.18G  1.68T      153K  /mnt/vol1/iocage/jails/organizr
vol1/iocage/jails/organizr/root                        1.18G  1.68T     2.01G  /mnt/vol1/iocage/jails/organizr/root
vol1/iocage/jails/plex                                  112G  1.68T      160K  /mnt/vol1/iocage/jails/plex
vol1/iocage/jails/plex/root                             112G  1.68T     95.3G  /mnt/vol1/iocage/jails/plex/root
vol1/iocage/jails/radarr                               6.36G  1.68T      160K  /mnt/vol1/iocage/jails/radarr
vol1/iocage/jails/radarr/root                          6.36G  1.68T     4.75G  /mnt/vol1/iocage/jails/radarr/root
vol1/iocage/jails/sonarr                               5.38G  1.68T      173K  /mnt/vol1/iocage/jails/sonarr
vol1/iocage/jails/sonarr/root                          5.38G  1.68T     4.72G  /mnt/vol1/iocage/jails/sonarr/root
vol1/iocage/jails/tautulli                              911M  1.68T      153K  /mnt/vol1/iocage/jails/tautulli
vol1/iocage/jails/tautulli/root                         911M  1.68T     1.17G  /mnt/vol1/iocage/jails/tautulli/root
vol1/iocage/log                                         332K  1.68T      332K  /mnt/vol1/iocage/log
vol1/iocage/releases                                   4.14G  1.68T      141K  /mnt/vol1/iocage/releases
vol1/iocage/releases/11.1-RELEASE                      1.21G  1.68T      141K  /mnt/vol1/iocage/releases/11.1-RELEASE
vol1/iocage/releases/11.1-RELEASE/root                 1.21G  1.68T     1.17G  /mnt/vol1/iocage/releases/11.1-RELEASE/ro                                                                                                                     ot
vol1/iocage/releases/11.2-RELEASE                      1.22G  1.68T      141K  /mnt/vol1/iocage/releases/11.2-RELEASE
vol1/iocage/releases/11.2-RELEASE/root                 1.22G  1.68T     1.19G  /mnt/vol1/iocage/releases/11.2-RELEASE/ro                                                                                                                     ot
vol1/iocage/releases/11.3-RELEASE                      1.29G  1.68T      141K  /mnt/vol1/iocage/releases/11.3-RELEASE
vol1/iocage/releases/11.3-RELEASE/root                 1.29G  1.68T     1.25G  /mnt/vol1/iocage/releases/11.3-RELEASE/ro                                                                                                                     ot
vol1/iocage/releases/11.4-RELEASE                       422M  1.68T      141K  /mnt/vol1/iocage/releases/11.4-RELEASE
vol1/iocage/releases/11.4-RELEASE/root                  422M  1.68T      416M  /mnt/vol1/iocage/releases/11.4-RELEASE/ro                                                                                                                     ot
vol1/iocage/templates                                   141K  1.68T      141K  /mnt/vol1/iocage/templates
bkwolfe@Packers:~ %

Code:
bkwolfe@Packers:~ % zpool status -v
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0B in 0 days 00:05:26 with 0 errors on Thu Jul 16 03:50:26 2020
config:

        NAME                                          STATE     READ WRITE CKSUM
        freenas-boot                                  ONLINE       0     0     0
          gptid/85e3c1fb-723f-11e5-9b6a-d050996fddf0  ONLINE       0     0     0

errors: No known data errors

  pool: vol1
 state: ONLINE
  scan: scrub repaired 0B in 0 days 17:07:24 with 0 errors on Mon Jul  6 19:07:29 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        vol1                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/851a9a63-7241-11e5-8dae-d050996fddf0  ONLINE       0     0     0
            gptid/8576901e-7241-11e5-8dae-d050996fddf0  ONLINE       0     0     0
            gptid/884f1352-7cbc-11e7-b6fd-d050996fddf0  ONLINE       0     0     0
            gptid/862c6c85-7241-11e5-8dae-d050996fddf0  ONLINE       0     0     0
            gptid/145d739a-0788-11ea-bb64-0cc47a905f51  ONLINE       0     0     0
            gptid/28244418-9c05-11e7-9da4-d050996fddf0  ONLINE       0     0     0

errors: No known data errors
bkwolfe@Packers:~ %
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
OK; most of the jail datasets are mounted... not sure if vol1/iocage/jails/CentOS-paoqys is something special...

Pool is healthy.

What does iocage list show?
 
W

Wolfeman0101

Patron
Joined
Jun 14, 2012
Messages
428
Code:
bkwolfe@Packers:~ % iocage list
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/iocage_cli/list.py", line 82, in cli
    _list = iocage.list(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 1344, in list
    return ioc_list.IOCList(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_list.py", line 76, in list_datasets
    ds = list(ds)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 84, in get_dependents
    if ds.locked:
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 90, in locked
    return not self.mounted or (
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 79, in mounted
    return self.properties['mounted'] == 'yes'
KeyError: 'mounted'
bkwolfe@Packers:~ %
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
Does zfs mount -a show any error?

I suspect that error is coming from the jail that isn't mounted.
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Here's the problem: vol1/iocage/jails/CentOS-paoqys/root didn't mount. Not sure what happened to this jail, but that's why iocage can't start. You may have to rebuild this jail. You'll likely have to delete the vol1/iocage/jails/CentOS-paoqys dataset from Storage->Pools to get iocage to start.
 
sretalla

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
Perhaps we can see if it's even trying to mount...

zfs get mountpoint vol1/iocage/jails/CentOS-paoqys

Maybe you will also want to zfs send | recv that dataset somewhere else first before destroying it so you can salvage some of your 4GB of data from it later. (or maybe it's 202 GB...)
 
Yorick

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
CentOS-paoqys This is the naming convention that bhyve uses for zvols, and the name CentOS also hints at that. My best guess is this is a zvol that was created inside iocage. Interesting mistake to make.

send/recv into a new zvol - this time on the pool itself or inside a zvols dataset, verify that worked by pointing the VM to it and starting it up, then delete the old zvol.

That should do it.
 
Patrick M. Hausen

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Yorick said:
send/recv into a new zvol - this time on the pool itself or inside a zvols dataset, verify that worked by pointing the VM to it and starting it up, then delete the old zvol.
Click to expand...
Why not just zfs rename? :wink:

Patrick
 
Last edited:
Yorick

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Patrick M. Hausen

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Yorick said:
Only because I wasn't aware moving a zvol that way is a thing. That's a much better idea, agreed.
Click to expand...
It's very much like link() followed by unlink() in a traditional filesystem.
 
You must log in or register to reply here.

Similar threads

A
Problem after upgrade TrueNAS 12.0-U2 > U3
Replies
5
Views
2K
atomique90
A
G
TrueNAS 12.0 Stable update failing
Replies
5
Views
1K
winnielinnie
winnielinnie
S
Unable to create pool - invalid continuation byte
Replies
8
Views
2K
appliance
A
E
After upgrade I got this error
Replies
5
Views
2K
Jessep
J
danjb
Upgraded to TrueNAS 12.0, can't create or delete VM's
Replies
1
Views
967
danjb
danjb
Top