iocage jails will no longer start

[JJT211]

I just made the switch over to iocage a few days ago and everything was working fine until I got a warning message that one of my boot usb drives went down. I tried rebooting server and it would not reboot until I removed the faulty usb drive. Once I removed faulty drive, all appeared fine as I booted up. But now I noticed that 2 of my 9 jails will not start. Could it be related to server being in a degraded state? Is this normal? I seem to recall having a usb boot drive fail before and I didnt miss a beat. Whats the deal now?

Also, I think may be worth mentioning is I also have the ZFS pool upgrade pending warning. Could this be related?

 

[dlavigne]

Any errors in /var/log/messages?

 

[JJT211]

Any errors in /var/log/messages?

I dont have that directory. I think because I just booted back up. I am getting this in the lower console messages

Code:

Feb  5 09:47:02 freenas daemon[4178]: ==> Log data will now stream in as it occurs:
Feb  5 09:47:02 freenas daemon[4178]:
Feb  5 09:47:08 freenas daemon[4178]:	 2018/02/05 09:47:08 [WARN] agent: Check 'freenas_health' is now warning
Feb  5 09:47:09 freenas daemon[4178]:	 2018/02/05 09:47:09 [ERR] agent: failed to sync remote state: No cluster leader
Feb  5 09:47:09 freenas daemon[4178]:	 2018/02/05 09:47:09 [WARN] raft: Heartbeat timeout from "" reached, starting election

 

[JJT211]

Any errors in /var/log/messages?

OK, maybe im missing something. How do I navigate to that directory? When im in the shell, I type in "ls" and I do not get a var directory. I must be missing something. If you mean /var/logs/messages in the jail, I cannot access anything in my jails cuz they will not start

Anyways, as an update, my new USB drive arrived and I still wasnt able to successfully replace it. Freenas recognized the drive, and I got something like a "Disk has been replaced" message, but I waited and nothing happened. I tried again and still nothing. I just saw a brief 1MBps spike under disc activity but my system continued to display degraded state warning message. Ive replaced a failed boot drive before, so im confident I did this correctly.

Since my jails weren't functioning properly and I doubt the failed usb drive was related anyway, I decided to just perform a fresh install and restore config. I just finished all of that and now my jails are all reported as "down", they will not start and I cannot delete them either.

How do I delete these jails and start over?

 

[JJT211]

When I try to start any jail from the shell, I get this:

root@freenas:~ # iocage exec plex service plexmediaserver_plexpass start
plex is not running, starting jail
mount: /mnt/iocage: No such file or directory
jail: /sbin/mount -t nullfs -o ro /mnt/volume/movies /mnt/iocage/jails/plex/root/mnt/movies: failed


I have no idea what to do other than start completely over from scratch. That would suck. :(

 

[LotLits]

So, I had a similar problem that ended up causing me to delete the jail before I realized my issue. For me, the issue was that the mount path changed from /mnt/tank/iocage/jails to /mnt/iocage/jails (which doesn't exist). You can `iocage fstab -e` and correct the paths and try to start them.

I suspect (although haven't tested or anything) that maybe the issue was due to having everything on an encrypted drive and it ends up messing with the jail paths when trying to autostart the iocage jails because the storage hadn't been unlocked yet. Not positive though.

It happened for me after an upgrade to 11.1-U1 from 11.1 so it could be related to that as well. Not sure.

 

[JJT211]

Hmmm....I think that may be it.

So using your example the command should look like this?

iocage fstab -e /mnt/iocage/jails /mnt/tank/iocage/jails

 

[LotLits]

Actually, you can just run:

iocage fstab -e <jailname>

That'll pop you into vi to edit the fstab as if it was local.

 

[JJT211]

j

 

[LotLits]

Yeah, the fstab shows from the root of the freenas. So, if your pool name was 'volume':

Code:

/mnt/volume/movies	  /mnt/iocage/jails/plex/root/mnt/movies  nullfs  ro 0 0

would become:

Code:

/mnt/volume/movies	  /mnt/volume/iocage/jails/plex/root/mnt/movies  nullfs  ro 0 0

You should be able to ls from your freenas: /mnt/volume/iocage/jails/plex/root/mnt/movies

I think you could also remove it with the following, although I've never tried it and then just readd it although I'm not positive:

Code:

iocage fstab -r plex "/mnt/movies /mnt/volume/movies nullfs ro 0 0"

Basically, whatever you did to add it with -r instead of -a.

All of them would have to be corrected before it would start, I would expect.

 

[JJT211]

Ignore my last post, I caught it as it was the same issue you were having before.

I fixed it and am now able to start my plex jail.

root@freenas:~ # iocage exec plex service plexmediaserver_plexpass start
plexmediaserver_plexpass already running? (pid=50215).
True

But when I try to navigate to my jail in browser it times out.

When I try to ping from shell I get this:

root@freenas:~ # ping 10.1.1.200
PING 10.1.1.200 (10.1.1.200): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down

 

[JJT211]

When you say:

You should be able to ls from your freenas: /mnt/volume/iocage/jails/plex/root/mnt/movies

Do you mean I should be able to navigate to that directory straight from the shell? If so, this is all im getting when I "ls" from root directory

root@freenas:~ # ls
.bashrc .gdbinit .k5login .profile .ssh
.cshrc .history .login .shrc


Shouldn't I be able to see my volume?

 

[LotLits]

Sorry, was just noting that so we make sure the jail path is at least there. In theory, if your jail started up then your fstab is probably good now.

The issue you're having now looks like networking on the jail which I didn't run into and am not quite sure about. You could iocage console <jail> and check ifconfig to see the interface has an IP and netstat -rn to see there default route. Maybe try pinging default gateway. I don't have much experience with the networking and iocage.

 

[JJT211]

No probs, I really appreciate all of your help, thx!

For anyone else, im not able to connect to my jail. I've tried pinging my gateway from inside jail and pinging my jail from console and I get nothing.

If I ifconfig I get:

root@plex:/ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
vnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:60:14:fa:0a
hwaddr 02:30:a0:00:06:0b
inet 10.1.1.200 netmask 0xffffff00 broadcast 10.1.1.255
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

And when I netstat -rn:

root@plex:/ # netstat -rn
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 10.1.1.1 UGS vnet0
10.1.1.0/24 link#2 U vnet0
10.1.1.200 link#2 UHS lo0
127.0.0.1 link#1 UH lo0


And I still cannot delete or stop jail from GUI.

 

[JJT211]

Ok, I got it figured out now.

I had already deleted my jails by the time I figured this out (so not sure it was necessary or not) but I had to create a few tunables and reboot server to fix networking error. See thread

https://forums.freenas.org/index.php?threads/iocage-vnet0-and-bridge0.59964/