SOLVED Successfully Installed Jellyfin in a TrueNAS CORE Jail yesterday, today box hangs while booting

[hunter]

I managed to install Jellyfin a few days ago using the instructions and packages available here:

jellyfin-server-freebsd/Installation_TrueNAS_GUI.md at main · Thefrank/jellyfin-server-freebsd

jellyfin-server component for freebsd. Contribute to Thefrank/jellyfin-server-freebsd development by creating an account on GitHub.

github.com

I had restarted TrueNAS a few times while installing Jellyfin and configuring it last night. I added several shares to the jail and let Jellyfin scan my media libraries.

Today, I can't get TrueNAS to deliver any services including its own web management interface. I fired up IPMI View to see what's going on and see an error message I can't understand, repeted every couple minutes in the iKVM console (see attached .jpg of the screen).

Can anyone see how to regain the use of my NAS server?

 

[Whattteva]

Jellyfin isn't even on the FreeBSD ports tree. Plex and Emby are, but not Jellyfin. I'm guessing you're using an unofficial package?

EDIT:
My bad, I missed the link for the github repo you posted.

Which machines are 192.168.1.2 and 192.168.1.13?

 

[hunter]

Jellyfin isn't even on the FreeBSD ports tree. Plex and Emby are, but not Jellyfin. I'm guessing you're using an unofficial package?

EDIT:
My bad, I missed the link for the github repo you posted.

Which machines are 192.168.1.2 and 192.168.1.13?

192.168.1.2 is the WAN side of a 2nd router I use. (My media server, printer, etc, are behind the first router and 2nd router protects my desktop pc that I use to log in to more sensative sites). I use the double NAT'd pc to access the TrueNAS web page so those messages are the results of my attempts to log into the TrueNAS server (I believe). When I attempt to access the TrueNAS management portal, I get a web page saying:
"Connecting to TrueNAS ... Make sure the TrueNAS system is powered on and connected to the network. "

192.168.1.13 is the address of the TrueNAS CORE server itself.

 

[Whattteva]

192.168.1.2 is the WAN side of a 2nd router I use. (My media server, printer, etc, are behind the first router and 2nd router protects my desktop pc that I use to log in to more sensative sites). I use the double NAT'd pc to access the TrueNAS web page so those messages are the results of my attempts to log into the TrueNAS server (I believe). When I attempt to access the TrueNAS management portal, I get a web page saying:
"Connecting to TrueNAS ... Make sure the TrueNAS system is powered on and connected to the network. "

192.168.1.13 is the address of the TrueNAS CORE server itself.

Gonna need more details on your network layout here. By the way, I'd probably change the subnets to be different and not both on 192.168.1.0/24 cause it is very confusing and could potentially cause conflicts.

Let me summarize this so I have a better picture:

• I'm guessing you have 192.168.1.1 as your main router,
• Your second router's WAN is 192.168.1.2.
• Second router's LAN subnet?
• 192.168.1.13 is your TrueNAS on the first router?

For the record, Jellyfin definitely works fine (I just installed it this morning). Thanks for making me aware about that GitHub repo btw.

 

[hunter]

Sorry, I thought I gave the relevant network details. The 2nd router runs on the 192.168.0.1/24 subnet. I have never gotten confused and am unaware of any conflicts since 10 or so years ago when I went with that topology.

You are correct on all the other points about my network.

Glad you found the github repo and gave it a spin. I was very impressed, my first time using Jellyfin. Jellyfin worked great inside the iocage jail for me for the first several days I had it installed, and survived several reboots without incident. I'm not sure why I have this problem after only adding a jail. I captured more error messages as it was booting up. I think the 2nd screen shows some services that refused to start and are most likely part of my problem. Please help. I (iocage) exported my Jellyfin jail just in case I had trouble later. I can get a root command prompt using iKVM. I wonder if removing the Jellyfin jail from the shell would let me use the web interface again? Please help if you can. Thanks

 

[Whattteva]

I've never seen those vmware-checkvm calls. Are you running this in a hypervisor?
Jellyfin working fine for a while and suddenly failed leads me to believe that you may have corruption on the boot device. It's not installed on a flash drive, is it? Cause those are notorious for dying quick deaths under intensive TrueNAS use.

Do you, by any chance, have backup of your working config file? If so, you may have to clean install TrueNAS and restore the config as a last resort.

It looks like Nginx was able to start correctly, but all the middleware calls are failing. Quite honestly, I have low confidence that removing Jellyfin will recover your TrueNAS web UI.

Could you post the output of

Code:

ifconfig

 

[hunter]

My TrueNAS server runs on bare metal. I don't know the answer if you were asking about the programs in the jails. The boot device is a very old, small, SSD. It has not showed any signs of failure before now.

I found a .tar file from April, which is when I believe I installed a recent version of TrueNAS. I think I have more recent backups of the configuration on the server itself. (Bad selection where to save these I realize now).

Can you explain how one goes about with a clean install? Do I end up letting the installer reformat the SSD drive? I created a boot drive from the latest .iso of TrueNAS. I haven't been successful changing the bios settings to boot off of it through ipmi.

I will post the output of ifconfig next week when I am back home. Thank you for your help.

 

[spuky]

from your screenshots it looks like trunenas is fully booted....

you should be able to login typing root and then your password

then type:

Code:

/etc/netcli

and it will show the IP for the webinterface...

and let you configure an additional IP if the default ip is blocked by Jellyfin

or use

Code:

iocage stop 'jail name'

to stop the jellyfin jail

 

[Whattteva]

from your screenshots it looks like trunenas is fully booted....

you should be able to login typing root and then your password

then type:

Code:

/etc/netcli

Oooh, I never knew that the code running on the console is this python script. Good to know, thanks for that.
On another note... how the heck do you exit this thing without force-killing it? There's no option to exit, unlike pfSense.

 

[spuky]

On another note... how the heck do you exit this thing without force-killing it? There's no option to exit, unlike pfSense.

on my install option 9) shell drops me back to the shell.... (quit or exit would be better names maybe...)

 

[Whattteva]

on my install option 9) shell drops me back to the shell.... (quit or exit would be better names maybe...)

Yeah, but then typing exit on the shell drops back to the menu. If I run this on an SSH session, I essentially can't exit out of the SSH session without killing it.

 

[hunter]

Back home again. Here is the output of the ifconfig command as well as the results of choosing option 1 after typing /etc/netcli

When I tried 'iocage jellyfin stop', I received a message saying there were no pools & iocage requires a pool. Guess it never gets as far as mounting the 7 drive pool on my server.

 

[Whattteva]

Back home again. Here is the output of the ifconfig command as well as the results of choosing option 1 after typing /etc/netcli

When I tried 'iocage jellyfin stop', I received a message saying there were no pools & iocage requires a pool. Guess it never gets as far as mounting the 7 drive pool on my server.

I'm guessing en0 is your NIC? Seems like it at least initializes the network card correctly.
Try running sockstat -4 and maybe see if some essential service is missing (whatever the middleware is).

The only thing I can see there is it looks like your boot drive might be corrupted and you may want to do a clean install and restore your backup config. You can do a clean install by reinstalling from the TrueNAS ISO. Would probably be my next step unless someone else here has a better idea.

 

[hunter]

I'm not sure I have ever seen the name of the NIC. It's a stock SuperMicro board, with just the stock networking chip(s). Unfortunately, I haven't been able to get into the BIOS and I no longer seem to be able to boot off the SD drive I was using. So I was not able to regain access to try sockstat -4.
Not sure what happened, I removed the boot SD for a few boots to see if that would let the board boot off of the USB boot key I burned from the TrueNAS ISO file. No luck! When I do try to boot into the BIOS, I end up at a blank blue screen that says "AB" in the lower right hand corner.

I am very frustrated with this board. Guess I can try hooking it up to a monitor to see if I have any better luck that way getting into BIOS, unless someone has a better suggestion for me to try next.

 

[hunter]

It turns out that the BIOS for this board stopped allowing access after 1/1/2020 and Supermicro put out a new BIOS that fixes that problem. For the benefit of anyone having a similar issue with their Supermicro board, here is the link to the discussion of it I found online:

Supermicro X9SCD won't enter BIOS (error code: AB)

I've just flashed the BIOS of 3 different X9SCD motherboards, all housed within different MicroCloud chassis. I've done this hundreds of times, with this exact BIOS version (x9scd8.612), with no is...

serverfault.com

I also had to discover that f11 during boot with this board gets you to a screen to choose the boot device, without needing to enter the BIOS. & While I was at the Supermicro site, I found and flashed a new firmware (specifically for just the IPMI part of the board) under the support section for this board. After upgrading the board BIOS to 2.3a, I can access the BIOS and change the boot order.

After catching up that part, I booted TrueNAS core 13.0U2 from a USB key I made from the .ISO file. When I told it where to install, it asked if I wanted just a new boot environment (rather than to reformat the whole disk and lose the one I was having the problem with), created it, and then when I booted, TrueNAS found the database, updated it, and all the configuration carried over without needing the backup configuration file.

To my surprise, everything seems to be working OK (including the new Jellyfin jail) after this. Watching it closely to see if the same thing happens again. My fingers are crossed.

Hey @Whatteva, thank you for your help. Be sure to check out the Android Jellyfin app because the Chromecast functionality works fine on that one. I believe there is an app for iPhones too. If your TrueNAS server had any issues after installing Jellyfin, please let me know. My TrueNAS server has been one of the most reliable appliances in my home--I was so surprised to have something like this happen to it.

 

[Whattteva]

It turns out that the BIOS for this board stopped allowing access after 1/1/2020 and Supermicro put out a new BIOS that fixes that problem. For the benefit of anyone having a similar issue with their Supermicro board, here is the link to the discussion of it I found online:

Supermicro X9SCD won't enter BIOS (error code: AB)

I've just flashed the BIOS of 3 different X9SCD motherboards, all housed within different MicroCloud chassis. I've done this hundreds of times, with this exact BIOS version (x9scd8.612), with no is...

serverfault.com

Oh wow. I'm glad mine hasn't had that problem. I've been running this Supermicro chassis since 2013, too.

I also had to discover that f11 during boot with this board gets you to a screen to choose the boot device, without needing to enter the BIOS. & While I was at the Supermicro site, I found and flashed a new firmware (specifically for just the IPMI part of the board) under the support section for this board. After upgrading the board BIOS to 2.3a, I can access the BIOS and change the boot order.

After catching up that part, I booted TrueNAS core 13.0U2 from a USB key I made from the .ISO file. When I told it where to install, it asked if I wanted just a new boot environment (rather than to reformat the whole disk and lose the one I was having the problem with), created it, and then when I booted, TrueNAS found the database, updated it, and all the configuration carried over without needing the backup configuration file.

To my surprise, everything seems to be working OK (including the new Jellyfin jail) after this. Watching it closely to see if the same thing happens again. My fingers are crossed.

Glad to know that you've fixed the problem without any destructive measures!

Hey @Whatteva, thank you for your help. Be sure to check out the Android Jellyfin app because the Chromecast functionality works fine on that one. I believe there is an app for iPhones too. If your TrueNAS server had any issues after installing Jellyfin, please let me know. My TrueNAS server has been one of the most reliable appliances in my home--I was so surprised to have something like this happen to it.

I haven't tried the Chromecast feature yet, but I'd give it a spin. Thanks for the heads up!