Upgrade from Bluefin to Cobia. Failed to start ix-netif.service

[cplatte]

Hello,
i went from 22.12.4.2 to 23.10.1. I did the upgrabe with the GUI. After the reboot i got several failed services.

Failed to start ix-netif.service
Failed to start ix-zfs.service - Import ZFS pools
Failed to start console-setup.service - Set console font and keymap
Failed to start rpcbind.service

After a while i see the normal Console Setup. but if i tried to open the Network configuration nothing happen. But i can open the CLI.
I also tried to do a clean install of cobia and import my config afterwards but nothing changed.
There are screenshots attached to this Post.
Is it Possible to reset the network config bevor boot?
Maybe the start of the console-setup.service is due to my german Keyboard?

I also tried to boot in 22.12.4.2 but if i treid this i got an error message about the zfs version and unsuported feature. Is it Possible to import my zfs pools back to Bluefin?


Mainboard: ASRock X570 Phantom Gaming X
CPU: AMD Ryzen 7 5700x
GPU: NVIDIA GTX 1050Ti
Storage: 8x 8TB WD HDD, 3x 2 TB NVME SSD
NIC: Mellanox Connect X-3

 

[joeschmuck]

Just to ask a few questions, and I am not the person to help you out, this is not in my wheelhouse, but the answers you provide may help out.

1) How much RAM do you have? This is just a formality unless the answer is not enough.
2) What is your boot drive?
3) When you installed the clean copy, was it over your original installation? I ask because if it was, then you can't just make the old boot environment active and then your system would be like it was before you attempted the upgrade. If you still have the original boot drive and can roll back, you should. This is selected during boot time.
4) The comment about zfs version not supported? Did you upgrade your feature set? I ask because normally this is a question in the GUI after you have upgraded TrueNAS and it asks you to upgrade. Of course NEVER UPGRADE YOUR POOL at least for a few weeks, or even months. I never upgrade my pool because there are no new features that affect me, and I can roll back if needed. So I'm curious if your pool was actually upgraded and was it via GUI?
5) Just curious, how are your hard drives connected to the computer? I don't think this has anything to do with the issue.
6) You have hard drives and NVMe drives, is this two pools or are the NVMe drives doing something special. If the NVMe drives are a different pool from the hard drives, then leave them out if you perform the steps below. Add them after the hard drives are properly working.

As I said, this is not my wheelhouse however if this were my problem, I would:

1) Disconnect all the hard drives
2) Try like hell to get the system to bootstrap to TrueNAS 23.10.1
3) Once I was able to do that, then create a clone of the boot environment so you can roll back if something goes wrong while troubleshooting.
4) Activate the new clone and reboot.
5) Once the GUI is online, restore your configuration file.
6) If that works, reboot and ensure the system comes back up. You will not have a pool since you have it disconnected but the GUI should be there.
7) Shutdown and reconnect your hard drives only.
8) Power on and observe, hopefully the pool loads up.
9) Make sure all works. If you have an NVMe pool to add, AND everything is working, then shutdown and add the NVMe drives, power up and cross your fingers.
10) Maybe you can't get the hard drives online, and if the NVMe drives are a pool, install those and disconnect the hard drives. See if you can get that working.

You can see, I'm taking small steps here to hopefully get the system running or to identify what does not work. It is a lot of small steps, take your time. Rework consumes a lot of time.

11) If you make some configuration change and the system refuses to boot correctly, during bootup switch the active environment to the original boot environment and start over. Delete the clone and then recreate a new clone and activate it. Start all over.

I'm certain someone else will chime in here but if you have more information to provide, please do so. I wish I have a single command you could run and all is right in the world, but that is beyond me right now.

Best of luck to you.

EDIT: If none of that works, and someone else is not able to help, and you have enough RAM, there is another way to get TrueNAS to run. You could install ESXi 8 (free). Absolute last option to try if nothing at all works to get the new TrueNAS to run. I would prefer a person use a different computer first. Just sayin'

 

[cplatte]

Just to ask a few questions, and I am not the person to help you out, this is not in my wheelhouse, but the answers you provide may help out.

1) How much RAM do you have? This is just a formality unless the answer is not enough.
2) What is your boot drive?
3) When you installed the clean copy, was it over your original installation? I ask because if it was, then you can't just make the old boot environment active and then your system would be like it was before you attempted the upgrade. If you still have the original boot drive and can roll back, you should. This is selected during boot time.
4) The comment about zfs version not supported? Did you upgrade your feature set? I ask because normally this is a question in the GUI after you have upgraded TrueNAS and it asks you to upgrade. Of course NEVER UPGRADE YOUR POOL at least for a few weeks, or even months. I never upgrade my pool because there are no new features that affect me, and I can roll back if needed. So I'm curious if your pool was actually upgraded and was it via GUI?
5) Just curious, how are your hard drives connected to the computer? I don't think this has anything to do with the issue.
6) You have hard drives and NVMe drives, is this two pools or are the NVMe drives doing something special. If the NVMe drives are a different pool from the hard drives, then leave them out if you perform the steps below. Add them after the hard drives are properly working.

As I said, this is not my wheelhouse however if this were my problem, I would:

1) Disconnect all the hard drives
2) Try like hell to get the system to bootstrap to TrueNAS 23.10.1
3) Once I was able to do that, then create a clone of the boot environment so you can roll back if something goes wrong while troubleshooting.
4) Activate the new clone and reboot.
5) Once the GUI is online, restore your configuration file.
6) If that works, reboot and ensure the system comes back up. You will not have a pool since you have it disconnected but the GUI should be there.
7) Shutdown and reconnect your hard drives only.
8) Power on and observe, hopefully the pool loads up.
9) Make sure all works. If you have an NVMe pool to add, AND everything is working, then shutdown and add the NVMe drives, power up and cross your fingers.
10) Maybe you can't get the hard drives online, and if the NVMe drives are a pool, install those and disconnect the hard drives. See if you can get that working.

You can see, I'm taking small steps here to hopefully get the system running or to identify what does not work. It is a lot of small steps, take your time. Rework consumes a lot of time.

11) If you make some configuration change and the system refuses to boot correctly, during bootup switch the active environment to the original boot environment and start over. Delete the clone and then recreate a new clone and activate it. Start all over.

I'm certain someone else will chime in here but if you have more information to provide, please do so. I wish I have a single command you could run and all is right in the world, but that is beyond me right now.

Best of luck to you.

EDIT: If none of that works, and someone else is not able to help, and you have enough RAM, there is another way to get TrueNAS to run. You could install ESXi 8 (free). Absolute last option to try if nothing at all works to get the new TrueNAS to run. I would prefer a person use a different computer first. Just sayin'

Hello,

1) 128gb DDR4 ECC
2) My Boot drive is an SATA SSD connected via USB
3) No i installed the clean copy on a different device. Thats the Point i canot rollback to Bluefin because of the feature set of ZFS
4) Not that i know about that. I never upgarded the ZFS version Manually. Does the Upgrade to cobia do it by it self?
5) The HDDs are all connected directly to the motherboard via SATA
6) The NVME drives are an seperated pool


I will try what you suggest. Thank you.

In this moment i have accces to the linux shell an i can see that under /mnt that all of my pools available. I treid also to do a clean install of 22.12.4.2. I tried there to import my config. Without my pools. I have the same issue that i canot open the GUI. It only shows ngix in the browser. After i delete my bridge config over the console setup and put evrything to DHCP i regain acces to the GUI.
That let me thing that i maybe only need to delete the network config on my COBIA system. But the Problem is then i try to Open The Network setting in the console setup nothing happend. The only thing i can open is the Linux shell or the truenas cli.
Did you know maybe how i can reset the network config with the treunas cli or even with linux shell.

Thank you for your help!

 

[joeschmuck]

After i delete my bridge config over the console setup and put evrything to DHCP i regain acces to the GUI.

I'm sure I am misunderstanding you but... If you have GUI access, can't you go in and change the network configuration from the GUI?

I do not know how to change the network configuration from the CLI. Did you Google 'trunas cli network' or something like that?

Try this...

Cannot Change IP on Interface through GUI

I am trying to set an IP on one of my interfaces and it will not save. Within the GUI I go to the interface and disable DHCP and Autoconfigure IPv6 and it allows me to enter in an Alias. I enter in 10.0.1.2/24 and hit save and nothing changes. A small progress bar at the top of the small config...

www.truenas.com

Does the Upgrade to cobia do it by it self?

I have never seen that happen. I move my pool between TrueNAS 13 (drives formatted probably around FreeNAS 11) and SCALE and never upgrade my ZFS feature set. That would screw me and I would not be able to roll back to TrueNAS 12 when I need to.

n this moment i have accces to the linux shell an i can see that under /mnt that all of my pools available.

This is very good. So your pool was mounted. What version of TrueNAS is this with?

So, you install 22.12.4.2. and you are able to reach the GUI, correct? Yes I think. Or is it the new version of TrueNAS that allows access to your pools? If it's the old version, I don't think your pool was upgraded.

It shits on you if you load your config file? yes I think.

How only is your config file? Did you make it just before your upgrade? Do you have a slightly older config file where your system was still in the same type of configuration. You can try to restore the older one. My script give me a copy of the config file once a week. I have a lot of copies in case something becomes corrupt.

I have a lot of ideas but I don't want to flood you with things to try until you have tried a few first.

 

[cplatte]

I'm sure I am misunderstanding you but... If you have GUI access, can't you go in and change the network configuration from the GUI?

I do not know how to change the network configuration from the CLI. Did you Google 'trunas cli network' or something like that?

Try this...

Cannot Change IP on Interface through GUI

I am trying to set an IP on one of my interfaces and it will not save. Within the GUI I go to the interface and disable DHCP and Autoconfigure IPv6 and it allows me to enter in an Alias. I enter in 10.0.1.2/24 and hit save and nothing changes. A small progress bar at the top of the small config...

www.truenas.com

I have never seen that happen. I move my pool between TrueNAS 13 (drives formatted probably around FreeNAS 11) and SCALE and never upgrade my ZFS feature set. That would screw me and I would not be able to roll back to TrueNAS 12 when I need to.


This is very good. So your pool was mounted. What version of TrueNAS is this with?

So, you install 22.12.4.2. and you are able to reach the GUI, correct? Yes I think. Or is it the new version of TrueNAS that allows access to your pools? If it's the old version, I don't think your pool was upgraded.

It shits on you if you load your config file? yes I think.

How only is your config file? Did you make it just before your upgrade? Do you have a slightly older config file where your system was still in the same type of configuration. You can try to restore the older one. My script give me a copy of the config file once a week. I have a lot of copies in case something becomes corrupt.

I have a lot of ideas but I don't want to flood you with things to try until you have tried a few first.

Sorry that was my mistake. No i have no GUI acces on My Cobia system.
What i descripted was only a test system with bluefin.
The Config file i generated bevor the upgrade.

With Cobia i mean my System with my data and evrything. With this Cobia System i see in the Linux shell that my zfs pools are mountet and accessible. After this system boot up i can only access option 6 and 7.
So that means zfs and the pools seems to be working fine. But i can not access the GUI.

 

[cplatte]

Sorry that was my mistake. No i have no GUI acces on My Cobia system.
What i descripted was only a test system with bluefin.
The Config file i generated bevor the upgrade.

With Cobia i mean my System with my data and evrything. With this Cobia System i see in the Linux shell that my zfs pools are mountet and accessible. After this system boot up i can only access option 6 and 7.
So that means zfs and the pools seems to be working fine. But i can not access the GUI.

Thats what i see after boot. But everything i tried to open except option 6 and 7 doenst work. I also tried to connect different network Cards.

 

[joeschmuck]

That is very odd. I hope someone swings on by to provide you some advice on where to go from here.

 

[cplatte]

That is very odd. I hope someone swings on by to provide you some advice on where to go from here.

I found the Problem. As i mention bevor the problem was the bridge network configuration. To solve this i deactivated all onboard network cards in the BIOS an discconected all other NICs. Then the system bootup normal. Now in the console menu it was finally possible to delete my bridge config.
Bevor all of that i tried also to delete this bridge interface with the truenas cli. but this was not possible. It run evrytime in a timeout.

Now everything works fine! Thank you for your help.
It seems to be that this behavior is a bug with the config file. Because even then i tried to upload my config file (that was generated also with bluefin same version) to Bluefin it doesnt work.