Middlewared Not Started During Boot

[gladiola]

I'm a FreeNAS newbie who's been using FreeBSD for about ten years. I tried my first FreeNAS install last night, and it has not gone well. I cannot get the system to stand up completely; I see persistent error messages about the middlewared not started. As near as I can tell, it creates a cascade of other errors that I see.

I think I am probably experiencing the same kind of condition described in this unsolved thread: https://www.ixsystems.com/community/threads/middleware-failed-to-start.75493/

I am installing FreeNAS 11.2-U3 to a Dell T110 Poweredge with 8GiB RAM and four HDD in the box. The ISO was written to a USB drive. BIOS set to boot from the internal drive. I can tell you that I had to try many times to write the ISO. The installer was so quiet that several times I had thought it had just completely stalled out and was doing nothing. I wasn't sure if I was wrestling with the BIOS settings or the installer of a new kind of system I had not used before.

I would clear the USB by switching it back over to a Win10 machine and reformatting the drive. Once written to drive, it took about an hour for the install process to complete. The console output was extremely slow. Compared to other FreeBSD installs from ports, I did not see the type of output I would have been expecting. Out of exasperation, I just left the machine alone and let it run. At one point, it was about a one word change every ten or twenty minutes. Very slow. It eventually picked up speed after the base os install part 1 segment. I did see and receive a "Success" dialog that matches the one on the directions. When I powered up this morning for its initial trial, that's when I started seeing the middlewared not started errors.

The installer was written using a separate machine. I wrote the ISO using imageUSB from Proxmark. It's the one that comes with memtest86. I've used it before without trouble. I had also tried writing to DVD. That imaging would work, but when I was conducting that install on the T110 itself, I kept getting errors that said it would not let me write to da0, where the USB was waiting. It was being recognized by the installer dialog; but, it would eventually fail. When I switched over to writing the installer to USB using a separate machine, I was able to succeed.

Once the BIOS of the T110 is rigged to recognize the internal USB, it'll go to the FreeNAS installer. Aside from the cascade of middlewared not started errors, it looks like the startup process is rolling along. When I get to the end, I do get to a "Console setup" menu that allows me to go to shell.

When I give a service -e I can see several services running, but none of them are middlewared.
If I give a service middlewared start, then I see a PID of such a service already running.

When I drill down into /usr/local/etc/rc.d and cat the middlewared file, I can see the middlewared_start() function. I can see the error message in that function that I was witnessing earlier during startup. When I do a printenv, I cannot see LD_LIBRARY_PATH set. When I look at the PATH variable, I don't see a value that is as complex as the one set in the "else" block that follows the tmux stuff.

When I say I think I'm seeing other errors related to this middlewared not started problem, I mean:

• "The web interface could not be accessed."
• "Error: attempt to write to a read-only database"
• Python scripting errors about a server cert
• Many warnings about the middlewared not started, but they seem to subside near the middlewared plugins section.

I did get an IP and try to access it from an adjacent machine. I did not get a login form. I received a slowly appearing logo. Then I received a warning dialog telling me something like NAS had not started. I was not able to use the web interface at all. I noticed during some of the forum threads I skimmed that seems to be the main mechanism for making config adjustments. So, I'm a little worried about that. However, I suspect that if we can get the middlewared up and running that things might go better.

I suspect that the service is getting used in a way, but that it is working partially and too late. Since it's not ready when it seems to be needed, I don't think the install will work as it is. I thought I would ask for help.

Please advise.

What is the preferred way to get the middlewared service started at the right time during boot? Any advice you can provide might be helpful.

 

[Chris Moore]

I was not able to use the web interface at all. I noticed during some of the forum threads I skimmed that seems to be the main mechanism for making config adjustments.

There are very few things that can be done from the CLI. The Web GUI is intended to be where setting are changed and the Web GUI hands input over to the middleware which is what takes the action on the OS. All settings are saved in the configuration database and many of the files that you might edit to change things from the command line are regenerated during boot based on the settings saved in the database. For that reason, most settings that are not made from the Web GUI will not survive a reboot.

So, I'm a little worried about that. However, I suspect that if we can get the middlewared up and running that things might go better.

What you have described is a failed installation. I can't say what it is that is causing the problem, but this needs to be reinstalled to ensure it is all working. What media are you attempting to install to exactly?

 

[gladiola]

Just maybe considering changing that wait value in the start script to be ridiculously long. Everything else took forever. Maybe I just need to give it a lot of time to succeed.

Chris:

I am using a USB thumb drive plugged in to an internal port on the Dell motherboard. PNY 32GB 3.0 USB. It's the gold standard for $8.99 drives. I saw the exact model number in the dmesg, but I just turned off the machine.

I will consider a fresh attempt to write the installer to the drive. I didn't detail that above, but my successful attempt involved stripping the HDD from a laptop to prevent accidents; then I used a DVD ISO of the downloaded installer from the website to write to the USB. The USB was then transferred to the inside of the Dell T110. Initial powerup wouldn't work because the BIOS needs to detect a USB before it'll switch over to giving us the option to activate a USB as a HDD on boot. Reboot and the machine found it.

I will write back later and follow up.

 

[Chris Moore]

PNY 32GB 3.0 USB. It's the gold standard for $8.99 drives.

Lately, since the switch to the BSD boot loader, we have seen a lot of problems using a USB drive as the target for the OS install. The current recommendation is a small capacity SSD. Please see this:

Hardware Requirements
http://www.freenas.org/hardware-requirements/

Also, we have seen a higher than usual failure rate from USB 3.0 devices because they tend to overheat under the workload that the FreeNAS OS applies to them.

I will consider a fresh attempt to write the installer to the drive. I didn't detail that above, but my successful attempt involved stripping the HDD from a laptop to prevent accidents; then I used a DVD ISO of the downloaded installer from the website to write to the USB. The USB was then transferred to the inside of the Dell T110. Initial powerup wouldn't work because the BIOS needs to detect a USB before it'll switch over to giving us the option to activate a USB as a HDD on boot. Reboot and the machine found it.

To be clear, there is one USB drive that you are using to install from and another drive that you are installing to? FreeNAS can't be installed to the same drive that the installer booted from. That will never work. It did years and years ago in a very old version, but it does not work that way now.

 

[gladiola]

I would run the installer on the DVD while selecting the USB in the install script as the destination for FreeNAS.

 

[gladiola]

Tried some more. Hit a kernel panic this time. Photos of the messages.
Installing from DVD to USB. Selected destination drive in curses. Waited about five minutes. Panic.

 

[gladiola]

Ran it again and got the same panic.