FreeNAS 11 becomes unreachable after a while, and won't shut down

[lctr01]

Hi,
I tried to update to FreeNAS 11-U1, but the server kept getting connection refused when trying to get an IP. So I rolled back to FreeNAS-11-Release and since then it keeps becoming unreachable after a couple of hours. Furthermore, it won't shut down anymore.

It's an HPE Microserver Gen8, with a fairly basic setup and 3 jails (Plex, Transmission, Sonarr)

I can connect to the server through ILO and through the remote console I see nothing relevant. But when connecting via browser directly to the GUI or any jail, it keeps loading forever.
Giving the reboot signal via remote console or directly, ends up in the system freezing

Code:

90 second watchdog timeout expired. Shutdown terminated
init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode
init: some processes would not die: ps axl advised

At this point it won't register any keystroke, so the only option I have left is to force shut down.

 

[KrisBee]

I would expect Ilo and GUI to be on different NICs , so ilo should always work. Hopefully you have saved the current config. A reinstall on a new usb stick, if your using one, maybe the quickest way to get going. But you need to checkout the HP's dual NIC to ensure it is working. Monitoring the boot process via the remote console might offer some clues to hardware status. Could be the failed upgrade screwed up your system.

 

[lctr01]

Thanks,
yes, they're on different ethernet ports, so that's why it works.
I can boot it just fine and export the settings then. At that stage, I am be able to shutdown and reboot normally. The issue happens after a while that it runs.

When it boots,I see a lot of warning/errors flashing just before the GRUB screen where you select what to boot. but they flash so quickly that I am not able to read what it says

I still have an image of FreeNAS 9 on the same stick, I'll try to load that and see if it's more stable

 

[KrisBee]

How long before is crashes? Anything in FreeNAS logs? I have upgraded to 11.1-U1 on my own microserver but it's not in use at the moment. So far no crashes in the hours it has run. I've forgotten, but is there a way to record boot messages via remote console? e.g:
https://www.youtube.com/watch?v=uqSt2qujRFk

It's the usual game of deciding if its a hardware or software problem.

 

[lctr01]

It just happened again. Lasted about 3 hours.

Code:

transmission-daemon[9093]: UDP Failed to set receive buffer: No buffer space available (tr-udp.c:65)
transmission-daemon[9093]: UDP Failed to set receive buffer: requested 3194304, got 42080 (tr-udp.c:84)
freenas kernel: pid 9684 (Plex Transcoder), uid 972 was killed: out of swap space
[system.alert:400] Alert module '<samba4.Samba4Alert object act 0x814b107b8>' failed: timed out

The web interface could not be accessed.
Please check your network configuration.

So I entered 11 - Reboot.

Code:

Shutdown NOW!
shutdown: [pid 18627]

System shutdown time has arrived
Stopping cron.
Waiting for PIDS: 8788.
Stopping zfsd.
Waiting for PIDS: 8778.
90 second watchdog timeout expired. Shutdown terminated
init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode
init: some processes would not die: ps axl advised

How can I find out what 8778 was? Will the PID change at the next reboot?


These are the messages I see flashing just before the GRUB boot

Code:

error: compression algorithm 99 not supported
error: compression algorithm 99 not supported
error: compression algorithm 99 not supported
error: compression algorithm 99 not supported
error: compression algorithm inherit not supported
error: compression algorithm inherit not supported
error: compression algorithm 99 not supported
error: compression algorithm 99 not supported

 

[toadman]

11.1 has a known memory leak. But I haven't heard of one in 11. https://forums.freenas.org/index.php?threads/smb-memory-leak.59682/

But it sure looks like a memory issue. What is your hardware? What output do you get for sysctl -a | grep free_target?

Not sure yet about the compression messages. Does that happen during a fresh boot?

 

[lctr01]

Hi,
It's a HPE Microserver Gen8

Build FreeNAS-11.1-RELEASE
Platform Intel(R) Celeron(R) CPU G1610T @ 2.30GHz
Memory 3895MB
Load Average 0.64, 1.36, 1.69

Code:

[root@freenas ~]# sysctl -a | grep free_target								 
vm.v_free_target: 20571														 
vm.stats.vm.v_free_target: 20571												
vfs.zfs.arc_free_target: 6743   

 

[toadman]

Then it's the memory leak. So you have two issues:

• The first is the 4GB system memory. Everyone here will tell you 8GB is the minimum and you probably won't get further support running less than that value just because there are so many known issues. It's known the middleware takes more memory in the 11+ versions. AND you are running jails. So I think you are likely to have continued issues unless you upgrade to 8GB+.
• The second is the known memory leak in 11.1 release (I think.) https://redmine.ixsystems.com/issues/27422

You can either roll back to 11.0-U4 or forward to 11.1-U1 (which has some other issues that may spam your logs, but otherwise seems stable to me). But you need to do one or the other to eliminate that known memory leak (which is obviously hitting you).

On a side note, I have been playing with the 'free_target' values because I was seeing swaping, even on a 16GB system. I suggest you add tunables of type "sysctl" and set both vfs.zfs.arc_free_target and vm.v_free_target to 65536. You can do it at the command line as well with sysctl vfs.zfs.arc_free_target=65535. (But you need to add the tunables for it to be persistent on a reboot.)

 

[lctr01]

Thanks.
I jumped from 9.10.2-U6 to 11.1-Release, so if that's the case I'll go back to 9.

To be fair, the system ran just fine from mid december until last week when I tried to update to 11.1-U1, but I'll look into adding another module of ram since there's an empty slot

Thanks!

 

[toadman]

Yep, I have no doubt it ran. Just letting you know as you move forward and ask questions on issued you'll probably be met with "less than 8GB" replies and folks will stop there. And know the 11+ series will use more RAM (I think) because more functionality is being added to the appliance.

Yes, rolling back to 9.x will eliminate the memory leak. Unless you upgraded your pool, in which case I am not sure 9.x will work anymore.

 

[KrisBee]

Thanks.
I jumped from 9.10.2-U6 to 11.1-Release, so if that's the case I'll go back to 9.

To be fair, the system ran just fine from mid december until last week when I tried to update to 11.1-U1, but I'll look into adding another module of ram since there's an empty slot

Thanks!

The HP microserver gen8 tolerates a fairly wide range of ECC UDIMMs from hynix, micron, crucial and samsung. You can probably get away with mixing manufacturers as long as specs are the same. Even seen pc3-14900e used, which can be the cheapest per GB on fleabay. No need to pay more than £20 for a 4GB stick and it doesn't have to be official HP memory for it to boot OK.

 

[lctr01]

Thank you both. I have purchased 1x EQ HP 4GB PC3-12800E DDR3-1600 ECC Unbuffered UDimm Server Memory Ram 820077-B21 refurbished from Ebay for £20, which should match what I already have inside the server. (1 x 4GB PC3-12800E DDR3 UDIMM)

 

[lctr01]

The ram arrived this morning and now I have 8gb.

I have not experienced the issue above since yesterday when I booted into FreeNAS 9....I'll wait to upgrade until the memory leak is resolved.